# Long context vs RAG: a model takes 700,000 words, so why search?

URL: https://systemdesignsimple.com/long-context-vs-search-rag/
Published: 2026-11-16
Category: AI Systems
Publisher: System Design Simple (https://systemdesignsimple.com)

## Summary

Long context sends the whole document with every question; RAG searches an index and sends only the matching chunks. Google said in Feb 2024 that Gemini 1.5 Pro could process over 700,000 words, but repeated input can still incur charges, with discounts when caching applies, and Liu et al. (2023) found facts mid-prompt were used worse. RAG sends less but fails when search misses.

## Key facts

- **Gemini 1.5 Pro:** Google, Feb 15, 2024: an experimental 1 million token window in limited preview, able to process over 700,000 words; standard window 128,000 tokens
- **Caching:** Gemini API docs (read 2026-11-16): implicit caching is on by default for Gemini 2.5 and newer, and savings apply when a request hits a cache
- **Lost in the Middle:** Liu et al. 2023 (arXiv 2307.03172): accuracy follows a U-shaped curve, highest when the relevant passage is at the start or end of the input
- **Worse than no documents:** For one model tested in the paper, the worst case in the 20 and 30 document settings scored below its closed-book accuracy of 56.1%
- **RAG:** Lewis et al. 2020 (arXiv 2005.11401) paired a generator with a dense vector index of Wikipedia and a neural retriever

For long context vs RAG, start with full-document input when one document fits and you need only a few answers; test retrieval for repeated questions over a large collection. RAG reduces prompt size, but its usefulness depends on retrieving the evidence each answer needs. A model that can take 700,000 words can read the whole document, yet repeated input can still incur charges, with discounts when caching applies, and research from 2023 found facts in the middle of a long prompt were used less reliably.

## How much can a long context window hold? {#how-much}

A context window is the amount of text, counted in tokens, that a model can take in on one call. On Feb 15, 2024, Google announced Gemini 1.5 Pro with a standard 128,000 token window and an experimental window of up to 1 million tokens, offered to a limited group of developers and enterprise customers in preview. Google wrote that this lets 1.5 Pro process "over 700,000 words" in one go ([Google](https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/)). The same post reports that in its Needle In A Haystack test, 1.5 Pro found an embedded fact 99% of the time in inputs as long as 1 million tokens.

[Image: Google's Gemini 1.5 post: up to 1 million tokens in production, and 1.5 Pro can process over 700,000 words | Google's post "Our next-generation model: Gemini 1.5", Feb 15, 2024. Captured 2026-11-16. Source: [blog.google](https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/).]

Sending the whole document with every question still has a price in tokens and in accuracy. We covered what happens when a chat outgrows its window in [why a long AI chat forgets your name](/my-chat-forgot-my-name/).

## What does sending the whole document cost? {#cost}

A per-token API bills the input tokens of every request. The Gemini API pricing page lists prices per 1M tokens ([Gemini API pricing](https://ai.google.dev/gemini-api/docs/pricing)). If the document is the prompt, each new question sends it again, and those tokens are counted again. This is the same arithmetic as [why an AI agent re-sends its history](/why-an-ai-agent-re-sends-its-history/).

Caching changes the price of the repeat. The Gemini API docs, read on 2026-11-16, say implicit caching is on by default for Gemini 2.5 and newer models: "We automatically pass on cost savings if your request hits caches" ([Gemini API docs](https://ai.google.dev/gemini-api/docs/caching)). The docs suggest putting large, shared content at the start of the prompt and sending requests with a similar prefix close together.

[Image: Gemini API docs, Implicit caching: enabled by default for Gemini 2.5 and newer models, cost savings passed on when a request hits caches | The Implicit caching section of the Gemini API docs. Captured 2026-11-16. Source: [ai.google.dev](https://ai.google.dev/gemini-api/docs/caching).]

Cached tokens still count toward the context window and may be billed at a discounted rate. Prefix caching can reuse the prefix's stored attention keys and values, reducing repeated prompt processing; subsequent tokens can still attend to that prefix ([Transformers docs](https://huggingface.co/docs/transformers/en/kv_cache)).

## Does the model use everything in a long prompt? {#middle}

Liu et al. (2023) put the passage that answers a question at different positions among 10, 20 or 30 documents. They found "a distinctive U-shaped performance curve": accuracy was highest when the passage sat at the start or end of the input and dropped when it sat in the middle ([arXiv 2307.03172](https://arxiv.org/abs/2307.03172)).

For one model tested in the paper, the worst middle positions in the 20 and 30 document settings scored below its closed-book accuracy of 56.1%. Closed-book means the model got no documents and answered from what it learned in training.

[Image: Lost in the Middle paper: performance on the multi-document question task is lower than when predicting without any documents, closed-book 56.1% | From Liu et al. 2023, "Lost in the Middle", arXiv 2307.03172v3. Captured 2026-11-16. Source: [arxiv.org](https://arxiv.org/html/2307.03172v3).]

These were models from 2023, and newer ones may behave differently, so test on your own data. The paper also ran a search case study: on open-domain questions, reader accuracy stopped improving long before retriever recall did, and 50 retrieved documents helped only a little over 20.

## How does RAG search work? {#rag}

Retrieval-augmented generation (RAG) is a design where a search step picks a few relevant passages and only those go into the prompt. Lewis et al. (2020) built it from a generator model plus "a dense vector index of Wikipedia", searched by a pre-trained neural retriever ([arXiv 2005.11401](https://arxiv.org/abs/2005.11401)).

In an app the steps are: split the documents into chunks, build an index once, then for each question search the index and send the top few chunks with the question. The prompt stays small whatever the size of the collection. We walked through the retrieval step in [how AI finds one line in 100 pages](/how-ai-finds-one-line-in-100-pages/), and why a search for car finds vehicle in [embeddings explained](/why-search-for-car-finds-vehicle/).

Our lab uses BM25, a keyword ranking method. Our BM25 index stores a posting list for each term: which chunks contain it and how often. At query time, matching terms contribute scores. Rare terms receive more weight, repeated occurrences have diminishing returns, and chunk length adjusts the score. The five highest-scoring chunks enter the prompt. This implementation matches words, so a reworded question can miss evidence that uses different vocabulary.

Where RAG breaks is retrieval: if search misses the chunk that holds the answer, the model never sees that evidence and answers without it. A retry loop helps, as in [agentic RAG](/agentic-rag-search-retry-loop/).

## Long context vs RAG compared {#compare}

| | Long context | RAG (search, then read) |
|---|---|---|
| What goes in the prompt | The whole document, every question | The question plus the top few chunks |
| Input tokens per question | The size of the document | A few chunks |
| Main risk | The fact sits in the middle and gets used less | Search misses the right chunk |
| Caching | A cache hit lowers the cost of the repeated prefix | Less to cache; the prompt is small |
| Setup | None | Chunking, an index, a retriever |
| Good for | Questions that need the whole document at once | Many questions over a large or growing collection |

## A simulation: tokens and search misses on 1,000 pages {#simulation}

This is our own simulation. The script in <code>labs/ai-38</code> (<code>node compare-long-context-vs-rag.mjs</code>, Node 22.13.1, run on 2026-11-16) builds 1,000 synthetic pages of about 400 words, plants 30 facts and asks one question per fact. It sends no prompt to any model, so it measures nothing about answer accuracy or real response time.

| Number | Long context | RAG, BM25 top 5 | Kind |
|---|---|---|---|
| Input tokens per question | 772,750 | 1,484 | Estimated by a heuristic counter, not a real tokenizer |
| Cost per 1,000 questions | $2,319 | $5.20 | Modeled at example prices ($3 in, $15 out per 1M tokens), no caching |
| Prompt read time | 77.7 s | 0.55 s | Modeled at an assumed reading speed |

Cost assumes 50 output tokens per answer: `(input × $3 + 50 × $15) / 1,000,000` per question. Modeled prompt time is `0.4 + input / 10,000` seconds, plus measured search time for RAG. Both formulas are assumptions written into the script.

We did measure the search side. Over 450 retrieval runs (30 facts moved to the start, middle and end, 5 trials), it found the right chunk in its top 5 for 315 of 315 runs where the question named the module, and missed 23 of 135 runs where the question was reworded the way a person asks. BM25 search averaged 0.732 ms per question in this run. The found rates by position, 96% at the start, 96% in the middle and 92.7% at the end, describe BM25 search and say nothing about lost in the middle in a model.

With synthetic text and 30 facts, read these numbers as a sketch of the trade-off. Search cut the estimated input by a large factor, and all 23 retrieval misses came from the 135 runs that used reworded questions.

## What we saw when we ran it {#we-ran-it}

We ran `node labs/ai-38/compare-long-context-vs-rag.mjs` on 2026-11-16. It built 1000 synthetic pages and did 450 retrieval runs: 30 facts, each placed at the start, middle and end, with 5 trials. The screen showed the long context prompt at 772,750 approximate tokens per question against 1,484 for RAG with BM25 top 5, and a modeled cost of $2319.00 against $5.20 per 1,000 questions. BM25 missed 0 of 315 direct question runs and 23 of 135 paraphrased ones.

Position also changed the BM25 result. The found rate was 96% at the start and middle and 92.7% at the end, and 7 of the 30 facts changed result when only their position moved. The run made no model calls, so it says nothing about answer accuracy. The check worth copying is reworded questions: test retrieval with the phrasing your users actually type before you trust it.

![Terminal output of our ai-38 run: node compare-long-context-vs-rag.mjs](/img/posts/long-context-vs-search-rag/lab-run.webp)
*Our terminal after running the ai-38 lab script.*

![Screenshot of the official page: Google: Our next-generation model: Gemini 1.5 (Feb 15, 2024)](/img/posts/long-context-vs-search-rag/source-1.webp)
*Google's official announcement post, the page we cite for the long context window.*

The post states both points we rely on: the experimental long context window, offered in limited preview, and how much text Google said the model can take in one go.

## How to choose: test both on your questions {#test}

Take 20 real questions from your users, run each through both designs, and record accuracy, latency and input tokens. Mark every answer that has no supporting passage in what the model saw. If you build with an AI coding agent, you can paste this:

```text
For this feature, compare full-document and top-chunk retrieval on 20 real questions; report accuracy, latency, input tokens, and flag answers lacking supporting evidence.
```

## Our take {#our-take}

For one big document and a handful of questions, sending it all is the simplest design and needs no index. For many questions over a large collection, search sends far fewer tokens per question. Its weak point is the miss, so measure recall on reworded questions before trusting it. Whichever you pick, check answers against the passages the model actually saw; see [AI citation checks](/ai-source-quote-wasnt-there/).

This article summarizes Google's Gemini 1.5 post (Feb 15, 2024), the Gemini API caching and pricing docs (read 2026-11-16), the Hugging Face Transformers caching docs, Liu et al. 2023 and Lewis et al. 2020. The table of token, cost and time numbers comes from our own simulation on synthetic text; no model or real workload was involved.

## Frequently asked questions

**Is RAG still needed with long context windows?**
Often, yes. Long context resends the whole document as input on every question, with discounts only when caching applies, while RAG sends a few matching chunks and depends on search finding them.

**What is lost in the middle?**
Liu et al. (2023) found that the 2023 models they tested answered best when the relevant passage sat at the start or end of a long input and worse when it sat in the middle, a U-shaped curve.

**Does prompt caching make long context cheap?**
It can make a repeated prefix cheaper: cached tokens may be billed at a discounted rate, and prefix caching can reuse the stored attention keys and values. Cached tokens still count toward the context window.

## Sources

- [Google: Our next-generation model: Gemini 1.5 (Feb 15, 2024)](https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/)
- [Gemini API docs: Context caching](https://ai.google.dev/gemini-api/docs/caching)
- [Gemini API docs: Pricing](https://ai.google.dev/gemini-api/docs/pricing)
- [Hugging Face Transformers docs: Caching](https://huggingface.co/docs/transformers/en/kv_cache)
- [Liu et al. (2023): Lost in the Middle: How Language Models Use Long Contexts](https://arxiv.org/abs/2307.03172)
- [Lewis et al. (2020): Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401)
