System Design Simple logoSystem Design Simple Follow on YouTube

AI SystemsSystem Design, simply · AI 38

Long context vs RAG: a model takes 700,000 words, so why search?

Long context vs RAG: a big context window can hold a whole document, but every resend costs input tokens and facts in the middle can be missed. Test both.

Long context vs RAG: a model takes 700,000 words, so why search?: diagrams from the video

Drafted with AI assistance, then checked and edited by a human engineer · How we write

For long context vs RAG, start with full-document input when one document fits and you need only a few answers; test retrieval for repeated questions over a large collection. RAG reduces prompt size, but its usefulness depends on retrieving the evidence each answer needs. A model that can take 700,000 words can read the whole document, yet repeated input can still incur charges, with discounts when caching applies, and research from 2023 found facts in the middle of a long prompt were used less reliably.

How much can a long context window hold?

A context window is the amount of text, counted in tokens, that a model can take in on one call. On Feb 15, 2024, Google announced Gemini 1.5 Pro with a standard 128,000 token window and an experimental window of up to 1 million tokens, offered to a limited group of developers and enterprise customers in preview. Google wrote that this lets 1.5 Pro process "over 700,000 words" in one go (Google). The same post reports that in its Needle In A Haystack test, 1.5 Pro found an embedded fact 99% of the time in inputs as long as 1 million tokens.

Google's Gemini 1.5 post: up to 1 million tokens in production, and 1.5 Pro can process over 700,000 words
Google's post "Our next-generation model: Gemini 1.5", Feb 15, 2024. Captured 2026-11-16. Source: blog.google.

Sending the whole document with every question still has a price in tokens and in accuracy. We covered what happens when a chat outgrows its window in why a long AI chat forgets your name.

What does sending the whole document cost?

A per-token API bills the input tokens of every request. The Gemini API pricing page lists prices per 1M tokens (Gemini API pricing). If the document is the prompt, each new question sends it again, and those tokens are counted again. This is the same arithmetic as why an AI agent re-sends its history.

Caching changes the price of the repeat. The Gemini API docs, read on 2026-11-16, say implicit caching is on by default for Gemini 2.5 and newer models: "We automatically pass on cost savings if your request hits caches" (Gemini API docs). The docs suggest putting large, shared content at the start of the prompt and sending requests with a similar prefix close together.

Gemini API docs, Implicit caching: enabled by default for Gemini 2.5 and newer models, cost savings passed on when a request hits caches
The Implicit caching section of the Gemini API docs. Captured 2026-11-16. Source: ai.google.dev.

Cached tokens still count toward the context window and may be billed at a discounted rate. Prefix caching can reuse the prefix's stored attention keys and values, reducing repeated prompt processing; subsequent tokens can still attend to that prefix (Transformers docs).

Does the model use everything in a long prompt?

Liu et al. (2023) put the passage that answers a question at different positions among 10, 20 or 30 documents. They found "a distinctive U-shaped performance curve": accuracy was highest when the passage sat at the start or end of the input and dropped when it sat in the middle (arXiv 2307.03172).

For one model tested in the paper, the worst middle positions in the 20 and 30 document settings scored below its closed-book accuracy of 56.1%. Closed-book means the model got no documents and answered from what it learned in training.

Lost in the Middle paper: performance on the multi-document question task is lower than when predicting without any documents, closed-book 56.1%
From Liu et al. 2023, "Lost in the Middle", arXiv 2307.03172v3. Captured 2026-11-16. Source: arxiv.org.

These were models from 2023, and newer ones may behave differently, so test on your own data. The paper also ran a search case study: on open-domain questions, reader accuracy stopped improving long before retriever recall did, and 50 retrieved documents helped only a little over 20.

How does RAG search work?

Retrieval-augmented generation (RAG) is a design where a search step picks a few relevant passages and only those go into the prompt. Lewis et al. (2020) built it from a generator model plus "a dense vector index of Wikipedia", searched by a pre-trained neural retriever (arXiv 2005.11401).

In an app the steps are: split the documents into chunks, build an index once, then for each question search the index and send the top few chunks with the question. The prompt stays small whatever the size of the collection. We walked through the retrieval step in how AI finds one line in 100 pages, and why a search for car finds vehicle in embeddings explained.

Our lab uses BM25, a keyword ranking method. Our BM25 index stores a posting list for each term: which chunks contain it and how often. At query time, matching terms contribute scores. Rare terms receive more weight, repeated occurrences have diminishing returns, and chunk length adjusts the score. The five highest-scoring chunks enter the prompt. This implementation matches words, so a reworded question can miss evidence that uses different vocabulary.

Where RAG breaks is retrieval: if search misses the chunk that holds the answer, the model never sees that evidence and answers without it. A retry loop helps, as in agentic RAG.

Long context vs RAG compared

Long contextRAG (search, then read)
What goes in the promptThe whole document, every questionThe question plus the top few chunks
Input tokens per questionThe size of the documentA few chunks
Main riskThe fact sits in the middle and gets used lessSearch misses the right chunk
CachingA cache hit lowers the cost of the repeated prefixLess to cache; the prompt is small
SetupNoneChunking, an index, a retriever
Good forQuestions that need the whole document at onceMany questions over a large or growing collection

A simulation: tokens and search misses on 1,000 pages

This is our own simulation. The script in labs/ai-38 (node compare-long-context-vs-rag.mjs, Node 22.13.1, run on 2026-11-16) builds 1,000 synthetic pages of about 400 words, plants 30 facts and asks one question per fact. It sends no prompt to any model, so it measures nothing about answer accuracy or real response time.

NumberLong contextRAG, BM25 top 5Kind
Input tokens per question772,7501,484Estimated by a heuristic counter, not a real tokenizer
Cost per 1,000 questions$2,319$5.20Modeled at example prices ($3 in, $15 out per 1M tokens), no caching
Prompt read time77.7 s0.55 sModeled at an assumed reading speed

Cost assumes 50 output tokens per answer: (input × $3 + 50 × $15) / 1,000,000 per question. Modeled prompt time is 0.4 + input / 10,000 seconds, plus measured search time for RAG. Both formulas are assumptions written into the script.

We did measure the search side. Over 450 retrieval runs (30 facts moved to the start, middle and end, 5 trials), it found the right chunk in its top 5 for 315 of 315 runs where the question named the module, and missed 23 of 135 runs where the question was reworded the way a person asks. BM25 search averaged 0.732 ms per question in this run. The found rates by position, 96% at the start, 96% in the middle and 92.7% at the end, describe BM25 search and say nothing about lost in the middle in a model.

With synthetic text and 30 facts, read these numbers as a sketch of the trade-off. Search cut the estimated input by a large factor, and all 23 retrieval misses came from the 135 runs that used reworded questions.

What we saw when we ran it

We ran node labs/ai-38/compare-long-context-vs-rag.mjs on 2026-11-16. It built 1000 synthetic pages and did 450 retrieval runs: 30 facts, each placed at the start, middle and end, with 5 trials. The screen showed the long context prompt at 772,750 approximate tokens per question against 1,484 for RAG with BM25 top 5, and a modeled cost of $2319.00 against $5.20 per 1,000 questions. BM25 missed 0 of 315 direct question runs and 23 of 135 paraphrased ones.

Position also changed the BM25 result. The found rate was 96% at the start and middle and 92.7% at the end, and 7 of the 30 facts changed result when only their position moved. The run made no model calls, so it says nothing about answer accuracy. The check worth copying is reworded questions: test retrieval with the phrasing your users actually type before you trust it.

Terminal output of our ai-38 run: node compare-long-context-vs-rag.mjs Our terminal after running the ai-38 lab script.

Screenshot of the official page: Google: Our next-generation model: Gemini 1.5 (Feb 15, 2024) Google's official announcement post, the page we cite for the long context window.

The post states both points we rely on: the experimental long context window, offered in limited preview, and how much text Google said the model can take in one go.

How to choose: test both on your questions

Take 20 real questions from your users, run each through both designs, and record accuracy, latency and input tokens. Mark every answer that has no supporting passage in what the model saw. If you build with an AI coding agent, you can paste this:

For this feature, compare full-document and top-chunk retrieval on 20 real questions; report accuracy, latency, input tokens, and flag answers lacking supporting evidence.

Our take

For one big document and a handful of questions, sending it all is the simplest design and needs no index. For many questions over a large collection, search sends far fewer tokens per question. Its weak point is the miss, so measure recall on reworded questions before trusting it. Whichever you pick, check answers against the passages the model actually saw; see AI citation checks.

This article summarizes Google's Gemini 1.5 post (Feb 15, 2024), the Gemini API caching and pricing docs (read 2026-11-16), the Hugging Face Transformers caching docs, Liu et al. 2023 and Lewis et al. 2020. The table of token, cost and time numbers comes from our own simulation on synthetic text; no model or real workload was involved.

Frequently asked questions

Is RAG still needed with long context windows?

Often, yes. Long context resends the whole document as input on every question, with discounts only when caching applies, while RAG sends a few matching chunks and depends on search finding them.

What is lost in the middle?

Liu et al. (2023) found that the 2023 models they tested answered best when the relevant passage sat at the start or end of a long input and worse when it sat in the middle, a U-shaped curve.

Does prompt caching make long context cheap?

It can make a repeated prefix cheaper: cached tokens may be billed at a discounted rate, and prefix caching can reuse the stored attention keys and values. Cached tokens still count toward the context window.

Sources

  1. Google: Our next-generation model: Gemini 1.5 (Feb 15, 2024)blog.google
  2. Gemini API docs: Context cachingai.google.dev
  3. Gemini API docs: Pricingai.google.dev
  4. Hugging Face Transformers docs: Cachinghuggingface.co
  5. Liu et al. (2023): Lost in the Middle: How Language Models Use Long Contextsarxiv.org
  6. Lewis et al. (2020): Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasksarxiv.org

ai systemsraglong contextcontext windowlost in the middlesystem design

System Design Simple logo
System Design Simple

Written by a software engineer and system architect with 10+ years of building production web, payment, ticketing and accounting systems in Japan and Nepal. About us