AI SystemsSystem Design, simply · AI 33
Agentic RAG: the AI searched, didn't like what it found, and searched again
Agentic RAG lets an agent decide when to search again. We test one design (grade, rewrite, retry) and measure what each extra round costs.

Drafted with AI assistance, then checked and edited by a human engineer · How we write
Agentic RAG lets an agent decide when and how to retrieve information while preparing an answer. We tested one design: grade the retrieved results, rewrite a failed query and retry within a cap. In our simulation run on 2026-11-15, one search found the right page for 20 of 40 questions, a cap of 2 searches found 32 and a cap of 4 found 38. Every extra round cost about 1.2 modeled seconds and a few hundred tokens, and the fourth round rescued nothing.
Why one search misses: everyday words, technical pages
You type "Why does the site keep forgetting that I signed in?" The help page that answers it is titled "Session cookie expiry". None of your words appear on that page, so a keyword search ranks other pages above it.
This is vocabulary mismatch, and we measured it before: in our RAG retrieval lab, BM25 found 29 of 30 same-word questions but only 4 of 30 reworded ones. Embeddings narrow the gap (why a search for car finds vehicle) without closing it. Plain RAG retrieves once, and the model answers from whatever came back.
What is agentic RAG?
The design we tested has three parts:
- A grader reads the question and the top results and answers yes or no.
- A rewriter writes a new query in the words a technical document would use.
- A cap stops the loop after N searches, found or not.
The grader only decides whether to continue. Each rewrite starts from the original question, and the steps stack: round 2 adds dictionary synonyms (forgetting adds session cookie expiry, and the original words stay), round 3 expands technical terms, and round 4 removes terms appearing in more than 4% of documents. Retrieved results do not influence these rewrites. The grader passes when the known answer page is in the top three; no answer is generated or checked.
The CRAG paper (Yan et al., January 2024) adds a lightweight evaluator that scores the retrieved documents and, based on its confidence, triggers different retrieval actions, including web search (arXiv 2401.15884). Self-RAG (Asai et al., October 2023) trains one model to retrieve on demand and critique passages with special reflection tokens (arXiv 2310.11511). Anthropic's engineering post of June 13, 2025 says its research subagents, after tool results, "evaluate quality, identify gaps, and refine their next query" (Anthropic Engineering).
What the simulation measured
The script labs/ai-33/measure-agentic-search-retries.mjs builds 200 short made-up tech pages: 40 answer pages, 80 near misses that share one term with an answer page, and 80 filler pages. It asks 40 questions with one known answer page each: twenty in the page's own words, twenty in everyday words the page never uses. Search is plain BM25 (k1 = 1.2, b = 0.75), top 3. Node 22, seed 33; the counts repeat on every run.
setting found technical everyday avg rounds loop tokens/q modeled s/q
one-shot 20/40 20/20 0/20 1.00 0 0.03
agentic, cap 2 32/40 20/20 12/20 1.50 321 1.17
agentic, cap 4 38/40 20/20 18/20 1.75 396 1.46
The 0 of 20 for everyday wording is built in: those questions avoid the page's words, and BM25 only matches words, so the useful number is how many of them the loop recovers. Technical questions all hit on the first search and never paid for a retry.
The cap-4 run, round by round:
| Round | Questions that ran it | Found | Estimated tokens | Modeled time added per question |
|---|---|---|---|---|
| 1 (first search + grade) | 40 | 20 | 7,081 | 0.59 s |
| 2 (retry 1) | 20 | 12 | 5,740 | 1.15 s |
| 3 (retry 2) | 8 | 6 | 2,429 | 1.19 s |
| 4 (retry 3) | 2 | 0 | 593 | 1.14 s |
What each extra round costs
Each retry round pays for a rewrite, a search and a grade, and two of those are model calls.
- Round 2 cost 478 retry tokens and 1.92 modeled seconds per question it rescued.
- Rounds 3 and 4 together cost 504 tokens and 1.97 modeled seconds per rescue. Round 3 alone was cheaper per find (405 tokens); round 4 spent 593 tokens and found nothing, which pulls the average up.
- The whole cap-4 loop used 15,843 tokens, and 7,081 of them graded the first search of all 40 questions, including the 20 that were already right.
- 13% of the cap-4 loop tokens went to the 2 questions that were never found.
If most of your traffic hits on the first try, that first grade can cost more than the rescues are worth.
In Anthropic's data, agents used about 4 times the tokens of chat interactions and multi-agent systems about 15 times. Those figures describe whole agent systems, not this loop. We covered why each agent turn resends the conversation in why an AI agent re-sends its history.
How many retries should an agent make?
Set the cap from your logs. Here the third search still found 6 of the 8 questions left; the fourth found 0 of 2. Those two were never in reach of the rewriter:
- "Everything slows to a crawl right at the top of every hour" (a cache stampede page). The last query tried was
everything crawl right top hour: the word table had no useful entry. - "The server gets slower every day until we reboot it" (a memory leak page). The rewrite drifted to
query slower reboot orm eager join, a database topic.
The cap-4 run also sent 2 repeated queries. With this fixed index and deterministic retriever, a repeated query costs a full round and cannot return anything new, so a cheap guard is to stop when the rewriter produces a query it already tried.
Anthropic's post also put an effort rule into its agents' prompts: simple fact-finding needs one agent with 3 to 10 tool calls. This is their task-allocation heuristic; tool calls do not map directly to this lab's search rounds.
What we saw when we ran it
We ran node labs/ai-33/measure-agentic-search-retries.mjs on 2026-11-15, and it printed one table with three settings. A single search found 20 of 40 answers, and it found the answer page for none of the 20 everyday-wording questions. With a cap of 4 rounds the agent found 38 of 40, including 18 of the 20 everyday questions.
Our terminal after the ai-33 run, with the three settings and the round-by-round costs.
Round 2 rescued 12 questions for 5,740 approximate retry tokens, about 478 per rescue. Rounds 3 and 4 together rescued 6 questions for 3,022 retry tokens, an average of about 504 per rescue, and that average includes round 4, which ran for 2 questions and rescued none. Round 3 on its own worked out to about 405 tokens per rescue. The extra rounds were wasted on the 2 questions that were never found. The memory-leak rewrite drifted into database terms ("query slower reboot orm eager join"), and the cache-stampede rewrite only dropped words ("everything crawl right top hour") without adding a useful term. Raising the cap from 2 to 4 bought 6 more answers in this lab, and both misses came from the rewriter, so that is the part to watch.
The official page below confirms that Anthropic's research subagents evaluate tool results, identify gaps and refine their next query.
Anthropic Engineering's post on how it built its multi-agent research system.
The loop in code
This version takes your own search, grade and rewrite and adds a repeated-query stop (the lab logged repeats and kept going, so its totals include them). Real grade and rewrite calls are model calls that return promises, so both are awaited. Run it with node agentic-search-loop.mjs.
// agentic-search-loop.mjs: the loop shape from labs/ai-33, Node 22, no packages
const words = (s) => s.toLowerCase().split(/[^a-z0-9]+/).filter(Boolean);
export async function searchWithRetries(question, { search, grade, rewrite, cap = 3 }) {
const tried = new Set();
let query = question;
for (let round = 1; round <= cap; round++) {
const key = words(query).join(' ');
if (tried.has(key)) return { found: false, reason: 'repeated query', rounds: round - 1 };
tried.add(key);
const hits = await search(query);
if (await grade(question, hits)) return { found: true, rounds: round, hits };
if (round < cap) query = await rewrite(question, round + 1);
}
return { found: false, reason: 'cap reached', rounds: cap };
}
// demo: two pages, a keyword search and a one-entry synonym table
const pages = [
{ title: 'Session cookie expiry', text: 'session cookie expiry idle timeout' },
{ title: 'Password reset', text: 'password reset email link' },
];
const SYN = { forgetting: ['session', 'cookie', 'expiry'] };
const search = async (q) => pages.filter((p) => words(q).some((w) => words(p.text).includes(w)));
const grade = async (_question, hits) => hits.some((p) => p.title === 'Session cookie expiry');
const rewrite = async (question, round) =>
round >= 2 ? [...words(question), ...words(question).flatMap((w) => SYN[w] || [])].join(' ') : question;
console.log(await searchWithRetries('Why does the site keep forgetting me?', { search, grade, rewrite }));
To run the full lab, use node measure-agentic-search-retries.mjs in labs/ai-33 (Node 22, no install, no network). It also prints every question that was never found.
What this simulation does not show
- The grader is an answer key that is always right. A real agent uses a model as the grader, and a model can be wrong: a false "no" spends extra rounds, and a false "yes" stops early with the wrong pages.
- The rewriter is a fixed word table. A model could reach terms the table lacks, or drift where the table does not.
- Modeled latency assumes 400 ms per model call, 0.05 ms per input token, 15 ms per output token and 30 ms per search. Token estimates round each prompt and reply up from characters divided by four.
- The 200 pages and 40 questions are made up, with a 50/50 wording split we chose. Dense or hybrid retrieval would miss different questions.
Our take
In this run, query expansion recovered vocabulary misses: everyday questions went from 0 of 20 to 18 of 20. Test the loop on representative questions before adopting it, and choose a cap from retrieval gains and cost on held-out traffic. We did not compare against applying the same expansion before the first search, so this experiment does not isolate the benefit of adaptive grading.
This article describes general patterns with a small simulation. It does not describe any company's private systems or measure any AI product. Numbers come from one run on 2026-11-15 with an answer-key grader and a fixed rewrite table.
Frequently asked questions
What is agentic RAG?
Agentic RAG lets an agent decide when and how to retrieve information while preparing an answer. One common design grades the retrieved results, rewrites a failed query and retries within a cap.
What is the difference between agentic RAG and corrective RAG?
Corrective RAG (CRAG, Yan et al., 2024) is one published design: a lightweight evaluator scores the retrieved documents and triggers different retrieval actions. Agentic RAG is the wider pattern of an agent deciding when and how to search again.
How many search retries should an AI agent make?
Set a cap and measure it. In our simulation, the third search still found 6 of 8 remaining answers and the fourth found 0 of 2. Anthropic's engineering post says simple fact-finding needs one agent with 3 to 10 tool calls.
Sources
- Anthropic Engineering: How we built our multi-agent research system (Jun 13, 2025)anthropic.com
- Yan et al.: Corrective Retrieval Augmented Generation (arXiv 2401.15884, 2024)arxiv.org
- Asai et al.: Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (arXiv 2310.11511, 2023)arxiv.org
- Our labs repository (lab ai-33)github.com