Semantic cache: can AI safely reuse a similar answer?

A semantic cache saves an AI answer and hands it out again when a new question looks close enough to an old one. In our lexical simulation of a semantic cache, "How many calories in an apple pie?" scored 0.88 against a cached question about an apple, and every cutoff we tested still reused some wrong answers.

How does a semantic cache decide to reuse an answer?

A normal cache, like the cache-aside pattern in our free course's caching module, needs the exact key. A semantic cache relaxes that. It turns the question into a vector, usually with a sentence embedding model such as Sentence-BERT, and treats nearby vectors as the same question. The GPTCache documentation describes the parts of one open-source semantic cache: an embedding function, scalar storage for questions and answers, a vector store for similarity search, and a similarity evaluation that decides whether a candidate is good enough. Search uses the same nearest-neighbour idea, as in why a search for car finds vehicle and how search skips most vectors. When the nearest stored question is the wrong one, the cache returns an answer intended for another request.

In our lab one lookup took a median of about 11 ms on one Mac: build the vector, then scan 10,060 rows linearly (the 60 cached questions plus 10,000 synthetic random rows added only to time the scan). A real system would use a vector index instead of a linear scan.

What our lab measured

We wrote 60 question pairs. In each pair, question A is in the cache and question B is new. 30 pairs are real paraphrases ("How do I reset my password?" and "I forgot my password, what now?"). 30 look alike but need a different answer: today and tomorrow, Austria and Australia, enable and disable, apple and apple pie.

Our lab rules allow only the Node.js standard library, so no neural embedding model ran. The vector is lexical: hashed words plus character trigrams, with a 35-word synonym table we wrote by hand and stop words removed. The numbers describe a word-overlap cache on these pairs; a neural embedding model would score them differently.

Each new question is compared with all 60 cached questions (and the synthetic rows). A reuse counts as correct only when the nearest row is that question's own paraphrase partner. Each reuse stands for a model call the cache would have avoided; no model was called.

Cutoff	Reused (simulated avoided calls)	Wrong reuses	Notes
0.80	30 of 60 (50%)	6 of 30 (20%)	apple pie, water boiling on Everest, last 10 vs last 5 orders, 3 reversed pairs
0.90	18 (30%)	3	only the reversed pairs
0.96	8 (13.3%)	3	only the reversed pairs
0.90 with checks	17 (28.3%)	2	both reversed pairs

To reproduce, run node labs/ai-20/run.mjs from the repository root. It prints the whole sweep, the most dangerous look-alikes and the paraphrases it missed; our run is saved in labs/ai-20/results.json.

Why every tested cutoff made errors in our lexical lab

The run prints the look-alikes with the highest scores next to the real paraphrases with the lowest:

look-alikes (wrong if reused)                  paraphrases (right if reused)
1.000  Convert 5 kilometers to miles           0.709  How to capture the screen on a Mac?
0.881  How many calories in an apple pie?      0.715  How can I save a Word file as PDF?
0.859  Boiling point of water on Everest?      0.719  How fast does light travel?
0.819  Show my last 10 orders                  0.720  What is a 404 error?

The two distributions overlap. Any cutoff low enough to catch "How fast does light travel?" (a paraphrase of "What is the speed of light?") also lets the apple pie through. Raising the cutoff cut hits faster than errors: from 0.80 to 0.96, hits fell from 30 to 8 while wrong reuses only fell from 6 to 3, so the share of reuses that were wrong rose from 20% to 37.5%.

Three pairs never separate. "Celsius to Fahrenheit" and "Fahrenheit to Celsius" contain the same words, and our vector ignores word order, so reversed conversions get identical vectors and score 1.000, which no threshold can separate. We did not test neural embeddings, so we make no claim about how they score these pairs.

Which checks help, and what slips through

We added two simple guards before reuse:

// guard.js · simplified from labs/ai-20/run.mjs
const TIME = /\b(today|tomorrow|tonight|now|yesterday)\b/i;
const PERSONAL = /\b(my (order|orders|refund|home|last|balance)|near me)\b/i;
// key tokens: anything with a digit, and capitalised words that are not just the first word
function guardAllows(newQ, cachedQ) {
  if (TIME.test(newQ) || PERSONAL.test(newQ)) return false;   // never serve these from cache
  return sameSet(keyTokens(newQ), keyTokens(cachedQ));        // 10 vs 5, B12 vs D, Everest vs nothing
}

At 0.90 the guards removed exactly one reuse, and it was a wrong one ("Directions to my home from the airport", caught as personal). Across the sweep they removed most of the wrong reuses at loose cutoffs, but the reversed conversions have the same numbers and names, so they pass every guard. The capital-letter rule is only a rough heuristic for names; "store open" and "store close" share every key token too. A verifier model that asks "do these two questions want the same answer?" may reject additional mismatches. It adds inference cost and can make errors of its own; we did not run one.

What must match besides the question

Identical text can need different answers for different users, conversations, permissions or document versions. Filter candidates by tenant and permissions before similarity search. Reuse only within matching conversation context, instructions, model configuration and source-data versions, and invalidate entries when those dependencies change. Our lab compared question text only, so it tests none of this.

Trade-offs

Choice	Good for	Costs you	Use when
Exact-match cache	avoids similarity-based mismatches when the complete request context matches	misses every rewording	repeated identical prompts, templates, retries
Semantic cache, low cutoff	many avoided calls	look-alikes get the wrong answer	answers are generic and a wrong one is cheap
Semantic cache, high cutoff plus guards	fewer wrong reuses	far fewer hits; reversed or negated pairs still slip	FAQ-style traffic with stable answers
Reuse with a verifier	may reject mismatches a score misses	an extra model call per hit, with its own errors	a wrong answer costs more than a model call

Our take

A semantic cache pays off when many people ask the same stable question in different words. Personal or changing answers can be reused only inside a tight scope (same user or tenant, same permissions, same source-data version) with an expiry that matches how fast the answer changes. In our lexical lab, reversed conversions got identical vectors that a threshold cannot separate; we did not test neural embeddings. If you deploy a semantic cache, build a labelled set of your own look-alikes first, report wrong reuses next to the hit rate, and log every reuse so you can check the answers later. Batching and queues, as in when 1,000 people ask an AI at once, address load without reusing another question's answer.

The lab is a lexical simulation on one machine with 60 question pairs we wrote; no AI model and no neural embedding model ran, and the cache stores question vectors, not real answers. It describes general patterns from public projects and papers, not any company's private system.