# How recommendation systems work: retrieval and ranking

URL: https://systemdesignsimple.com/how-ai-picks-what-to-watch/
Published: 2026-11-12
Category: System Design
Publisher: System Design Simple (https://systemdesignsimple.com)

## Summary

Recommendation systems work by retrieving a manageable candidate set and ranking it for the current user. A cheap retrieval step compares a user embedding with precomputed title embeddings through an approximate index and keeps a few hundred candidates; a heavier model ranks only those. Titles retrieval drops never reach the ranker, so measure retrieval coverage separately from ranking quality.

## Key facts

- **Two stages:** Covington, Adams and Sargin (RecSys 2016) describe YouTube recommendations as a candidate generation model followed by a separate ranking model
- **Why retrieval is cheap:** In a two-tower retriever, the item tower uses only item-side features, so title embeddings can be precomputed and searched with an approximate nearest neighbour index
- **Our lab number:** In our synthetic HNSW lab published 2026-11-05, searches at efSearch 320 touched an average of 776 of 20,000 vectors across 200 queries, with mean recall@10 of 94.4%
- **Assumed arithmetic:** With assumed costs, ranking 1,000,000 titles at 8,000 multiply-adds each is 8 billion per request; a 64-number flat scan plus 500 ranked titles is 68 million, about 118 times less
- **JioStar write-up:** According to OpenAI's write-up published 2026-11-11, JioStar built a voice and text discovery assistant in about eight weeks, in more than 15 languages

Recommendation systems work by retrieving a manageable candidate set and ranking it for the current user. A cheap retrieval step narrows a large catalog to a few hundred titles, a heavier model ranks only those, and caching keeps the cost per user low.

## What do you need to know first? {#before}

You need the idea of a [vector embedding](/why-search-for-car-finds-vehicle/). Our design question: a video app's home screen shows rows of titles from a large catalog, fast, to millions of people. Catalog sizes and costs below are our assumptions, not facts about any company.

> Definition: An embedding is a list of numbers a model produces for a user or a title, placed so that related things score high together.

> Definition: Retrieval, also called candidate generation, is the first stage: it returns a few hundred probably relevant titles from the whole catalog using a score cheap enough to search at that scale.

> Definition: Ranking is the second stage: a heavier model scores each candidate for this user and sets the final order.

> Definition: A two-tower model is a retriever with one network for the user and one for the title, scored by the dot product of their embeddings.

> Definition: K is the number of candidates passed from retrieval to ranking.

## How do recommendation systems work? {#how-it-works}

They score cheaply first and expensively second. Covington, Adams and Sargin's [paper on YouTube recommendations](https://research.google/pubs/deep-neural-networks-for-youtube-recommendations/) (RecSys 2016) describes this split as a candidate generation model followed by a separate ranking model.

### Stage 1: retrieval, mostly built offline

In this two-tower design, the item tower uses item-side features independently of the requesting user. Precompute item embeddings, then refresh them when those features or the item model change. Store them in an approximate nearest neighbour (ANN) index, which finds close vectors without comparing the query with every stored vector and sometimes misses a true match. At request time you run the user tower once, from watch history, language and context, and ask the index for the nearest titles.

HNSW (hierarchical navigable small world) is a common ANN index built as a layered graph. efSearch controls the search breadth in this HNSW implementation. ANN recall@10 is the fraction of the exact ten nearest vectors returned by approximate search. In [our HNSW vector search lab](/how-search-skips-96-percent-of-vectors/) published 2026-11-05, searches at efSearch 320 touched an average of 776 distinct vectors out of 20,000 across 200 queries, with mean recall@10 of 94.4%. So in that test about 5.6% of exact top-ten matches were missed, using the rounded recall. The data was synthetic and small, so it shows the shape of the effect and cannot forecast results for a million-title catalog.

Large systems often merge several retrievers, such as similar to what you watched, popular in your language and new episodes of shows you follow.

### Stage 2: ranking, a richer joint model

The [TensorFlow Recommenders documentation](https://www.tensorflow.org/recommenders/api_docs/python/tfrs/tasks/Retrieval) calls the two-tower score factorized retrieval. The embeddings can still encode genre, language and taste, but the factorized form limits how freely user and title features interact in the score. A ranker combines them directly, for example how often you finish this genre at this hour. Features of the current user and title pair need online computation or lookup, which can use a feature store.

### Caching

Title embeddings and the index are built offline and shared by every request. Cache user embeddings only while their input features remain valid. Candidate lists or finished rows can be cached too; the same reuse idea for AI answers is in [semantic caching](/two-questions-look-alike-can-ai-reuse-the-answer/). Key cached rows by user or segment and relevant context, and version the model and index together. Expire or invalidate rows when session activity or catalog availability changes.

## What does two-stage ranking save? {#example}

Every value below is assumed. The count covers scoring only; tower inference, feature lookups, index traversal and cache overhead are left out.

```text
catalog                    1,000,000 titles
ranking model              8,000 multiply-adds per title
retrieval score            dot product of 64 numbers = 64 multiply-adds per title

rank everything            1,000,000 × 8,000            = 8,000,000,000 per request
two stages, flat scan      1,000,000 × 64 = 64,000,000
                           + 500 × 8,000  =  4,000,000  =    68,000,000 per request
ratio                      8,000,000,000 / 68,000,000   ≈ 118 times less work
```

The ranker's 4 million is set by K and does not grow with the catalog. With an ANN index instead of a flat scan, retrieval cost depends on index settings, data and the recall you need; the [Faiss wiki](https://github.com/facebookresearch/faiss/wiki/Faiss-indexes) lists exhaustive and approximate index types for that choice. To reproduce our lab number, run `node run.mjs` in [`labs/ai-12`](https://github.com/systemdesignsimple/labs/tree/main/labs/ai-12) (Node.js 22, no packages); it prints vectors touched and recall@10 for each efSearch from 10 to 320.

## What does JioStar's write-up report? {#source}

According to [OpenAI's write-up](https://openai.com/index/jiostar-vijay-seshadri/) published 2026-11-11, JioStar's chief architect, Vijay Seshadri, led a voice and text assistant for JioHotstar that answers questions about movies, shows and live cricket, built in about eight weeks in more than 15 languages. The write-up says the team decided early how to balance accuracy, latency and cost, and built sports statistics tests with sports magazine editors. It does not describe retrieval, ranking or caching, so the stages above describe the general pattern only. The 500 million in the headline is the platforms' audience, not the assistant's users.

## How does candidate recall limit recommendations? {#lesson}

The ranker can only order titles that retrieval kept, so the final list is capped by the candidate set. Metrics computed only within the retrieved candidates can hide retrieval misses. Measure retrieval coverage separately, alongside end-to-end quality from held-out viewing and live experiments.

One offline check we find practical is recall of the full-catalog ranker's top results:

1. For a sample of users, score the whole catalog with the ranker once (slow, fine for a test) and keep its top 10.
2. Count how many of those 10 appear in retrieval's K candidates.
3. Raise K until the count levels off, then weigh the last steps against ranker cost, which grows with K.

This measures agreement with your own ranker, not viewer satisfaction. It is also a different number from ANN recall, which compares the index with exact vector search. You need both: the index can lose titles the retriever would have chosen, and the retriever can lose titles the ranker would have chosen. The same fan-out cost question for a social feed is in [design Instagram's feed](/design-instagram-feed/).

## What can go wrong with retrieval and ranking? {#trade-offs}

Most failures come from retrieval dropping titles or from stale caches, while ranking metrics look fine.

| Option | Good for | Costs you | Use when |
|---|---|---|---|
| Rank everything | No retrieval misses | 8 billion per request in our assumed example | Catalogs of a few thousand items |
| Retrieve, then rank | Ranker cost fixed by K | Quality capped by retrieval coverage | Large catalogs, tight latency |
| Larger K | Better coverage | Ranker cost grows linearly with K | Coverage still rising |
| Cached rows | Lowest cost per request | Staler results, invalidation | Anonymous or slow-changing segments |
| Fresh per request | Reacts to live events | Full pipeline cost every time | Live sports, a session that just changed |

A retriever that relies on learned item IDs may struggle with new titles that have little interaction data. Use metadata features or a fresh-content candidate source, and measure retrieval coverage separately for new titles. An ANN index rebuilt with lower settings can lose recall while ranking metrics stay flat, and rows cached under an old model version keep serving after a deploy.

## Our take {#our-take}

Two stages is the sensible default once a catalog is too large to score per request, and retrieval coverage belongs on the same dashboard as ranking quality. We could not test the assistant in the write-up, which gives no latency or recall figures, so the numbers here are our lab and labelled arithmetic only. For more depth, read the [HNSW paper](https://arxiv.org/abs/1603.09320) on the layered graph search.

## Practice {#practice}

**Q (Intermediate): The ranker's offline metrics improved, but people watch less. What do you check first?**
Retrieval coverage. Metrics computed only within the retrieved candidates can hide titles retrieval dropped, so check recall of the full-catalog ranker's top results, ANN recall and end-to-end held-out metrics.

**Q (Advanced): A cricket match goes to a super over. What happens to cached rows, and how would you design for it?**
Cached rows go stale while the live moment matters most. Keep live content in a separate fresh retriever, or invalidate by event, with short expiry for segments that follow the match.

This article explains a general design pattern with assumed numbers and one result from our own synthetic lab. Facts about JioStar come only from OpenAI's write-up of 2026-11-11, which does not describe retrieval, ranking or caching systems.

## Frequently asked questions

**What is the difference between retrieval and ranking in a recommendation system?**
Retrieval picks a few hundred plausible titles out of a large catalog with a score that can be indexed, usually a dot product of separately computed embeddings. Ranking runs a richer joint model over just those candidates, combining user and title features directly, and sets the final order.

**Why not run the ranking model on every title?**
Because its cost grows with the catalog. With an assumed 8,000 multiply-adds per title, one request over 1,000,000 titles costs 8 billion; ranking 500 retrieved titles costs 4 million, plus whatever retrieval costs.

**What should a recommendation system cache?**
Title embeddings and the index, built offline; user embeddings only while their input features remain valid; and candidate lists or finished rows keyed by user or segment and relevant context. Version the model and index together, and expire or invalidate rows when session activity or catalog availability changes.

## Sources

- [OpenAI: JioStar's chief architect on AI discovery at 500M-user scale (2026-11-11)](https://openai.com/index/jiostar-vijay-seshadri/)
- [Covington, Adams and Sargin, Deep Neural Networks for YouTube Recommendations (RecSys 2016)](https://research.google/pubs/deep-neural-networks-for-youtube-recommendations/)
- [TensorFlow Recommenders: tfrs.tasks.Retrieval (factorized retrieval)](https://www.tensorflow.org/recommenders/api_docs/python/tfrs/tasks/Retrieval)
- [Malkov and Yashunin, Efficient and robust approximate nearest neighbor search using HNSW graphs (arXiv 1603.09320)](https://arxiv.org/abs/1603.09320)
- [Faiss wiki: Faiss indexes](https://github.com/facebookresearch/faiss/wiki/Faiss-indexes)