Reranking

Compass

A second pass that reorders retrieved passages by relevance, which is usually what separates a demonstration from production.

What is reranking?

Reranking is a second pass that reorders the passages an initial retrieval returned. The first pass favours speed across the whole corpus, where the second examines each candidate passage against the question in detail. Only the reordered top passages are passed to the model, so reranking decides what the answer is actually written from.

Why the first pass retrieves and the second decides

A retrieval system that returns the correct passage in eighth place has already found the answer and will then fail to use it. The model attends to what arrives first, and a fluent yet irrelevant passage at the top displaces the one that mattered. Reranking is the gap most often mistaken for a model problem. It is also why a demonstration over a few hundred documents behaves nothing like a deployment over an enterprise corpus.

  • The generation step treats the first passages as the answer, whatever their rank deserved.
  • A cross-encoder reads the question and the passage together, which an embedding cannot.
  • Reranking recovers the accuracy a fast first pass trades away for coverage.
  • It is the cheapest accuracy improvement available once an index already exists.

How reranking works

  • The first pass retrieves a generous candidate set with an approximate method.
  • Each candidate is paired with the question and scored by a cross-encoder.
  • Candidates are reordered by that score rather than by their original distance.
  • The highest-scoring passages within the context budget are selected.
  • Scores below a floor are discarded instead of padding the context.
  • Both the original rank and the reranked position are retained for diagnosis.

How reranking is evaluated

Reranking is evaluated by comparing the ordering before and after it, on the same candidate set. Measure the following per question type:

  • mean reciprocal rank before and after reranking
  • change in the position of the known correct passage
  • proportion of questions where reranking moved a wrong passage to the top
  • latency added per question by the second pass
  • candidate set size needed before accuracy stops improving

Limitations, risks and common failure modes

  • A candidate set too small means reranking has nothing correct to promote.
  • Cross-encoder latency makes the answer arrive too late to be used.
  • Reranking is tuned on an evaluation set whose questions resemble no real query.
  • A reranker trained on general web text misjudges internal document conventions.

Related terms

ChunkingHow a long document is divided before indexing, which quietly determines whether retrieval ever finds the right passage.Query rewritingReformulating what a person asked into what the index can actually answer.Retrieval augmented generation (RAG)Answering from documents retrieved at the moment of the question, so the answer reflects your material rather than the model's memory.Semantic searchFinding material by what it means rather than by the words it happens to contain.Vector databaseA store that holds numerical representations of text and returns the passages closest in meaning to a query.Context engineeringDeciding what a model should have in front of it for a given task, which has largely displaced prompt wording as the discipline that matters.

All 315 terms