Listwise reranking

Compass

Reordering retrieved passages by having a model compare the whole candidate set at once, instead of scoring each passage in isolation.

What is listwise reranking?

Listwise reranking reorders a set of retrieved passages by presenting them to a model together and asking for a single ranked order. Pointwise reranking scores each passage on its own, and pairwise reranking compares two at a time. The listwise approach lets the model see the candidates side by side, so it can prefer the passage that answers the question over others that merely mention the same subject.

Also called: Listwise rerank, LLM reranking

Why relevance is easier to judge in comparison

A passage that scores well in isolation may be outranked by another that answers the question more directly, and a model judging one passage at a time has no way of knowing that. Seeing the candidates together also exposes duplication, where several passages repeat the same point and crowd out a distinct one that mattered. Listwise reranking makes these comparisons explicit. The trade is a larger and slower model call, so it is usually applied to a short list that a cheaper method has already narrowed.

  • Relative judgements are more reliable than absolute scores for close candidates.
  • Near-duplicate passages can be recognised and demoted in a single pass.
  • The model can favour passages that answer the question over ones that discuss it.
  • It works on top of any first-stage retrieval without retraining it.

How listwise reranking works

  • A first-stage retrieval and fusion step produces a short list of candidates.
  • The candidates are presented to a model with the question and an instruction.
  • The model returns an ordered list of passage identifiers.
  • Longer lists are processed in overlapping windows and the orders are merged.
  • Passages the model places below a set threshold are excluded from the answer context.
  • The original and reranked positions are both stored for later diagnosis.

How listwise reranking is evaluated

Listwise reranking is evaluated against the ordering it replaces, on the same candidate sets. Measure the following per question type:

  • mean reciprocal rank before and after the listwise pass
  • position of the known correct passage in each ordering
  • consistency of the ordering when candidates are presented in a different sequence
  • latency and cost added per question
  • candidate list length beyond which accuracy stops improving

Limitations, risks and common failure modes

  • Results change with the order in which candidates were presented.
  • Long candidate lists exceed the context the model can attend to reliably.
  • Identifiers in the response do not match any passage that was supplied.
  • Latency makes it unsuitable for interactive use without a short list first.

Related terms

RerankingA second pass that reorders retrieved passages by relevance, which is usually what separates a demonstration from production.Reciprocal rank fusion (RRF)Merging the ranked results of several retrieval methods by their positions in each list, so methods with incomparable scores can be combined fairly.Order effectThe influence of position in the pile on the score a submission receives.Mean reciprocal rank (MRR)A measure that rewards finding the correct passage near the top rather than merely somewhere in the list.Context assemblySelecting and ordering the retrieved material a model reads before it answers, with every source labelled and within a fixed budget of tokens.Hybrid searchCombining keyword matching with meaning-based search, because each one fails on cases the other handles.

All 315 terms