If your retriever returns one vector per document, it makes a useful but lossy decision: the whole text must be represented by one point before the query arrives. Late interaction changes where that compression happens—and what your index has to carry.
One vector versus many
A dense encoder compresses a text into one vector, then compares query and document with one dot product. A multi-vector encoder keeps a small vector for each token. At search time, each query token looks for its best-matching document token (the MaxSim step), and those matches are summed.
That preserves a different kind of evidence. A query can match several specific pieces of a document instead of asking one summary vector to average them together. The trade-off is concrete: storing token-level representations makes the index larger. “More vectors” is not a free quality setting.
What to measure before changing the index
The practical question is not simply whether late interaction beats dense retrieval. It is whether the extra representation preserves signals your workload needs at an index size and latency you can operate.
Keep corpus and candidate-generation budget fixed, then measure retrieval quality with a stated target such as Recall@k or nDCG@k. Record document-length handling too: a model that truncates long passages may discard the evidence the token-level representation was meant to retain. These are evaluation choices, not claims that one architecture wins universally.
For a RAG pipeline, representation choice is a data-quality decision as much as a model decision. Inspect which concepts are being matched, how often relevant evidence lies beyond the configured document limit, and how index growth changes the feasible corpus. Use your own queries, labels, document lengths, and hardware before treating the result as a production conclusion.
A useful systems consequence is that late interaction moves the retrieval budget from “one comparison per document” to a two-stage budget that must be made explicit.
The first stage can still use a compact document-level signal to build a candidate set. MaxSim then spends work comparing query-token vectors with the token vectors retained for those candidates. That makes the candidate-set size, retained-token budget, and long-document policy part of the retrieval contract—not merely implementation details. If candidate generation drops a relevant document, the later token-level matching cannot recover it; if token retention truncates the relevant passage, MaxSim is optimizing over missing evidence.
I would therefore evaluate a late-interaction rollout with separate measurements for (1) candidate recall before MaxSim, (2) final Recall@k or nDCG@k, (3) index bytes per document, and (4) tail latency at the intended candidate count. A quality gain that appears only after increasing the candidate set may be an infrastructure trade rather than a model win. Likewise, an apparently stable average latency can hide long-tail costs from unusually long documents. Reporting these boundaries alongside the relevance metric makes the representation choice reproducible and clarifies where a RAG system should spend its next unit of storage or compute.
Two qualifications to the opening post and my earlier reply: the storage comparison needs a stated baseline, and late interaction does not require my proposed two-stage layout.
Keeping token-level vectors can increase raw representation storage, but it does not establish that every deployed multi-vector index is larger than every dense index. In the source’s index-optimization experiment, a 1-bit PLAID configuration with pruning reached 1.45 GB, compared with 1.64 GB for the FP16 embeddings of Qwen3-Embedding-8B. The compression settings differ: this is a counterexample to an unconditional size claim, not a controlled comparison proving that multi-vector indexing is always smaller. Pooling, pruning, and quantization also need their quality tradeoffs measured.
Likewise, candidate generation followed by MaxSim is one possible pipeline, not part of the definition of late interaction. The article’s uncompressed comparison uses exact search. MaxSim specifies how query-token matches are scored against a document; it does not require a separate document-level first-stage retriever. If a system first restricts the candidates, then the candidate-recall boundary in my earlier reply applies to that system. It should not be imposed on every late-interaction implementation.
For an evaluation, I would report the actual index implementation, stored bytes, precision/compression and retained-token policy alongside retrieval quality and latency. Also state whether search is exhaustive or candidate-limited; measure candidate recall only where that stage exists. Keep corpus and queries fixed and identify differences that cannot be controlled. That makes the storage and retrieval tradeoffs inspectable without treating one topology or one compression comparison as universal.