Check approximate neighbor recovery against an exact reference

Separate candidate recovery from the comparison rule

Faiss documents inverted-file search as visiting selected lists; it can miss the nearest neighbor when that neighbor’s cell is unvisited.

Write down the query set, candidate snapshot and comparison rule before evaluating a change. Preserve the same inputs across the alternatives. If a team changes the representations at the same time, a result difference cannot be attributed solely to the search procedure. Obtain the exact configuration and the reason the owner intends to compare it.

Ask the receiving owner what the proposed test would establish. Recovering the nearest vector under a chosen rule is not the same question as returning the most useful article to a reader. Keep those reviews separate so a technical diagnostic does not become an unsupported claim about customer understanding or content quality.

Use a small synthetic example to make the search boundary visible before requesting a production evaluation. A result list by itself does not identify which alternatives were examined. Retaining that evidence helps the reviewer distinguish a failure to consider a candidate from an incorrect calculation applied to a candidate that was considered.

Make the missed candidate inspectable

Use scalar vectors A = −0.01, B = −3.99, C = 1 and D = 3. Cells centered at −2 and 2 contain AB and CD respectively. Each vector belongs to its nearest center.

For query 0.1, the nearer center is 2. Searching only CD returns C at distance 0.9, missing A at distance 0.11. For query 2.2, that same cell returns D at distance 0.8, which is also the exact nearest result.

Keep the cell assignments beside the original values. A reviewer should be able to reconstruct the complete search without access to a commercial index or private embedding corpus. In this example the cell centers are also the means of their assigned values; the grouping is deliberately explicit rather than being inferred from a screenshot.

The local Python execution compares the restricted candidate search with an exhaustive reference. An independent rational-distance calculation checks both query outcomes and the cell assignments. No Faiss or Qdrant service was run. This is a transparent teaching model of candidate restriction, not a claim that a particular deployed index has reproduced these results.

Measure recovery without turning it into a speed claim

Qdrant defines recall@k by overlap with the exact top-k set. Here k = 1: one-cell search recovers one of two exact answers, averaging 0.5. Searching both cells recovers both, averaging 1.0. Candidate distance comparisons rise from two to four per query; elapsed time was not measured.

Preserve the per-query record as well as the average. The aggregate score would otherwise hide the boundary query that revealed the problem. A production evaluation needs a justified query set, fixed candidate identities and an appropriate definition of the reference results. This two-query teaching example is not a representative estimate of any service’s behavior.

If the receiving evaluation changes filters or candidate eligibility, reconcile those changes before comparing recovery. A missing candidate might have been deliberately excluded for a valid product reason. The exact and restricted calculations must be answering the same permitted retrieval question, rather than being judged against different universes without explanation.

Ask the implementation owner for separate timing and resource evidence if the business decision concerns performance. A count of examined candidates can explain the model, but it does not measure real latency, memory use or operating cost. Keep the conditions of any eventual benchmark beside its results instead of extrapolating a production speedup from this tiny calculation.

Turn the discrepancy into a bounded acceptance decision

Prepare a handoff containing the fixed inputs, reference answers, visited candidates, returned identities and the specific missed case. Include the settings used for each comparison and the exact artifact versions. A statement that search accuracy improved is too broad unless the audience knows which quantity was checked and under what conditions.

Have the responsible owner decide what recovery evidence is sufficient for the intended use. Do not invent a universal threshold from this example, and do not assume that maximizing a single technical score settles the product tradeoff. The owner may need to consider supported performance requirements and separate content judgments before choosing a configuration.

Retain the boundary case after a justified change so a later regression can be explained. If the underlying representations or catalog change, create an updated reference rather than treating the old expected answer as permanently authoritative. The record should distinguish a changed decision problem from a defect in executing the current one.

Close the exercise with the supported finding and the untested production questions. No live collection was rebuilt, no setting was changed and no customer query was sent to an external service. The result demonstrates why a plausible neighbor is insufficient evidence that the intended candidate was recovered.

Make the reasoning concrete.

A fictional content library receives a plausible search result after a configuration change. The team wants to know whether the limited search recovered the same candidate as a complete comparison.

The team requests the bounded candidate-recovery record before considering a configuration change. It makes no quality announcement or production modification.

Take the questions into your next brief.

Use these prompts in your own notes. Nothing is submitted from this page.

1. Inputs

   Which query set, candidate snapshot and metric stay fixed?

2. Coverage

   Which candidates did each search actually examine?

3. Reference

   How was the exact comparison independently reconstructed?

4. Summary

   Which per-query miss is hidden by the average?

5. Decision

   What performance and usefulness evidence remains untested?

Watch for the wrong conclusion.

• Changing the candidate universe while comparing recovery scores.
• Accepting a plausible answer without an exact reference.
• Calling fewer comparisons a measured latency improvement.

Sources & scope

Original scalar vectors, local search model and review procedure. Named sources support the identified mechanisms and evaluation distinction. No production service, latency benchmark or semantic-quality study was executed.

• Faiss: Inverted-file indexes →Primary software documentation · Supports: Supports the candidate-restriction mechanism.
• Qdrant: Measuring ANN Recall →Primary software documentation · Supports: Supports exact-reference recovery measurement, separate from relevance.