Search and Retrieval

Information needs and retrieval stages

A corpus is the collection being searched. A query expresses an information need: what someone wants to find or understand. Relevance means usefulness for that need. The wording alone may omit important context, such as the desired revision or the requester’s access. NIST’s retrieval terminology separates needs, queries, documents, and judgments. [3]

A constructed documentation corpus contains procedure P, historical revision r1, current revision r2, an ERR-42 glossary, and an operators-only recovery guide. Mira can read shared documentation; Leon can also read the guide. Revision r2 specifies a 30-second retry wait, with an adjacent exception prohibiting retries during migration. Revision r1 specifies 10 seconds. [4][5]

Eligibility determines which records may participate; ranking expresses preferences among eligible records. A high score cannot compensate for a failed mandatory filter. Similarity measures resemblance under a representation; relevance concerns the need. Factual correctness concerns truth, while authority concerns the source’s standing. Even a passage supporting a statement can itself be wrong. [9][10]

An index helps locate records. Candidate generation produces a shortlist, often with initial scores. A reranker scores that existing shortlist again before final selection. Retrieve-and-rerank bounds expensive comparisons; candidate depth and returned-result count are separate controls. [11]

A completed search can find nothing useful. If the corpus contains no instructions for ERR-900, its nearest passages still have an ordering. Returning those passages merely because they occupy the first positions confuses relative closeness with evidence that the information exists. [12]

Searchable records and retrieval units

A search document can be an article, procedure, or database record. A derived search record needs its own identity and a connection to its source revision. Generated chunk keys may change during updates; they should not substitute for durable source identity. [13]

A chunk is a selected portion of content. Smaller chunks can isolate a match but separate it from qualifications. Larger units preserve more surroundings while asking one representation to cover more subjects. Overlap repeats neighboring content; it preserves continuity at the cost of additional indexed text and potentially redundant candidates. [15]

The matching unit and the returned unit need not coincide. Passage A can locate procedure P-r2; a parent lookup can then recover its adjacent exception B. This retains specific matching without pretending A contains the entire procedure. Returning the parent increases the material transferred, so expansion should follow the task rather than happen indiscriminately. [7]

Table fragments need their column meanings. Repeating headers is one concrete preservation mechanism, implemented by Docling’s structural chunker. Document Understanding and OCR supplies the structured input; retrieval determines the independently searchable units. [18]

Repeated wording is not necessarily duplicate evidence. A changed number or negation can distinguish consequential revisions. Preserve identity and version before applying duplicate handling. Likewise, five retrieved passages from P-r2 remain five passages from one source revision, not five independent sources. [4][19]

Query analysis and matching conditions

A text analyzer identifies and transforms searchable terms. Normalization can change case or punctuation; stemming applies rules to word forms; stop-word removal drops listed terms. Lexical terms need not equal a language model’s subword tokens, whose purpose and boundaries belong to Tokenization. [20][21]

Document and query analysis must be compatible, not necessarily identical. A prefix-search index can store word prefixes while leaving the query intact. Expanding that query into prefixes too can introduce unintended shorter matches. Test the actual emitted terms for the intended matching behavior. [20]

Identifier fields preserve distinctions ordinary text analysis may split. Their configuration still matters: an indexing length limit can leave a source value unavailable for keyword lookup. Presence in the original document does not establish presence in every searchable field. [6]

Removing stop words can erase negation; applying the wrong language’s stemming rules can distort terms. Synonym mappings bridge vocabulary but can introduce ambiguous meanings. These transformations require task-specific tests, especially for short requests and identifiers. [24][25][26]

Query expansion adds terms to improve coverage. Pseudo relevance feedback uses leading results as presumed relevant material for reformulation. That assumption can cause query drift: early results about one subtopic redirect a broader search toward it. Preserve the original request when testing expansion, synonyms, or spelling corrections, and assess both recovered evidence and changed intent. [27]

Inverted indexes and lexical lookup

An inverted index maps terms to records containing them. A posting list stores occurrences for one term, including record identifiers and optionally frequencies and positions. Building these lists during ingestion lets queries locate candidates without repeatedly scanning all source text. [23]

AND intersects posting identities; OR unions them. The phrase retry after additionally requires consecutive positions in that order. Positions describe the analyzed term stream. Character offsets instead locate text for highlighting; they are not interchangeable. [23][28]

Lexical relevance and BM25

Term frequency counts occurrences within a record. Document frequency counts records containing the term. Inverse document frequency gives rarer terms greater discriminating weight. BM25 combines that weight with diminishing returns from repetition and a correction for document length. [29]

Rechunking changes record counts, term distribution, lengths, and average length, so it changes BM25’s inputs even without changing the source words. Tune parameters on development judgments. [1]

Field weighting can prefer title matches over body matches; a preference still differs from a required filter. Scores express ordering under a configured query and corpus, not probabilities. PostgreSQL’s text-ranking functions illustrate field weights and score transformations, but should not automatically be called BM25. Mapping a score into zero to one does not make it a percentage. [21]

Dense retrieval and representation compatibility

An embedding is a vector representation; an encoder produces it. Dense retrieval compares a query vector with stored passage vectors. A bi-encoder encodes the two inputs separately, allowing passage computation before requests arrive. DPR demonstrates this with separately encoded questions and passages scored by dot product. [30]

The embedding contract binds compatible query and document encoders, preprocessing, instructions, normalization, dimensions, and scoring. Equal dimensions alone are insufficient. Larger cosine or dot-product scores and smaller distances indicate closer matches under their respective rules; comparison scores explains the distinction. [30]

A request about time between retries may match a passage describing retry intervals without sharing those words. Exact identifiers need a separate test. Also test 10 versus 30 seconds and retry versus do not retry as controlled semantic distinctions. These are requirements for the documentation task, not reported failures of every embedding model. [31]

Nearby vectors can still represent weak evidence. The nearest available result supplies an ordering, not a guarantee that an answer-bearing passage exists. Evaluate the actual corpus and query distribution rather than interpreting similarity as answer confidence. [32]

Learned sparse retrieval instead produces weighted vocabulary terms, including expansions beyond the original wording. Matching remains inspectable through those terms, but broad expansions can increase query work. Sparse learned representations are distinct from both BM25’s occurrence statistics and dense vectors. [33]

Exact search and approximate vector indexes

Nearest neighbors are the best-scoring stored vectors under a chosen comparison. Exhaustive exact search scores every eligible vector. Approximate nearest-neighbor search, or ANN, avoids some comparisons and may miss exact neighbors. Its reference must use the same stored vectors, scoring rule, and eligible records. [34][2]

HNSW, hierarchical navigable small-world search, organizes vectors into proximity graphs. Sparse upper layers provide long-range navigation; lower layers explore more nearby candidates. More graph connections require storage and construction work. Broader candidate exploration generally spends more query work seeking better neighbor recovery. An omitted graph neighbor and a semantically irrelevant exact neighbor are different failures. [35]

IVF, an inverted-file vector index, assigns vectors to lists associated with representative centers. A query probes selected lists rather than every vector. More probes increase work. IVFFlat requires partition training; HNSW does not require that training step. Memory, construction, and update behavior remain implementation-dependent. [2]

Filtering a limited unfiltered shortlist can miss eligible neighbors outside it. Azure’s documented modes distinguish filtering during traversal, filtering per-shard candidates, and filtering the global top-k. Restrictive traversal filters may require additional exploration; post-filtering may return fewer results even when qualifying records exist. [36]

Compare index configurations at disclosed neighbor-recovery levels, not speed alone. Record search breadth, memory, build conditions, filtering, and update workload. Identical hardware or defaults can still favor one architecture. A faster configuration that omits more required neighbors answers a different engineering requirement. [37][38]

Hybrid candidates and rank fusion

Hybrid retrieval combines lexical and vector candidates for one need. In the constructed lists below, lexical retrieval finds A and C; vector retrieval finds B and A. Their identity-based union contains three records. Record D remains unavailable to fusion. [39]

The weighted column independently min-max normalizes each branch, assigns absent records zero, then combines 30% lexical with 70% vector. It ranks B above A, unlike RRF. These are example choices. Normalization bounds, score direction, missing-value policy, and weights all belong to the contract; equal extrema require an explicit rule. [40]

Result-derived normalization changes when retrieved extrema change. A shared numeric range does not establish calibration or relevance. Validate fusion on the intended queries. Candidate counts from each branch and the final returned count remain separate: additional lexical candidates can make a hybrid response larger than the vector branch’s requested count. [40][41]

Reranking and bounded scoring inputs

A cross-encoder processes the query and candidate together, allowing interactions absent from independent vector comparison. This richer scoring costs query-time computation for each pair, which motivates a bounded shortlist. Increasing candidate count or candidate length increases the work presented to the scorer. [42][43]

Learning to rank learns ordering from relevance examples. Training examples may describe individual items, preferred pairs, or judged lists. Serving can combine lexical, semantic, and field signals. Freshness should favor the revision appropriate to the task; a historical request should not automatically prefer the latest document. [44][45]

Published cross-encoder examples enable truncation and may expose logits as scores. Neither a retained source document nor a numeric output establishes that the scorer saw all qualifications or produced a probability. A separate full-precision vector rescore only recomputes vector comparisons; it is not joint cross-encoder inference. [46][47]

Improved specificity is a useful but bounded observation. Jonathan Fernandes’s station-help demonstration moved from broadly related advice to a particular assistance location after reranking. That single example illustrates a possible ordering improvement; it does not establish a general quality gain or independently verify the location. [48]

Relevance judgments and test collections

A relevance judgment assesses an item against a stated need. Qrels record query-item judgments; an assessor applies the criteria. A test collection binds these judgments to queries and documents. Pooling selects items for assessment from leading results of several systems, rather than judging every possible document. [3]

Each case should specify the original need, requester context, eligible corpus snapshot, passage or document counting unit, and relevance rubric. Reuse judgment design and independent assessment boundaries. A current procedure and its historical revision can receive different judgments because the requested task differs. [49][19]

Label meanings are local to the task. TREC’s passage benchmark, for example, treats its grade 1 as related but nonrelevant for binary metrics; the fixture above deliberately defines grade 1 differently. Disagreement can reveal an unclear criterion or genuinely partial usefulness. Automated judges require validation against independent judgments. [19][50]

Unjudged is not an assessor’s negative judgment, even when evaluation software scores it as nonrelevant. Pools may miss discoveries from a new retrieval method. Corpus releases can also remove records or change identifiers. Use judgments matched to the corpus that produced the run, and report assessment coverage. [49]

Coverage and ordering at explicit cutoffs

For a fixed Leon snapshot, assess five current passages. A is the retry procedure, B its useful exception, C the merely related glossary, D another relevant recovery procedure, and E an unrelated entry. Set grades to A=2, B=1, C=0, D=2, E=0. Thus the binary relevant set is G={A,B,D}. [52]

Use fixed-k Precision@k: divide relevant top-k items by k, treating unfilled positions as nonrelevant. Precision@2 and Precision@4 are both 1/2; Recall@2 is 1/3 and Recall@4 is 2/3. Precision over an actually returned shorter list uses its actual size instead. Candidate Recall@4 caps downstream recovery at 2/3. [51]

Reciprocal rank is 1 divided by the first relevant rank, or zero when none appears before the cutoff. MRR averages this quantity across queries. Here reciprocal rank is 1/2. It rewards reaching useful material early but ignores whether the other relevant records are found. [53]

ANN Recall@k uses exact vector neighbors as its reference set, rather than human-relevant passages. Recovering all exact neighbors can coexist with poor relevance. The two measures diagnose different boundaries: approximation quality and usefulness of the representation. [34][35]

Specify query averaging, tie-breaking, excluded cases, duplicate handling, and judgment coverage. Equal-query means give each included query one vote; they do not automatically represent traffic. For no-positive queries, report the chosen exclusion convention and empty-result behavior separately. Metric boundaries explain why missing denominators cannot silently become success. [53]

Controlled comparisons and failure localization

An ablation changes or removes one component to test its contribution. A query slice groups meaningful cases, such as identifiers, paraphrases, languages, long documents, or restrictive filters. Compare lexical, dense, hybrid, and reranked variants on matched requests and corpus versions, recording both quality and resource use. Retrieval effectiveness does not transfer uniformly across domains. [54]

A controlled replay can restore an omitted qualification while holding the query, candidate identities, scorer, and other settings fixed. An improvement supports that input boundary as a cause; it does not establish a deployable repair. Match length and position where feasible, because changing them introduces additional explanations. [55]

Comparators must share relevant constraints. Kuba Rogut’s code-retrieval experiment separated default reads, windowed reads, and windowed reads plus semantic search. Comparing only the first and third conditions would confound retrieval availability with read-window changes. Its context metrics also do not establish downstream task completion. [58][59]

Clicks depend on exposure as well as usefulness: a low-ranked result may never be examined. Missing clicks are therefore not explicit negative judgments. Deployed ordering influences the feedback later available for improvement. Controlled comparisons, uncertainty, and live evidence provide the broader experimental framework. [60]

Permissions across retrieval and disclosure

Authentication establishes identity; authorization determines permitted access. A principal is the user or group whose rights matter. A tenant is an organizational isolation scope. An access-control list, or ACL, records access relationships. Authorization boundaries apply to the requester, independently of the indexer’s broader credentials. [61][62]

Permission metadata must reach each searchable chunk and any separately queryable parent. Synchronization is path-specific: Azure’s SharePoint preview documents permission lag and parent-scope changes that subsequent indexer runs do not automatically capture. Inherited fields alone therefore cannot establish current access. [16]

Filtering before candidate selection and filtering a bounded shortlist have different coverage effects. Neither placement excuses unauthorized disclosure. Rejected content must not reach an unapproved reranker, snippet, cache consumer, or response. A restrictive post-filter can produce an empty list without proving that no eligible records exist elsewhere. [36]

Visible hits are only one disclosure surface. Elasticsearch documents global statistics in relevance scoring and possible aggregate exposure involving inaccessible documents. Counts, terms, and field names need isolation tests too. Its documented behavior is a product-specific boundary, not a claim that every filtered search system leaks identically. [63]

Content and permission freshness interact. After access is revoked, checking newly added content against an older ACL can disclose it. Zanzibar addresses this with consistency tokens tying authorization freshness to content versions. For this documentation service, the proposed policy withholds disclosure whenever the required authorization freshness cannot be established. [62]

Relevant, authorized text can still contain hostile instructions. Retrieval establishes neither instruction authority nor permission to perform an action described inside a document. Preserve the distinction between content and commands through untrusted-content handling. [64]

Revisions, deletion, and searchable visibility

An indexing acknowledgment need not mean a change is searchable. Refresh makes indexed changes visible to search. Elasticsearch can wait for that visibility or force a refresh; forcing frequent refreshes adds indexing, search, and merge costs. Visibility is a separate completion boundary. [57]

For the fixture, track four distinct facts: when the source changed, when ingestion processed it, when search could retrieve it, and which revision the application declares applicable to the requested period. The last is an explicit documentation rule. A newly ingested historical revision must not automatically replace the current procedure. [57][4]

Incremental indexing processes detected changes instead of rebuilding everything. Updating a parent can replace its derived chunk identities; cleanup must remove obsolete chunks from the current search view. Change detection and ingestion scheduling do not, by themselves, prove that the serving result has changed. [65][13]

A tombstone is an observable deletion marker. Removing a source without one can leave indexed records behind. The marker must survive long enough for processing and recovery. Connector restrictions matter: Azure’s documented blob soft-deletion policies exclude one-to-many indexing, which requires explicit deletion of those derived entries. [66]

Reindexing builds replacement searchable representations. A proposed migration prepares and validates the replacement, coordinates its query encoder and index route, verifies live results, and only then retires obsolete derivatives. An index alias can switch targets, but it does not coordinate encoder versions or caches. Alias action error handling also needs explicit configuration. [67]

Correction, revocation, and deletion need separate completion checks. A corrected procedure may remain accessible; a revoked one must become inaccessible to the affected user; a deleted one requires derivative cleanup. Verify serving replicas and caches as well as indexes. Lifecycle fulfillment distinguishes these obligations; searchable absence is not proof of physical erasure. [66][62]

Latency, capacity, and bounded degradation

The critical path is the dependency chain determining completion time. Fan-out sends work to multiple branches; waiting for every required branch exposes the request to stragglers. Parallel branch durations do not add directly to elapsed time. [68]

Measure end-to-end latency, including queueing before execution. The p95 and p99 are the times within which 95% and 99% of the measured requests finish; stage percentiles cannot simply be added. Operational measurements should retain workload, concurrency, completion status, and version context. [68][69]

Branch depth, ANN exploration, reranking count, and scoring-input length consume different resources. A deadline caps waiting, not the amount of relevant information that exists. Compare configurations at matched quality requirements; measure whether extra work recovers evidence or merely increases latency. [38][43]

Cancel unnecessary work after the deadline under the service’s execution policy. Record unfinished branches rather than labeling them empty. Vespa’s coverage and degradation fields illustrate how timeout, missing responses, and matching limits can accompany a returned list. Execution coverage is not relevance recall. [70]

A proposed cache contract keys content reuse by query and filters, corpus snapshot, and representation version, then revalidates requester access before disclosure. A cached ranking cannot grant enduring permission. The exact invalidation and consistency protocol must be established for the deployment; naming these fields does not implement it. [67][61]

The ranked retrieval response

A usable response preserves ordered content, source and revision identities, chunk locations, structural context, and the meaning of each score. Include corpus and representation versions plus execution status. Keep sensitive permission details out of the public payload. This is an application interface design, not a universal engine response schema. [13][46]

Grouping by source and revision can present several passages together. It does not establish semantic deduplication or diverse coverage. Engine hit counts may still count ungrouped passages. Historical revisions and distinct exceptions should remain separate when the task requires them; collapsing them because their wording overlaps would discard useful distinctions. [71]

Completion does not establish exhaustive relevance: a top-k cutoff can still exclude another useful record. Retrieval supplies selected evidence and the conditions of its selection. Retrieval-Augmented Generation uses such material to support answers; answer correctness and citation support require additional evaluation beyond a successful search. [8][10]

Open questions

1. 01

   Establishing completeness for exploratory retrieval remains difficult because another qualifying record may lie beyond any fixed cutoff. Progress would mean a coverage certificate or independently verified enumeration for a defined corpus snapshot, rather than merely increasing k. [8]

2. 02

   Preserving distant qualifications under bounded scoring remains unresolved across document shapes. Larger inputs cost more and can still truncate decisive material. Progress would retain cross-section exceptions in recorded scorer inputs while improving judged ordering under a fixed resource budget. [46]

3. 03

   Consistent revocation across projected chunks and cached results remains hard when permission changes arrive through different synchronization paths. Progress requires measured revocation completion for each path, including parent-scope changes and unavailable authorization services. [16]

4. 04

   Choosing search effort per request requires predicting where additional exploration will recover useful records that are currently unseen. Progress would reduce deadline failures at matched relevance recall across identifier, paraphrase, and restrictive-filter slices, without hiding underfilled responses. [38][36]