Data Quality and Curation

Fitness for purpose

Fitness for purpose means suitability for a particular task, population, and operating setting. Acceptance requirements follow the consequences of errors. A dataset can support historical analysis while being unsuitable for answering current questions. [1]

Accuracy concerns correct values; validity concerns permitted formats and ranges. Completeness concerns required information, uniqueness unwanted duplication, and consistency compatible representations. Freshness concerns whether information remains timely. Coverage concerns represented cases. These properties can conflict: requiring every field may exclude useful records whose uncertainty should remain explicit. [1]

Quality filters must preserve intended capabilities. In Krea's image curation, conventionally poor low-resolution CRT imagery could still serve a desired aesthetic. Removing it would reduce stylistic coverage rather than universally improve quality. [5]

Collection, identity and permitted use

The unit of observation is what one record represents: a request, message, customer, or event. Provenance records origins, production activities, and responsible parties. Data lineage describes source-to-derived-data connections. The terms overlap; both help explain how a record came to exist. [6][7]

A system of record is the designated authority for a business fact. A data contract states the producer–consumer agreement about identifiers, meanings, freshness, ownership, and accepted structure. Integration into existing work develops this distinction. Boundary validation catches contract violations before they propagate; it cannot establish factual correctness. [8]

Governance assigns rules and responsibility for data use. Accessibility, suitability, and permission are separate. Carry purpose and handling restrictions into inclusion decisions; unresolved eligibility belongs with the accountable owner. ODRL represents permissions, prohibitions, and duties, but representation is not enforcement. Privacy and Data Governance covers the broader controls. [11]

Inspection and evidence-preserving repair

Profiling summarizes dataset properties; inspection reads the records behind those summaries. Check types, ranges, missingness, dates, source counts, and document readability. An automatically inferred schema needs review: it can encode existing defects as expectations. A valid value can still be false, and an unusual value can be legitimate. [12]

Identifier checks include uniqueness and referential integrity: references must resolve to the intended records. Check these at producer–consumer boundaries, alongside required fields and source information. A structurally acceptable handoff still needs evidence that its contents mean what the consumer expects. [8]

Inspect transformation effects on counts, changed values, and affected slices. In the studied C4 web corpus, blocklist filtering removed benign scientific and identity-related material. A filter's apparent cleanliness can conceal a changed population. [16]

Replayable operations and edit history preserve different evidence. OpenRefine can export reusable operations, but individual cell edits are not extractable as repeatable operations. Preserve both the executable recipe and permitted records of manual repairs; replay alone does not establish semantic correctness. [17]

Duplicates, revisions and meaningful repetition

Deduplication detects and handles repetition under an identity rule. Content hashes find exact-content candidates. Normalized matches depend on the chosen transformation; near matches require a similarity rule. [18]

For $A=\{ab,bc,cd\}$ and $B=\{ab,bc,ce\}$, two shingles overlap among four distinct shingles, giving $J=1/2$. MinHash approximates this overlap at scale. [18]

Thresholds select candidates, not equivalent records. A changed number or negation can matter despite high overlap. Keep revisions, source references, and duplicate groups separate; select representatives without erasing distinct events or material changes. [18]

Label meaning and observation limits

A label is a recorded target or judgment associated with an example. In supervised learning, targets define what the model learns to predict; Supervision, baselines, and loss explains that relationship. Target validity concerns whether the recorded target represents the intended task. [3]

A proxy stands in for an intended outcome. In Zendesk's documented workflow, automated closure is a status transition, not customer-confirmed resolution. The figure separates that status from later evidence. [22]

No observed outcome is not necessarily a negative outcome. If the target is an event within a fixed horizon, an event-free case observed for less than that horizon remains unresolved. This is right censoring: follow-up ends before the later outcome is known. Delayed outcomes and comparable populations connects such evidence to executions. [25]

Acceptance also needs interpretation. A user can accept a suggestion and then rewrite or delete it. Preserve subsequent edits and overrides, while distinguishing error correction from changed intent or preference. [26]

Annotation, disagreement and correction

Annotation assigns labels under a stated procedure. A scheme defines available labels; guidelines explain their application, including examples and counterexamples. Pilot small batches, overlap independent judgments, and inspect errors before scaling. Inter-annotator agreement measures consistency between labelers; adjudication resolves disagreements into an annotation when a final decision is appropriate. [27]

Retain permitted individual annotations, evidence, guideline versions, and correction history. Major instruction changes can require reannotation. Agreement and expert error audits serve different purposes: labelers can consistently apply an inappropriate rule. [27]

For subjective tasks, majority voting can erase consistent minority interpretations. Preserving disagreement does not mean preserving objective mistakes; it keeps meaningful uncertainty available for investigation instead of manufacturing a single unquestionable “ground truth.” [29]

Coverage, selection and sampling

The target population comprises intended cases; the sampling frame identifies accessible cases. Coverage errors include omissions, duplicates, and erroneous inclusions. Sampling bias is systematic mismatch caused by selection. More observations from the same restricted frame do not establish broader generalization. [6]

Selection can determine which outcomes become visible. If only investigated cases receive definitive labels, labeled records describe that selected subset. The selective-labels literature demonstrates this mechanism for release decisions in criminal justice; transferring the reasoning requires identifying the decision that controls observation. [33]

Suppose an audit samples half of a rare group but one-tenth of a common group. Their design weights are respectively 2 and 10: inverses of inclusion probabilities. Unweighted pooling overrepresents the rare group. Keep population, collected, and curated proportions separate; weights cannot recover a group absent from the frame. [34]

Coverage can fail through both exposure and frequency. Instacart describes products unable to accumulate engagement evidence because they were not shown, alongside rare queries with too few observations. A more complex model did not remove those collection gaps. [35]

Independent data boundaries

Leakage is information crossing a boundary that the intended assessment assumes is protected. Fitting data determines learned behavior; development data guides choices; protected assessment data tests the selected system. Repeatedly changing the system after inspecting final results converts that assessment into development. [4]

Point-in-time correctness concerns what could have been known when a decision was made. In a historical replay, a correction received afterward must not become an earlier input merely because it describes an earlier event. Preserve both the event's timing and evidence of availability; occurrence timestamps alone cannot establish availability. [38][23]

Preprocessing can leak information without copying answers. Fit an imputation mean, scaling rule, or feature selector on fitting data alone, then apply the frozen transformation elsewhere. Deterministic record validation differs from estimating a transformation using protected data. [40]

Evaluation gathers evidence about behavior against intended success criteria. Additional filtering can fail to improve downstream behavior, as reported in particular reasoning-data experiments. Curation supplies documented inputs and protected boundaries; Evals develops the independent evidence needed for usefulness and release decisions. [43]

Dataset releases and reproducible lineage

A snapshot identifies a particular data state. A manifest records membership. Iceberg implements these through explicit file metadata, rather than treating every file currently in a directory as part of a release. [44]

A recipe and its recorded execution state are complementary. DVC separates pipeline declarations from lock records containing commands, parameters, dependencies, outputs, and content hashes. Reproduction also requires accessible artifacts and the relevant execution environment; a hash identifies content without making that content available. [45]

Datasheets for Datasets organizes purpose, composition, collection, processing, uses, and maintenance. Documentation should identify limitations and correction owners. Reproducibility remains subject to permitted retention; documenting a source does not authorize keeping it. [2]

Data change and verified recovery

Distribution shift changes data patterns; Distribution shift and prediction coverage explains its predictive significance. Compare incoming batches and releases against explicit expectations. A distribution alert identifies a discrepancy, not necessarily a defect or degraded outcomes. [12]

Assign an owner, investigate scope, contain unsuitable use, and identify affected consumers. Correction and recovery have separate completion conditions. Restoring a dataset does not automatically rebuild models, refresh indexes, or replace exports. Resume affected use only after the relevant checks; Failure localization connects this work to execution evidence. [48]

Open questions

1. 01

   Filtering rare but useful material remains difficult because generic quality scores can penalize the intended capability. Progress requires demonstrating preserved rare-style usefulness while reducing actual defects, rather than reporting only higher filter scores. [5]

2. 02

   Recovering evidence about selectively unobserved outcomes remains unresolved when earlier decisions determine which labels exist. Progress requires justified observations beyond the selected group, or explicit bounds on conclusions; reweighting observed cases alone may be insufficient. [33]

3. 03

   Separating legitimate subjective disagreement from annotation error remains difficult when both produce inconsistent labels. Progress would identify stable perspectives while independently detecting mistakes, preserving useful distinctions instead of maximizing agreement alone. [29]

4. 04

   Reproduction becomes difficult when required artifacts or environments are no longer available. Progress requires stating which results remain reconstructible and verifying reconstruction under actual access constraints, rather than treating saved hashes and recipes as sufficient. [45]