Vision AI

Visual tasks and inspection evidence

Classification assigns categories to an image. Detection locates individual objects using bounding boxes—rectangles specifying location and extent. Segmentation assigns image locations to regions. These visual task definitions specify different outputs: recognizing a defective scene does not identify every part or delineate its defect. [1]

Grounding connects a reference to an inspectable image region. A referring expression such as “the scratched part beside the blue fixture” requires selecting among candidates using appearance, location and relationships. Referring-expression comprehension supplies that localization task; returning a region does not independently verify every property in its description. [8]

The constructed workbench contains two parts, a narrow scratch and a blue fixture. Acquisition produces an observation; an encoder produces representations; a task head produces predictions. Inspection datasets such as MVTec distinguish image-level anomaly decisions from pixel-level defect localization. Neither benchmark task alone establishes fewer missed defects or a manageable review workload in a factory. [9]

Pixels, capture and spatial sampling

A raster image is a grid of pixels: spatial samples with channel values, such as red, green and blue components. Resolution specifies grid dimensions. A decoded RGB image can be an H×W×3 array; its encoded file bytes are a different representation. Decoding must succeed before array dimensions or channel values can be trusted. [10][2]

Pixels depend on illumination, surface reflectance and viewing geometry. Under ideal perspective, $x=fX/Z$: camera-space position $X$, depth $Z$ and focal scale $f$ determine projected position $x$. Increasing object size and distance together can preserve apparent size. Image formation therefore explains why a photograph does not uniquely determine physical dimensions or intrinsic surface color. [11]

Exposure is the interval over which a sensor gathers light. Longer exposure collects more photons but can blur moving edges, including a narrow scratch. Digital gain amplifies existing measurements and noise; it does not collect additional photons. Color channels also reflect sensor filtering and reconstruction, rather than three independent full-resolution measurements at every location. [12]

RGB numbers require a color-space interpretation. Encoded sRGB is nonlinear with respect to light: dividing an eight-bit channel by 255 changes its numeric range but does not linearize intensity. Follow the model's documented color and normalization conventions rather than automatically applying a physically motivated conversion. [13]

Aliasing occurs when sampling makes distinct fine patterns indistinguishable. Low-pass filtering before downsampling reduces aliasing by removing unsupported detail. Enlarging the resulting array interpolates values; it cannot uniquely recover what was discarded. [2]

Preprocessing and source coordinates

Decoder conventions belong in the input contract. OpenCV normally returns BGR channels and applies embedded EXIF orientation unless particular flags disable it. Record orientation before interpreting coordinates. Specify channel order, numeric range and normalization; identical shapes do not establish identical model inputs. [10]

Use continuous XYXY edges, with x rightward and y downward. Letterboxing preserves aspect ratio through resizing and padding. For crop origin $(l,t)$, actual resize factors $(s_x,s_y)$ and padding $(p_x,p_y)$, transform every box corner consistently. [3]

Crop an oriented 640×480 source at (100,60), retaining 400×300 pixels. Resize to 320×240, then pad 40 pixels above and below. The source box (200,120,300,220) becomes (80,88,160,168). Store actual resized dimensions because rounding affects scale. Inversion recovers retained coordinates, not cropped or clamped extent. [3]

Images and masks share geometric decisions but need different numerical treatment. Interpolating categorical labels can invent invalid category values; nearest-neighbor resampling selects existing labels. It can still erase thin regions during downsampling. Keep image identity and transformation history with predictions, following Evidence spans and coordinate mappings, so displayed regions remain tied to the correct source. [14]

Spatial features and visual encoders

An encoder maps pixels to a representation used by later computations; Representations and the distinctions they preserve explains that prerequisite. A convolutional filter computes weighted sums over local neighborhoods and input channels. Reusing its weights across positions creates a feature map. Each output channel records a different learned response, potentially useful for edges, textures or more complex patterns. [15]

Stride is the spacing between filter placements. Pooling summarizes neighborhoods, often by their maximum. A receptive field is the input region that can influence one output. For example, a 3×3 stride-one convolution followed by 2×2 stride-two pooling grows theoretical support from 3×3 to 4×4 while spacing outputs two input pixels apart. Learned weights ordinarily remain fixed during inference. [15]

A Vision Transformer divides an image into patches, flattens each patch and projects it into a vector called a patch token. Positional information preserves where patches came from. Attention mixes their representations; its general mechanism belongs in Queries, keys, values and information mixing. The patch, its embedding and later contextual representation are distinct objects. [16]

Equivariance means transforming an input transforms its output correspondingly; invariance means the output stays unchanged. Shared convolution supports translation equivariance under compatible shifts and boundaries. Striding breaks arbitrary one-pixel guarantees. Global pooling aggregates a spatial grid into a vector, removing explicit position indexing. Recognition can benefit from position tolerance while localization needs position to remain accessible. [17]

A feature pyramid combines coarse context with finer spatial features. In FPN, lateral projections of backbone maps are added to upsampled coarser maps, then filtered to form new prediction features. The resulting P-levels are newly combined representations, not renamed backbone outputs. Upsampling alone cannot recover discarded detail; the lateral path supplies retained local information. [18]

Recognition and category vocabularies

A task head converts features into task outputs. For classification, logits are unnormalized category scores. Softmax converts them into a distribution over mutually exclusive labels. During supervised fitting, cross-entropy penalizes low probability for the reference class. Fixed-parameter inference needs preprocessing and the forward computation, without parameter updates. [19]

Separate binary losses do not imply that real-world labels are statistically independent. Neither output contract supplies an inventory of instance locations: “scratch present” leaves “scratch on part A” unresolved. [20][17]

Closed-set recognition chooses among known categories, even when none describes the input. Open-set recognition permits rejecting unknown categories. A low-confidence rejection rule is not a reliable universal unknown detector: unfamiliar inputs can receive high scores. [21]

Calibration concerns whether comparable predictions assigned probability p are correct roughly fraction p of the time. It differs from ranking, accuracy and agreement between annotators. A confident label is not individual proof; Probability calibration and selective automation explains how confidence becomes testable evidence. [22]

Object candidates and bounding boxes

A detector returns category, box and score records. Box regression predicts position and extent. Proposal-based detection first generates candidate rectangles. Original Faster R-CNN uses anchors—reference rectangles at different scales and aspect ratios—and predicts objectness and box adjustments from shared features. Proposals are intermediate candidates, not final category-specific detections. [23]

Dense detection instead predicts from many spatial locations. Original YOLO assigns responsibility to the grid cell containing an object's center, with a fixed number of box predictions per cell. Its score conventions combine object presence, predicted overlap and category information. These historical details do not describe every later model using the name; small-object localization remains a task to test. [24]

Set prediction uses a fixed collection of outputs that can represent objects or no object. Original DETR trains with Hungarian matching: a minimum-total-cost one-to-one assignment between predictions and labeled objects. Matched outputs receive category and box losses; unmatched outputs learn no object. This coordinates supervision without an inference-time duplicate-removal stage, but does not guarantee correct or duplicate-free predictions. [25]

Non-maximum suppression, or NMS, retains high-scoring candidates and removes lower-scoring boxes whose IoU exceeds a threshold. It compares predictions with predictions, not references. Separate classes when required; equal-score choices can vary by implementation. Overlap does not establish identity, so suppression can remove a real neighboring object. [4]

Masks and object boundaries

A mask represents region membership at image locations. Semantic segmentation assigns categories to pixels; instance segmentation separates individual objects of the same category. Per-location scores retain uncertainty that selecting one hard label discards. A box encloses an extent but does not specify which enclosed pixels belong to the object. [1]

Panoptic segmentation combines countable objects, called things, with regions such as sky, called stuff. Each pixel receives one category and, for things, an instance identifier. Segments do not overlap; void can mark unspecified pixels. These identifiers distinguish objects in one annotation, not persistent identities across video. The format describes visible regions rather than hidden completion. [26]

Encoder-decoder segmentation compresses spatial information to gather context, then expands it for dense prediction. Original U-Net concatenates cropped fine-resolution features with expanding-path features. Cropping aligns maps because unpadded convolutions shrink them. A final projection produces per-pixel class logits; weighted cross-entropy supervises the map. Its output covers an interior region, rather than automatically preserving input dimensions. [27]

Promptable segmentation lets a point, box or mask indicate a requested region. Original SAM combines a reusable image embedding, prompt encoding and mask decoder. A point can ambiguously indicate an object or a part, so multiple candidate masks may be returned. Their estimated overlap scores are predictions, not overlap measured against an available reference. [28]

Fine boundaries require direct inspection. The portrait-segmentation example in Magic Editor describes missing hair strands that became obvious when a mask drove blur. Similarly, a plausible part mask can miss a narrow scratch or merge touching parts. Postprocessing may help, but its final measurement needs evaluation rather than visual plausibility alone. [29]

Grounded references and spatial relationships

A vision-language system uses images and language together; alignment and fusion belong in Multimodal Models and Applications. For grounding, appearance, location and neighboring-object relationships help distinguish candidates. Store the selected region with source identity and coordinate convention. Localization establishes where the model points; verifying “scratched” requires checking the indicated surface. [8]

Class names also need domain meaning. A volleyball “block” is an action; a cable defect named “thunderbolt” is not a lightning bolt. Descriptions can specify the intended appearance, but linguistic clarity does not establish that a model can perceive it. [30]

Spatial claims require a reference frame and an operational definition. In an image, “left of” can compare horizontal box centers; containment can compare region membership. Neither box overlap nor pixel adjacency proves physical contact. Counting visible instances establishes a visible count, not the number hidden behind other surfaces. [31][11]

Camera calibration relates image coordinates to viewing rays using lens and camera-pose parameters. A ray still does not specify a unique depth. Physical size and distance therefore need additional scale or geometric evidence; another calibrated view helps only when the same scene point can be matched. Image-relative coordinates alone are not metric measurements. [31]

Visual supervision and annotation meaning

Training fits parameters against targets; inference applies fitted parameters, as explained in Training, fitted state, and inference. An image label supervises a category decision. It does not specify a defect's boundary. Box targets constrain location and extent; pixel targets constrain a dense map. Choosing target granularity determines what correctness the fitting objective can directly reward. [19][25][27]

Amodal annotation estimates full object extent, including hidden portions. Visible-region annotation marks only observed portions. Both can label identical pixels differently; an inferred hidden boundary remains an interpretation. [5]

Background means a defined non-target region. Void or ignored regions instead indicate that the annotation does not supply an ordinary evaluated target there. Do not silently turn uncertain boundaries into background. General disagreement handling belongs in Annotation, disagreement and correction. [26]

Perception labels and policy decisions must remain separate. In one headphone-review workflow, hearing aids counted as a positive visual signal under the detector's convention but did not warrant a violation. A single rejection button mixed those judgments and could turn a permitted case into incorrect detector feedback. Other tasks may define the visual class differently. [32]

Visual pretraining and valid transformations

Pretraining learns reusable features before task-specific adaptation. Self-supervised learning derives targets from the images themselves. Related-view training creates two augmented versions of an image and rewards their association against competing examples. SimCLR places a projection head after the encoder for this objective, then discards that head when using encoder representations downstream. Contrastive learning explains the general comparison mechanism. [33]

Masked reconstruction provides a different target. MAE hides patches and sends only visible patch embeddings with positions through its encoder. A decoder combines encoded visible patches with mask tokens and predicts missing pixels. Loss is evaluated on masked locations; the decoder is later discarded. Reconstructing plausible content is a training signal, not proof that the actual hidden content is recoverable. [34]

Augmentation deliberately transforms training examples. Label invariance assumes $y(Tx)=y(x)$: transforming image x with T preserves target y. Spatial targets often require $y(Tx)=T_y(y(x))$, where $T_y$ transforms the target too. A crop may remove the scratch; a flip may reverse left-of. Neither operation is automatically valid merely because it is common in classification. [35]

Image-text pretraining rewards matching images and descriptions. CLIP learns separate encoders whose similarities distinguish paired examples from mismatches. At inference, candidate label descriptions can be compared with an image without fitting a new category head. This enables text-conditioned recognition, but similarity is not a calibrated truth probability and matching captions does not guarantee exact spatial perception. [36]

The distinctions rewarded by supervision matter. Captions can describe oppositely oriented objects identically, weakening the incentive to preserve orientation. In a drawing game, visible words matching the requested scene also raised CLIP scores, creating a shortcut around drawing it. These examples motivate tests of the intended visual relationship rather than reliance on semantic similarity alone. [37][38]

Richer annotations can amplify errors. Moondream's caption-expansion example turned an uncertain tiny object into a confidently described person and added unsupported detail. More descriptive training text is useful only when its added claims remain tied to visual evidence. [39]

Task adaptation and frozen-feature baselines

Adaptation should address a diagnosed mismatch. A narrow image-understanding contract can be more useful than broad reasoning ability: extracting a mathematical expression and solving it require different capabilities. Prompted extraction can reduce integration effort, but convenience does not establish superiority over a dedicated detector. [40][41]

Fine-tuning can improve in-distribution accuracy while worsening performance under distribution shift relative to frozen-feature probing. That observed possibility makes updating the encoder a testable intervention, not an automatic upgrade. Compare routes on protected cases with matched preprocessing and label meaning. [43]

A reported ten-example-per-class detector comparison motivates collecting a small labeled baseline before accepting zero-shot performance. Its incomplete training specification prevents a universal ranking. Record label granularity, resolution and fitted components so a comparison identifies what actually changed. [44]

Occlusion and unresolved visual evidence

Occlusion occurs when one surface hides another from view. Missing evidence also arises through cropping, blur and insufficient sampling. These differ from misinterpreting a clearly visible feature. Multiple underlying scenes can agree with the observed pixels, so a missing detection does not establish absence. [11]

Another view can resolve ambiguity only if it supplies usable observations of the relevant surface. Occlusion, reflection and weak texture can still prevent correspondence. A reviewer can resolve an interpretation dispute when evidence is inspectable; neither human review nor a larger model guarantees recovery of a hidden fact. [45]

Keep unresolved outcomes explicit. In a narrated product-image evaluation, neither source nor edited image established the stated wonton quantity, so the candidate was rejected. This illustrates a bounded rule: when a required fact cannot be verified, uncertainty must not silently become approval. [46]

Capture changes and visual shortcuts

Distribution shift changes input or outcome patterns; Distribution shift and prediction coverage supplies the general definition. Visual changes include illumination, viewpoint, sensor response and background. JPEG compression can additionally damage fine detail through lossy encoding. A revised defect definition changes the target itself, rather than merely changing the observation. [47][48]

Shortcut learning exploits an available correlation instead of the intended evidence. A defect classifier might learn a workbench background associated with rejected parts. Controlled background research keeps foregrounds while replacing backgrounds; comparing class-consistent and randomized replacements helps distinguish background signal from compositing artifacts. This is a useful test design, not measured factory performance. [49]

Calibration can deteriorate under shift, including when temperature scaling worked on nearby validation data. A high confidence or an absence of alerts therefore does not certify continued correctness. Evaluate actual errors in changed conditions rather than treating uncertainty estimates as a substitute. [50]

Visible text can alter CLIP's classification through a semantic shortcut. Multimodal instruction injection is different: attacker-controlled media influences generated responses or subsequent dialogue. Neither mechanism grants application authority. Treat inspected content as data; Prompt injection and instruction authority explains the control boundary. [51][52]

Evaluation populations and capture boundaries

Evaluation claims need a population: familiar objects under new captures, unseen objects, new cameras or new sites. Coverage and independent assessment separates that claim from the sample. Group related captures before splitting; randomly assigning files can place nearly identical evidence on both sides. Object, session and site boundaries answer different generalization questions. [53]

Group separation does not guarantee representative coverage. Include negative images, small defects, clutter, difficult lighting and consequential rare cases. Record uncertain labels rather than silently excluding them. RF100-VL illustrates varied viewpoints and imaging domains, but curated diversity is not a demonstrated random sample of a deployment. [47][54]

Class imbalance means some labels occur much more often than others. Training resampling changes which examples are encountered; loss weighting changes their optimization contribution. Either may help rare classes, but neither justifies balancing away deployment frequencies in evaluation. Overall accuracy can hide poor rare-defect detection. [55]

Keep adaptation and threshold selection separate from final assessment, following Independent data boundaries. A protected test is informative only for the deployment conditions it represents. Preserve source identities, capture conditions, preprocessing versions and labeling policies so changes in measured performance can be interpreted. [56][47]

Recognition and detection measurements

Define the counting unit before calculating a score. With “defective image” positive, a true positive is a correctly flagged defective image; a false positive flags a normal image; a false negative misses a defective image. Precision is TP/(TP+FP), and recall is TP/(TP+FN). Metrics, denominators and proxy failures explains why the same words can describe different units. [57]

Detection evaluation first matches predictions to references using class and overlap rules. Ordinary references match once; duplicates become false positives. This differs from NMS between predictions and DETR assignment during training. The constructed list below has three reference parts, an IoU threshold of 0.5 and no ignored or crowd regions. [58][25]

Average precision summarizes interpolated precision across recall levels. Main COCO AP averages over classes and ten IoU thresholds, 0.50–0.95, using 101 recall levels and a 100-detection cap. AP50 is different. Filtering low scores beforehand can truncate recall; a pooled precision trace is not class-averaged AP. [58]

Region overlap and boundary quality

The illustrated reference contains a 100-pixel region and a separate 10-pixel thin region. Both predictions retain 100 pixels with no extras: IoU is 100/110 ≈ 0.909 and Dice is 200/210 ≈ 0.952. One misses the entire thin region; the other misses an edge strip. Equal overlap scores can conceal different inspection consequences. [59][60]

Averaging rules matter. Cityscapes accumulates a dataset confusion matrix before computing class IoUs, excludes ignored-reference contributions as specified, and omits undefined zero-union classes from its mean. Predicting an ignored category at a valid reference pixel still misses that reference category. Averaging per-image IoUs or replacing undefined classes with zero changes the result. [6]

Boundary IoU compares interior bands near reference and predicted contours. Band width controls sensitivity. Dilation, erosion and displacement fixtures expose boundary errors, but a boundary-only score can underweight distant interior errors. Evaluate the required region and contour properties together; neither establishes physical defect dimensions without a measurement model. [60]

Evidence for grounded and spatial claims

Object recognition can succeed while relationships fail. Winoground pairs two images with captions containing the same words in different orders. Text scoring checks both caption choices; image scoring checks both image choices; group scoring requires all comparisons. This probes compositional matching, not region localization or every form of spatial reasoning. [7]

Score region selection, attributed properties, relations and answerability separately before accepting the complete claim. Explicit judgment criteria must resolve ambiguous cases. A fluent explanation is insufficient: narrated orientation failures included invented supporting details. Removing the image or consulting another model can be a diagnostic control, but neither alone establishes grounded reasoning. [61][7]

Operating decisions and reassessment

Usefulness requires comparison with an existing or simpler process under intended acquisition conditions. Measure missed defects per inspected part, erroneous rejections, review fraction and total processing time. Define acceptable limits before comparison. MVTec supplies inspection-like prediction tasks, not evidence that this workbench workflow reduces losses or labor. [9]

Selective prediction accepts some outputs and withholds others. Coverage is the accepted fraction; selective risk is error among accepted cases. Threshold changes alter that subset, so assess both quantities on representative labeled data. Zero coverage leaves conditional risk undefined. Probability calibration and selective automation explains the statistical requirements; confidence alone neither controls risk nor authorizes rejection. [65]

A useful regional input must preserve both the defect and enough context to assign it to a part. If tight crops reveal scratches but remove containing boundaries, more confident classification does not resolve ownership. Compare context-preserving crops before shifting ambiguous cases into automatic rejection. Measure resulting misses, false rejections and review demand together. [66][65]

Validate the complete path: decoding, orientation, preprocessing, coordinate mapping, displayed evidence and policy handling. Bind conclusions to camera, model, transform, label and threshold versions. Maintain regression cases that reflect real use. Capture changes, changed target definitions or consequential new failures require reassessment; there is no universal camera-change acceptance threshold. [67]

Protected development comparisons justify advancing a candidate for independent assessment, not claiming operational improvement. Live evidence and causal improvement covers that next claim; Evaluation records and reassessment explains how conclusions remain attached to their conditions. Tables suffice here because the decision depends on evidence and consequences, rather than a universal processing sequence. [56]

Open questions

1. 01

   Isolating resolution from representation changes remains difficult. Encoder checkpoints, connectors, token-selection methods and training often change together. Progress would hold learned weights fixed where possible, specify the intervention and separately report text recognition, counting and spatial relations, including failures rather than only aggregate averages. [62][63][64]

2. 02

   A unified grounding evaluation must separate correct regions, true attributes, valid relations and absent targets. Matching benchmarks cover only parts of this obligation, while ambiguous descriptions complicate reference judgments. Progress would provide independently adjudicated cases and separate scores before reporting complete-claim correctness. [7][8]

3. 03

   Recapture policies must improve available evidence without creating excessive repeat work. Another view can remain occluded or difficult to match, and an uncertainty score does not identify which observation would resolve a fact. Progress would measure resolved claims, remaining errors and recapture burden under defined capture failures. [45][65]

4. 04

   Camera-change reassessment needs task-specific evidence rather than universal thresholds. Changes can affect defect visibility, calibration and review demand differently. Progress would compare configurations on new object/session groups and demonstrate acceptable per-part misses, false rejections and review load before expanding use. [53][50]