Multimodal Models and Applications

Modalities and task evidence

A modality is a kind of information, not a file format. Text, visual appearance, and sound provide different observations. Video contains a visual sequence and may carry audio. Unimodal processing uses one modality; cross-modal processing connects modalities. The multimodal learning taxonomy distinguishes their representations, correspondence, and combination. [1]

Evidence can be redundant, providing another observation of the same fact; complementary, supplying a missing part; or conflicting, supporting incompatible claims. More inputs do not necessarily supply more independent evidence. [1]

A caption derived from a recording remains a transformation of that recording. It may help another component use the content, but it is not a second independent observation. Summarization can also change entities, circumstances, or relationships while preserving fluent wording. [5]

Input capabilities and media readiness

An interface should separately specify accepted media, supported combinations, processing limits, and producible outputs. A provider-neutral multipart request contains ordered text and media parts; each media reference carries its format, source identity, and relevant timing information. [6]

A service can hide several specialized components behind one endpoint. A voice pipeline might convert speech into text, generate a textual answer, and synthesize speech. Integrated processing changes those boundaries; an endpoint name alone reveals neither the internal architecture nor the information passed between stages. [9]

Upload, readiness, partial content, and terminal generation status require separate observations. Completion and runtime validation explains why returned content is not sufficient evidence of successful completion. [10]

Representations and information loss

An encoder transforms input into a representation used by later computation. An embedding is a numerical vector representation. Useful compression depends on which distinctions the representation preserves, not merely its size. Embeddings and Representation Learning develops that foundation. [11]

Representations need not share one vocabulary. Unified-IO 2 uses image patch features and audio spectrogram features as continuous inputs. A patch covers an image region; a spectrogram represents sound across time and frequency. Its generated images and audio instead use discrete codes. Shared computation does not make these representations identical. [8]

Preprocessing sets the evidence boundary. Sparse video sampling can omit a brief event; increasing resolution can increase token use and latency. [6]

Cropping removes regions. Resizing reduces spatial detail, while compression trades retained information against computation and may introduce artifacts. Segmentation creates boundaries that can separate related observations. Retaining the source allows later reprocessing; retaining only a reduced representation may make omitted evidence unavailable. [12]

Speech transcription produces text rather than preserving the waveform. Words alone need not retain non-speech beeps, delivery, or acoustic timing. Transcription, translation, and timestamps are also distinct outputs. Voice and Real-Time AI covers speech processing; Vision AI covers visual representations and tasks. [13]

Document pipelines face the same issue: extracting text, tables, and images separately can break their context. Converting figures into summaries adds another transformation. Page retrieval still needs continuity across boundaries, sometimes through page metadata and neighboring-page retrieval. [14][15]

Semantic, temporal, and spatial correspondence

Alignment establishes correspondence across modalities, distinct from preference alignment in post-training. Semantic correspondence concerns meaning; temporal correspondence concerns time; spatial correspondence concerns regions. Recording-level pairing does not identify each supporting segment. [1]

For example recording R1, assume complete capture, correctly annotated beeps, and indicator L visible throughout. A verified clock mapping places audio events on the video timeline. [16]

Spatial grounding identifies the relevant object or region, not just the frame. A reference to indicator L must survive cropping and coordinate changes. Evidence spans and coordinate mappings explains that bookkeeping. [17]

A sound spike coinciding with a visible event establishes a timing match, not the sound's identity. A door movement and a coincident spike still require evidence that the spike was a door slam. [18]

Paired supervision and learned connections

A dual encoder uses separate encoders for two input kinds. CLIP trains image and text encoders to favor matching pairs over alternatives. This contrastive objective makes comparison useful; it does not turn similarity into a probability that every described fact is true. Training relationships determine which pairs should match. [7]

Conditional prediction instead rewards producing an output given another input. CoCa combines image–text comparison with image-conditioned caption prediction. These objectives reward different operations, even when one model trains both. [19]

Coarse captions need not describe every attribute or event. Noisy descriptions provide incorrect associations; ambiguous descriptions may fit several sources. Caption construction changes conditional supervision, while filtering changes which examples the model encounters. Video-training experiments demonstrate these as distinct interventions, not a universal rule that longer captions are better. [20]

Recognizing the same objects is insufficient when their relationships differ. Winoground pairs images with captions using the same words in different arrangements. Correct matching requires preserving the arrangement's meaning, exposing distinctions that coarse matching can miss. [21]

Fusion and the point of interaction

Fusion combines information within representations or decisions. Early, intermediate, and late describe where interaction begins; terminology varies. The useful distinction is what information remains accessible when the paths meet. [1]

Concatenation places representations together; a learned connector transforms them into a compatible interface. Original LLaVA projects visual features into the language model's embedding width and trains that bridge with paired data. Matching widths alone is insufficient: coordinate compatibility requires a meaningful learned relationship. [22]

Cross-attention lets one representation draw information from another. Queries come from the representation being updated; keys and values come from conditioning representations. Attention mixes projected value vectors, not raw images or selected output tokens. Encoders, decoders and cross-attention explains the underlying mechanism. [23]

Dual-encoder retrieval can compare vectors without jointly processing their underlying features. Retrieval then returns candidates, not an interpreted answer. A document assistant must resolve image references and supply actual images to an appropriate answering model. [24]

Media-to-text pipelines expose inspectable intermediate transcripts or summaries and allow components to be replaced separately. Their later stages cannot recover distinctions those intermediates omitted. Integrated processing can preserve richer information paths, but removing an explicit intermediate also removes a convenient diagnostic boundary. Neither architecture is universally preferable. [9]

Conditioning and output capabilities

Conditioning makes an output depend on supplied information. In Unified-IO 2, predicted image or audio codes pass through corresponding media decoders. A media decoder converts an internal output representation into usable media; the codes are not themselves the finished image or waveform. [8]

Output mechanisms differ across architectures. Latent diffusion decodes a generated latent representation into pixels, while conditioning guides generation. Guidance does not guarantee preservation of every source detail. Generative Media covers generation and editing; Voice and Real-Time AI covers conversational delivery. [12]

A generated continuation is a prediction, not another observation of the original event. A plausible reconstruction may fill a missing interval, but it cannot establish what actually happened there. Structured answers likewise need semantic checks beyond parsing, as described in runtime validation. [26]

Cross-modal conclusions and grounding

Cross-modal reasoning uses relationships between observations to reach a conclusion. For R1, recognizing beeps and colors separately is insufficient: the task binds the second beep to indicator L's state. [4]

Grounding links a claim to specific source evidence. Recognition, correspondence, inference, and reporting can fail separately: a correct color with an incorrect supporting interval is not fully grounded. [17]

The evidence trace is an explanatory decomposition, not a reconstruction of model internals. Attention weights show numerical compatibility within a computation; they do not certify a semantic explanation. Correct-looking references make inspection possible but still require validation. [23]

Generated explanations provide a different kind of evidence from internal interventions. Demonstrating a causal role for an internal feature requires changing that feature and observing the effect, with access to model activations. Application-level evidence changes can instead test whether answers depend on the intended inputs. [27]

Missing and degraded evidence

Modality dropout withholds channels during training to improve robustness to missing inputs. ModDrop demonstrated benefits in gesture-recognition experiments. Such training cannot recover an unobserved event or make remaining evidence sufficient for every task. [28]

A task-specific policy can narrow the answer, request missing evidence, or abstain. Audio alone may establish beep order without establishing indicator state. Coverage is the fraction answered; selective risk is error among accepted answers. Selective automation explains why both matter when refusing unsupported conclusions. [30]

Recovery can restore access to a surviving source, but generating a replacement supplies an estimate. A fresh recording also cannot recover an unrecorded historical event. Those distinctions determine whether recovery expands the evidence or merely produces a plausible substitute. [26]

Conflicting inputs and modality dominance

Modality dominance occurs when one input drives answers despite decisive evidence elsewhere. MCR-BENCH holds audio fixed while changing accompanying descriptions. Tested models sometimes abandoned correct audio-based answers under contradictory text. This is behavioral evidence of a bias under those conditions, not a universal hierarchy of trustworthy modalities. [31]

For the indicator task, synchronized visual evidence determines what was lit; narration determines what was said. An announcement of an intended change can coexist with an unchanged indicator. If identity, timing, or visibility remains unresolved, preserve that uncertainty instead of inventing a reconciliation. [31]

Evidential relevance also differs from action authority. Instructions found in an image or recording are content to interpret, not permission to execute tools. Authorization must be enforced outside the model. Prompt injection and instruction authority explains this boundary. [32]

Multimodal correctness criteria

A test case needs source and derivative identities, pairing and clock mappings, the task, acceptable answers, supporting intervals or regions, and expected insufficient-evidence behavior. For R1, the answer reference is green; temporal and object references remain separate annotations. [17]

A rubric states criteria and rating guidance; a test oracle determines acceptable behavior. Oracles and human judgments explains their design. Separate criteria, reviewer preparation, and agreement checks help expose ambiguity. Agreement itself does not establish truth. [36]

Independent assessment requires more than withholding filenames. As an evaluation-design deduction, keep a recording, its clips, transcripts, and captions in the same data partition; otherwise closely related evidence can cross the boundary. Independent data boundaries covers this principle. Refresh tests with representative cases without repeatedly tuning against the held-out set. [37]

Controlled tests of cross-modal dependence

An ablation removes an input or component for comparison. Removing audio from a fixed system tests its behavior under missing input. A separately trained or configured audio-only baseline tests what that route can achieve. These are different comparisons; historical unimodal studies show why weak baselines can exaggerate multimodal benefit. [38]

Matched comparisons hold relevant conditions constant. The following R1 interventions are proposed tests, not model results. Unspecified content and processing remain fixed. Expected changes follow the evidence requirement rather than an assumption that every perturbation should reduce accuracy. [21]

Input removal can create unfamiliar conditions. A performance drop therefore does not uniquely identify useful cross-modal reasoning. Combining removal, plausible substitution, decisive edits, and invariance tests supports a narrower claim about which relationships affect behavior. [38]

Application usefulness and operating limits

Application evaluation includes capture, preprocessing, correspondence, inference, output handling, and user review. Intermediate checks help localize a failure, but location and cause are different conclusions. Failure investigation separates observed execution from competing explanations. [39]

A slice is a meaningful subgroup of cases. Report intact, degraded, missing, conflicting, and mismatched inputs separately, with capture conditions and media lengths. Coverage and independent assessment explains the broader sampling problem. A combined route must outperform a credible simpler route on the outcomes that justify adding it. [38]

A recovery prototype that exposes omitted evidence has not established release readiness. Keep the validated route available for intact captures, recover surviving originals selectively, and compare complete routing and outcomes on intact, recoverable, and unrecoverable cases before expanding automation. Retain the unsupported-answer gate. Unchanged model weights do not establish that changed preprocessing or routing preserves behavior. [42]

Diagnostic retention should be purposeful. Derived representations can retain sensitive information even when raw media is absent. Minimization and remaining identification risk explains why retaining embeddings is still a data-governance decision. [45]

Open questions

1. 01

   Fine-grained supervision from coarse descriptions remains difficult: a caption may omit the relation an application needs. Progress would preserve controlled attribute and event-order distinctions across new scenes, rather than improving only broad image–text matching. [20][21]

2. 02

   Reliable abstention under unfamiliar capture conditions remains unresolved. Missingness and noise change the evaluation population, so a previously useful confidence threshold can fail. Progress requires maintaining bounded error among answered cases while reporting coverage and recovery burden on shifted inputs. [46][30]

3. 03

   Shared audiovisual judges remain difficult across humans and stylized characters because visible motion may correspond to speech differently. Progress would define character-appropriate criteria and demonstrate agreement with qualified reviewers without treating one animation convention as universal. [47][36]

4. 04

   The operating value of richer modalities remains workload-dependent. Better answers can require more processing and review, while degraded modes change completeness. Progress requires matched end-to-end measurements that identify where added evidence improves supported completion enough to justify those costs. [44]