Structured Outputs and Tool Calling

Generated data and executable requests

A structured output follows an explicit machine-readable contract. A tool call names an operation and supplies arguments. Application code or a delegated runtime executes that request; generating it does not perform the operation. [1]

In a constructed ticket service, assigning ticket T-42 to user U-9 involves four distinct representations. Their similar contents conceal different evidential roles. [1]

Tools can request reads as well as writes. A workflow can also combine tool calls with a structured final answer. [4]

Output contracts and missing information

A schema describes admissible data. JSON represents values; JSON Schema expresses constraints on those values. Declaring a property does not require its presence: required does that. Unrecognized properties remain allowed unless constrained, for example with additionalProperties: false. [5]

The assignment contract must also explain field meanings. An assignee identifier names a user record, not a display name; an expected revision identifies the state on which a proposed change depends. Descriptions communicate these meanings to the model and maintainers, while executable validators enforce the checkable rules. [6]

A tagged union uses a distinguishing field to select a result shape. Here, proposal requires complete arguments, needs_input identifies information to request, and unknown explains unavailable evidence. Mutually exclusive tags prevent overlap: oneOf requires exactly one matching branch, whereas anyOf permits several. This design represents insufficiency without inventing an assignee. [11]

A schema dialect defines keyword semantics; a root $schema identifies it. A generation engine may implement only a subset. Record both the intended dialect and supported generation constraints: declaring a dialect does not install missing capabilities. Keep producer instructions, validators, and consumers on the same contract. [12]

Constraints during generation

A token references an entry in the model vocabulary. During generation, the model produces logits, relative scores for possible next tokens; a selection procedure chooses a continuation. Tokenization explains reconstruction of text from tokens. LLM Inference covers general selection and serving behavior. [13][14][15]

Constrained decoding removes next-token choices that cannot continue the requested structure. A grammar specifies legal compositions; a schema constrains completed values. An engine can compile supported schema rules into a grammar, track the generated prefix, and mask invalid choices before selection. OpenAI's described implementation preprocesses the grammar into a reusable artifact. [18]

Token boundaries need not coincide with grammar boundaries. One token can contain several structural characters, so checking it may require several parser transitions. Nested structures require remembering unfinished enclosing structures. XGrammar's implementation combines live parser stacks with cached token-validity information; it does not precompute every possible complete stack. [19]

Runtime support is part of the contract. The llama.cpp grammar documentation warns that unsupported schema features may be skipped during conversion and that output constraints do not automatically explain the task in the prompt. Inspect conversion warnings and test the generated language; an accepted schema file alone does not establish enforcement. [22]

Completion, parsing and runtime validation

The response envelope carries status and payload information; the application value is inside it. Inspect completion status before treating that payload as final. A refusal, token-limit cutoff, cancellation, or transport failure needs its own outcome path rather than an assumption that the requested record exists. [23][24]

Parsing converts serialized text into values. Runtime validation checks those values against the contract. A TypeScript annotation performs neither operation: type annotations disappear from emitted JavaScript. Incoming model data needs executable checks before a consumer relies on its declared type. [25][26]

A published partial-parsing example retains two completed strings while discarding an unfinished third; ordinary parsing rejects the incomplete array. Another incomplete object can already validate against a model. These examples establish that obtaining a valid value does not establish that the original response finished. [28]

Streaming completed records and streaming partial previews serve different consumers. A UI may display an evolving object, while an executor waits for the call's completion signal and final checks. Keep previews explicitly provisional so defaults or omitted unfinished fields cannot silently become final action arguments. [31]

Meaning beyond schema acceptance

Semantic validation checks values in their business context. A string can have the correct representation while identifying the wrong ticket. A domain invariant is a consistency rule that valid application states must preserve. A referential check establishes that an identifier points to an existing eligible record. [9][32][33]

A system of record is the designated authoritative source for a business fact. Privacy and Data Governance explains its ownership and field meanings. Checks against that system establish facts such as current membership; they cannot reconstruct an ambiguous intention that the user never supplied. [32]

A precondition is an obligation before invocation; a postcondition states what successful execution owes. Schema acceptance establishes only specified properties of a value. It does not establish the intended state change. Acceptance criteria connect success to observable outcomes rather than attractive representations. [36]

Tool contracts and operation selection

A tool definition gives the model a stable name, purpose, and argument contract. Its result contract and operational behavior tell the executor and downstream consumers what to expect. Descriptions guide selection; application code enforces the operation. Interfaces differ in which of these elements they expose directly. [39]

Command-query separation separates operations intended to change state from requests for information. That distinction helps callers reason about effects; it does not exempt reads from permission checks or resource limits. Credentials and the identity being represented belong in trusted execution context, rather than fields that the model may invent. [41][42][43]

Model Context Protocol, or MCP, standardizes exchanges between a host application's clients and capability servers. Those exchanges can discover and invoke tools. The model-facing proposal and the application's server invocation remain different interfaces; neither connectivity nor discovery grants execution authority. Model Context Protocol covers roles, transport, and lifecycle. [47]

Authorized dispatch and call identity

A principal is the person or service whose authority is exercised. Authorization determines permitted operations on particular resources. A trust boundary is a crossing where incoming data or claimed authority needs checking. Obtain identity from trusted application context, not a generated claim of consent. Access governance and AI Security explain the broader boundaries. [42]

When policy requires approval, bind it to the operation's significant details, identity, and lifetime. Changing the target or arguments invalidates the prior approval. Recheck relevant permissions and current state at execution: time-of-check to time-of-use failures occur when the checked operation differs from the one actually performed. [48]

Argument validation applies parsing, schema, and domain checks to a proposed invocation. A registry maps permitted names to implementations. Dispatch through that registry instead of evaluating generated text. Return results with the originating call identifier; completion order and function name cannot reliably distinguish several outstanding calls. [51]

# Interface-neutral: pre-invocation checks raise typed rejections.
def dispatch(call, context):
    require(call.status == "complete")
    handler = registry.require_known(call.name)
    args = parse_json(call.arguments, reject_duplicates=True)
    handler.validate_schema(args)
    handler.validate_domain(args, context)
    authorize(context.principal, handler, args)
    require_current_approval_if_needed(context, handler, args)
    result = handler.invoke(args, context)
    # Post-invocation validation: effects may already have completed.
    # Failure here does not establish that execution was prevented.
    handler.validate_result(result)
    return tool_result(call.id, result)

For the constructed service, assignment checks the expected revision atomically with the write. An earlier read alone leaves a race. This is optimistic concurrency control: reject a change based on stale state. HTTP If-Match provides an analogous precondition when the endpoint supports it; a failed comparison prevents applying the method. [54]

Independent reads may execute concurrently. A write whose arguments require a read must wait for its actual result. Several proposals emitted together do not establish independence, and a model-generated guess cannot substitute for the prerequisite observation. [55]

Results and justified completion claims

A result envelope associates an observation with its operation and status. Validate returned values before downstream use. Tool execution errors can be represented separately from protocol errors; neither should be converted into success merely because a response arrived. The versioned MCP tools contract explicitly distinguishes these error paths. [41]

An empty body also needs interpretation through the endpoint contract. HTTP 204 indicates successful fulfillment without response content. For asynchronous operations, distinguish errors that prevent starting from failures occurring after acceptance; a start response cannot stand in for the eventual operation result. [54][40]

Large results can be stored outside model context and summarized. Keep the authoritative identifier, outcome status, retrieval reference, and explicit omissions in the representation supplied downstream. A shorter summary must not turn partial evidence into a complete answer. Context Engineering covers selection and compaction. [57][58]

Boundary failures and bounded correction

Recovery depends on both the failed boundary and whether execution could already have occurred. A model refusal, an application rejection, and a tool failure are different outcomes. A repair request changes candidate data; it does not establish permission or resolve an uncertain external effect. [17][60][61]

Useful validation feedback identifies the rejected field and violated rule. In Samuel Colvin's extraction demonstration, a date validator rejects an interpretation, the framework returns the error, and another final-result call supplies a corrected value. That trace demonstrates repair for one case, rather than guaranteed convergence. [63]

Preserve the original request and relevant errors across correction attempts. Set attempt or elapsed-time limits and retain exhausted-attempt information. Instructor documents both bounded stopping and contextual re-asking. General retry scheduling belongs in Harness Engineering; the invocation boundary must still expose an explicit terminal outcome. [61]

Uncertain effects and safe retries

A ticket write can commit before its response is lost. The caller then lacks knowledge of the outcome; the service has not necessarily failed. Durable execution has the same gap when an external effect completes before its result is recorded. Replaying recorded history cannot recover a completion that was never recorded. [62]

Idempotency means repeated attempts have no additional intended effect beyond one execution. An idempotency key identifies the same logical operation across retries. Duplicate prevention depends on the receiving service coordinating that identity with the mutation; merely writing the key into a client log does not protect the external effect. [65]

Assume the constructed ticket service scopes keys to the caller, rejects changed parameters, atomically records each key with its mutation, and retains operation results throughout the retry window. It also exposes an authorized status lookup. These are explicit service requirements, not properties supplied by the schema or client library. [65][40]

Reconciliation resolves uncertainty using authoritative evidence. Look up the operation by its retained identifier. A completed result supports reporting completion without another mutation. A read showing the desired assignee is weaker when another actor could have made that change. Inconclusive evidence leaves the operation unresolved. [56]

Interface verification through state and failures

A fixture supplies controlled initial data; a trial executes a task; a trajectory records observations and actions. Executable evaluation cases explain these foundations. For an interface, inspect required calls as well as resulting state: a read can be essential while leaving the database unchanged. [69]

BFCL V3 combines state checks with required execution-path checks on each turn. This illustrates why final state alone can miss skipped information gathering. The ticket cases below are a proposed test design: they specify what the boundary should preserve, rather than report measured reliability. [69]

Deterministic boundary tests inject fixed proposals and failures, then assert allowed transitions and forbidden effects. Property-based testing generates many inputs against an executable predicate—for example, supported values survive serialization round trips. Generating unusual inputs is not itself an oracle, and reproducing a validator's implementation in its test can reproduce the same defect. [71]

Repeated model trials assess different behavior: selection, argument meaning, clarification, and final reporting. Keep structural acceptance separate from task success. Retain refusals, rejections, and unresolved outcomes in the results rather than evaluating only completed successes. A passing deterministic dispatcher test does not establish that the model chooses appropriate actions. [69][72]

Open questions

1. 01

   Portable constraint enforcement remains difficult when validators and generation engines support different schema features. Silent omissions can weaken an accepted contract. Progress would be a versioned conformance corpus that exposes unsupported constraints explicitly and tests identical boundary instances across engines. [22][12]

2. 02

   The effect of grammar constraints on task meaning remains workload-dependent. Locally legal token selection differs from conditioning on complete valid outputs. Progress requires matched evaluations that separate structural acceptance from content correctness while controlling the schema, model, and selection procedure. [21]

3. 03

   Meaning-preserving repair remains harder than satisfying another validator. A correction can remove an error while altering supported facts. Progress would track original evidence, changed fields, and independently judged task correctness across bounded retries, including exhausted cases. [30][71]

4. 04

   Recovery remains unresolved when a provider offers neither durable duplicate protection nor operation-specific status evidence. A checkpoint cannot determine an unrecorded external effect. Progress requires a receiver contract that resolves late retries, or an explicit unresolved outcome that prevents automatic duplicate submissions. [77][68]