Observability

Operational questions and evidence

Observability is the ability to investigate system behavior using available evidence. Telemetry is the data emitted about that behavior. An assistant that retrieves a policy, queries an order API, and drafts a customer reply needs evidence about both its execution and the result delivered. These distinctions follow the OpenTelemetry observability primer. [1]

The user’s task is obtaining an accurate reply about an order. That task may involve several HTTP requests and repeated attempts. Counting successful requests therefore answers a different question from counting successfully resolved tasks. [2][1]

Evaluation judges behavior against intended-use criteria. A valid response format or successful API call can coexist with an incorrect policy answer. Attach the judgment to the relevant task, preserving how it was obtained. Evals develops criteria, measurement validity, and calibration. [7][8]

A model’s written explanation is another output to inspect. Research has demonstrated explanations that omit influences affecting an answer. Observable inputs, actions, and results support an execution account; a plausible rationale does not reveal hidden computation or establish why the answer occurred. [9]

Spans, concurrency and completion boundaries

A trace connects execution records. A span records a timed operation; its attributes hold metadata, its events mark instants, and its status describes that operation. Parentage records execution structure, while timestamps show overlap. A span link associates related work without assigning another parent. OpenTelemetry trace concepts define this vocabulary. [10]

OpenTelemetry supplies shared instrumentation conventions and tooling. Automatic instrumentation can wrap supported client libraries, but application work still needs explicit coverage: assembling model input, validating a reply, and choosing an output destination. A captured model request alone cannot explain those decisions. [11][12]

Record validation failures, cancellation observations, and stream termination as distinct events. If client acknowledgment is absent, receipt remains unknown. Treating server completion as delivery would erase precisely the failure boundary needed to investigate a complaint about a missing reply. [15]

Identity across services and execution attempts

A correlation ID joins related records. Its meaning must be explicit: a task identifier follows user intent, while request, attempt, trace, and span identifiers distinguish execution units. One task can survive several executions. The following identifiers describe an application design, not a universal runtime schema. [2]

Context propagation carries execution identity across boundaries. With W3C Trace Context, the sender injects a traceparent value and the receiver extracts it before creating downstream spans. Independently instrumented services otherwise produce disconnected records. Incoming context is untrusted correlation metadata, never authorization. See context propagation. [16]

Queue consumers may use span links: a batch can have several message origins, but a span has only one parent. Administrative control also limits visibility. A third-party tool may remain one opaque call, while an instrumented server operated by the same team can contribute internal spans. [17][19]

Capture policy and execution provenance

Data minimization collects only evidence serving a defined purpose. Redaction removes or masks sensitive content. Apply both before export, including to error messages and identifiers. Hashing predictable identifiers does not guarantee anonymity; sampling fewer records does not make the retained payloads safe. OpenTelemetry’s sensitive-data guidance explains capture controls. [22]

A prompt is the instructions and content supplied to a model. The original user request differs from the assembled input, which may include history, retrieved passages, and tool definitions. Message order and serialization matter. Context Engineering explains assembly; observability records what the particular call actually received, within capture permissions. [23][4]

Provenance describes an artifact’s origins and production history: which inputs and activities produced it, and who was responsible. The W3C provenance model formalizes these relationships. A derived summary should retain links to its contributing sources rather than treating every document in a bundle as equal support. [27][26]

A document URL can later return different content. A reference hash can detect changed bytes when the reference is trusted, but cannot recover an unavailable document. Retained version identifiers improve comparison without guaranteeing reproducibility. [28]

Failure localization and competing explanations

Investigation begins with expected behavior, the actual user-visible result, and the completeness of available records. Compare inputs and outputs at component interfaces. Seek consequential discrepancies rather than automatically blaming the last error or the first suspicious timestamp. Several conditions can contribute to the same outcome. [31]

In the policy example, eligibility depends only on order age: the applicable policy allows 30 days, but the assistant receives an obsolete 14-day passage. The order API returns 20 days; the reply denies eligibility and claims 40 days. The source selection and the reply’s factual claim both require investigation. [32]

Fixing only one discrepancy may leave the denial unchanged: 20 days exceeds the obsolete 14-day window, while the incorrect 40-day claim exceeds the current 30-day window. A final pass/fail result can therefore conceal a repaired intermediate defect. Compare intermediate evidence as well as the final outcome. [32]

Inspect retry reasons, fallback choices, cache hits, and cancellation paths even when the final response succeeds. Grouping executions by path can reveal recurring associations, but a lower score on one path does not prove which operation caused it. Retrieval-Augmented Generation explains the retrieval-to-answer interfaces. [33][24]

Recorded evidence, replay and fresh execution

Reproduction has several meanings. Reading a historical result preserves that result. Reusing recorded dependency responses holds selected inputs fixed. Calling a model or external service again creates a new execution whose outputs may differ. Saved configuration narrows uncertainty but does not make these methods equivalent. [36][37]

A test double is a controlled substitute for a dependency. VCR.py’s none mode replays recorded HTTP interactions and rejects new ones. Other modes can make fresh requests. This protection covers intercepted HTTP only; filesystem writes, subprocesses, and other network paths need separate controls. Sandboxes and Execution Isolation covers containment. [38]

Checkpoint re-execution can run later nodes again. LangGraph’s Functional API can instead reuse persisted completed task results during resumption, while ordinary entrypoint code runs again. An unfinished task may repeat an external effect that occurred before its result was saved. Checkpointing therefore does not replace idempotency. Harness Engineering develops these recovery responsibilities. [36][39]

Task metrics, latency and usage accounting

A service-level indicator, or SLI, measures a defined service property. A service-level objective, or SLO, sets its target over a period; the allowed bad-event fraction is the error budget. Define eligible tasks, success criteria, observation period, and an accountable owner. Transport availability and correct task completion need separate indicators. [43]

Tail latency describes the slow end of a duration distribution. Record the observation boundary: client time to first response chunk differs from server time to first token, and a chunk need not equal one token. Application completion and client acknowledgment occur at still different boundaries. LLM Inference explains serving measurements. [46]

Tokens are model-specific processing units mapped to vocabulary identifiers, not reliably words or characters. Usage accounting must retain the relevant model and token categories. Tokenization explains counting. Observed usage, estimated monetary charges, and provider billing are separate records; translating between them requires an explicit accounting rule. [47][46]

The ledger contains two tasks and three attempts. Its known usage subtotal is 1,200 tokens; T2’s total and the overall total remain incomplete. With streaming include_usage, an interrupted connection may prevent receipt of the final aggregate usage chunk. Missing provider usage is not zero, even when some output text arrived. [48]

Retries can preserve completion while increasing serial waiting and dependency work. Record attempts, backoff, queue delay, rate-limit responses, and exhausted execution budgets. A timeout means the caller stopped waiting; downstream work or an external effect may continue. Cancellation and explicitly observed abandonment need separate outcomes rather than disappearing from successful-task statistics. [45][44]

Delayed outcomes and comparable populations

Feedback has its own identity and clock. Link a complaint or review to the originating task and release, retaining execution time, feedback time, criterion, judgment method, evaluator version, supporting evidence, and disagreement. A later label changes what is known about the execution; it does not change which version produced it. [6]

A cohort is a group selected for comparison. Drift is a change in observed inputs, behavior, or outcomes. Compare relevant task types and release versions at a common follow-up horizon: recent tasks have had less opportunity to receive complaints or downstream confirmations. An unresolved eventual outcome differs from a definitively missed deadline. [50][51]

A performs better within both groups but receives mostly hard tasks. B’s aggregate benefits from receiving mostly easy tasks. Aggregate rates weight groups by their observed sizes; different weights produce this reversal. Neither the aggregate nor the within-group association alone establishes a causal release effect. [53]

An outcome alert should lead to representative executions from the affected cohort plus targeted failure review. Experts can annotate traces and turn recurring problems into regression cases. Keep the representative sample distinct from the failure-enriched queue. Evals covers the validity of the resulting quality measures. [54][55]

Sampling and telemetry resource limits

Sampling retains a selected subset of telemetry. Head sampling decides near execution start; tail sampling waits for enough later evidence to select errors, slow tasks, or other paths. Tail sampling requires buffering and cannot recover upstream discards. OpenTelemetry sampling guidance explains these tradeoffs. [58]

Representative retention supports population analysis; targeted retention supports diagnosis. Preserve selection rules and inclusion probabilities where applicable, and keep reliable eligible-task counters separate. A failure-enriched diagnostic sample cannot directly estimate the production failure rate. [58]

Telemetry integrity and diagnostic coverage

Instrumentation and export can fail independently of the application. Standard simple and batching span processors export finished spans; active work may therefore be absent. A full SDK queue can drop spans, and flushing can time out. Requesting a flush does not establish complete delivery. [60]

Collector queues buffer destination outages, but capacity limits and exhausted retries can still lose records. Persistent storage helps across restarts without eliminating disk failure or prolonged-outage risk. Monitor occupancy, capacity, and export failures. A failed send may later succeed, so failed-send counts are not automatically lost-record counts. [61]

Coverage verification is a test contract for the instrumentation itself. Exercise known paths and compare expected records with received evidence. A quiet dashboard is persuasive only when the relevant execution boundaries and evidence pipeline are known to be working. [61][12]

Diagnostic telemetry is neither authoritative runtime state nor automatically a durable audit record. Where decisions require retained evidence, separately define integrity, access, retention, and recovery responsibilities. The available trace should state its coverage, not silently claim completeness. [66][18]

Open questions

1. 01

   Cross-organization trace continuity remains constrained by ownership and disclosure boundaries. Useful diagnosis needs more than an opaque duration, but shared telemetry can reveal sensitive operations. Progress would include an agreed minimal evidence contract that preserves task correlation, failure boundaries, and responsibility without exposing internal payloads. [73][19]

2. 02

   Timely quality alerts remain difficult when outcomes arrive selectively or unpredictably. Faster detection matters for containment, but immature feedback can misstate current quality. Progress would mean validated alert behavior under documented feedback delays, explicit unresolved counts, and sensitivity to which users never report outcomes. [74][52]

3. 03

   Privacy-preserving reproduction requires choosing which evidence survives deletion and minimization. Full content improves replay fidelity but expands exposure; references cannot restore deleted artifacts. Progress would be a purpose-specific capture design with demonstrated diagnostic usefulness, restricted access, and verified disposal across downstream copies. [75][38]