Engineering around learned behavior
What the application can guarantee
A model learns behavior from data: training shapes how it makes predictions or generates content, rather than a programmer specifying every decision directly. Machine Learning Fundamentals explains that distinction in detail.
The surrounding application turns a model response into a defined outcome. It can accept data that meets its declared contract or return an explicit validation failure for the caller to handle. Tests can supply dependency failures to check those handling paths, while an audit record preserves which check failed and why. These boundaries give the programmer specific behavior to inspect and improve.
Software tests check whether the application enforces those rules and handles the supplied outcomes correctly. Model evaluations examine whether its predictions serve the intended task: a permitted label can still misclassify the input, even when the model returns it consistently. Trainline describes this combination in Shipping complex AI applications: conventional software checks work alongside evaluations of model behavior. Both kinds of evidence matter, because explicitly programmed code can contain defects too.
The useful next step is to name those obligations. Once a component states what it accepts, what success means, and how failure appears, its callers have something concrete to check.
Contracts and boundaries
Specify obligations at interfaces
A type describes permitted values. A union can permit several alternatives; a function type describes accepted arguments and returned values. An interface names what a component exposes to callers. In TypeScript, an interface describes an object’s shape, but a descriptive name does not enforce its meaning: naming a string type “sanitized” does not sanitize its contents. The TypeScript Handbook makes these distinctions concrete.
A behavioral interface also carries obligations. A precondition states what must hold before an operation; a postcondition states what successful completion promises. An assertion makes a condition executable so a violation can be detected. Bertrand Meyer’s 1992 article Applying Design by Contract organized these responsibilities as an agreement between caller and implementation. At an untrusted boundary, the application must establish the relevant conditions before proceeding.
Checks establish separate properties
ExampleA structurally valid record can still fail application checks.
Read the diagram as text
- Candidate record. Identifier and proposed label.
- Validate structure.
- Reject invalid data.
- Check application constraints. Record exists; label is eligible.
- Reject ineligible proposal.
- Eligible proposal. Meaning remains to be assessed.
- Candidate record → Validate structure: inspect.
- Validate structure → Reject invalid data: fails.
- Validate structure → Check application constraints: passes.
- Check application constraints → Reject ineligible proposal: fails.
- Check application constraints → Eligible proposal: passes.
Static checking examines code before execution. Runtime validation examines values that actually arrive. TypeScript erases ordinary type annotations when producing JavaScript, so annotating a parsed response does not install a runtime check. External data still needs inspection.
A schema supplies machine-checkable rules for that inspection. Validation should produce an explicit outcome callers can handle. For example, Zod’s safeParse returns either validated data or an error, distinguished by a success flag. This keeps invalid input from being mistaken for an ordinary result. Validation establishes only the rules encoded in the schema.
Consider a model proposing a record identifier and a label, such as {"id":"t1","label":"billing"}. Structural validation checks field presence and permitted values. Application checks establish whether the record exists and whether that label is eligible for it. Neither check establishes that the text actually concerns billing. That is a question about the classification’s meaning, requiring evidence beyond its shape.
Microsoft’s July 20, 2023 TypeChat announcement applied this boundary to generated JSON: TypeScript definitions guided output, compiler checks tested conformance, and diagnostics could request repair. Applications could then perform further processing or user validation. This is machine-readable output—data another program can consume. Structured Outputs and Tool Calling develops its generation and execution mechanisms.
Separate responsibilities that change
A module is useful when it hides a decision that callers should not need to understand. David Parnas’s December 1972 paper On the Criteria To Be Used in Decomposing Systems into Modules contrasted two decompositions of a keyword-index program. Organizing around processing stages exposed decisions that could spread changes across the system. Hiding storage and other changeable decisions behind interfaces let callers remain stable. The enduring question is which knowledge a boundary keeps private.
Cohesion groups responsibilities that logically belong together. Loose coupling allows one component to change without forcing another to change. Keeping domain rules separate from persistence and messaging helps preserve that distinction. Splitting code into services does not automatically reduce coupling; these principles also guide modules inside one application.
Hide implementation decisions
ExampleThe coordinator depends on responsibilities; external representations remain behind adapters.
Read the diagram as text
- Application coordinator. Connects results without owning every implementation detail.
- Model adapter. Hides provider requests and response envelopes.
- Conversion and validation. Accepts data or returns explicit validation failure.
- Application policy. Decides eligibility from explicit inputs.
- Storage interface. Owns persistence operations and their outcomes.
- External model service.
- Database.
- Application coordinator → Model adapter: depends on model interface.
- Application coordinator → Conversion and validation: depends on data contract.
- Application coordinator → Application policy: depends on policy decision.
- Application coordinator → Storage interface: depends on persistence contract.
- Model adapter → External model service: depends on provider API.
- Storage interface → Database: depends on database operations.
A deep module hides substantial functionality behind a simple interface. Callers can use that functionality without understanding all its internal dependencies. Matt Pocock applies John Ousterhout’s deep-module principle in Software Fundamentals for AI Coding: group related behavior so developers and coding agents can locate and test it through a clear boundary. Depth concerns functionality relative to interface complexity, not a target number of files or lines.
Policy and external operations
One useful boundary separates decisions from operations. A pure function returns the same result for the same inputs and does not change pre-existing external state. Passing needed information explicitly removes hidden dependencies on mutable globals.
Gary Bernhardt’s 2012 Functional Core, Imperative Shell account applied that separation to a Twitter client. The core worked with values and decisions; the shell performed terminal, database, and network operations. This leaves decisions independently testable and reduces branching inside the code that performs effects.
For a model-assisted application, distinguish provider communication, conversion and validation, application policy, and persistence when they conceal different decisions. Dependency injection means supplying a dependency to a component instead of having the component construct it. Passing a model client as a function argument creates a substitution point; no framework is necessary.
Separation has costs. In Building Reliable Support Agents Using Effect, dependency substitution supports failure tests, but provisioning becomes difficult to trace across many layers. A wrapper earns its place by hiding a meaningful decision or providing a useful substitution point. Otherwise, it adds navigation without reducing what callers must understand.
Keep design visible during change
Those boundaries also need names that mean the same thing in discussion and code. Keep a shared design vocabulary of domain terms and a map of module responsibilities and interfaces. Use it during planning to describe which responsibility will change and how its callers will be affected. A glossary helps when its meanings guide both the conversation and the implementation.
With that boundary in place, work in steps small enough to check before extending them. Tests at the interface give feedback about behavior while refactoring changes the internals. Their usefulness still depends on choosing the right behaviors and deciding which dependencies to substitute. This connects architecture to iteration: clear, testable boundaries let developers and coding agents check a change without understanding every implementation detail at once.
Code boundaries also have behavioral limits. The 2015 paper Hidden Technical Debt in Machine Learning Systems showed how data dependencies, configuration, and feedback can couple apparently separate components. A provider adapter can contain an API change, but an unchanged interface does not establish that a replacement model makes equally useful predictions. Modularity limits implementation knowledge; behavioral compatibility still needs evidence.
State and effects
Enforce invariants when state changes
State is information that affects subsequent behavior. A transition changes that information; an invariant describes a condition valid observable states must preserve. An operation’s contract states its immediate obligations, while an invariant connects those obligations across operations. It generally must hold at the relevant entry and exit boundaries, not during every internal instruction.
Give the mutation boundary responsibility for preserving those conditions. The model supplies a proposal; the application decides whether it becomes an accepted fact. That distinction prevents a generated claim about completion from serving as its own evidence of a state change.
Reject a stale proposal
ExampleStale proposals cannot overwrite newer state.
Read version 3; propose a label.
Read the diagram as text
- Record t1.
- Snapshot: version 3.
- Proposed label: billing.
- Snapshot: version 4.
- Conditional update.
- Conflict; no write.
- Record t1 → Snapshot: version 3: earlier snapshot.
- Snapshot: version 3 → Proposed label: billing: basis.
- Record t1 → Snapshot: version 4: later snapshot.
- Proposed label: billing → Conditional update: expects 3.
- Snapshot: version 4 → Conditional update: current: 4.
- Conditional update → Conflict; no write: mismatch.
- Read. Read version 3; propose a label. Active: Record t1, Snapshot: version 3, Proposed label: billing. New: Record t1, Snapshot: version 3, Proposed label: billing.
- Concurrent change. Another writer commits version 4. Active: Record t1, Snapshot: version 3, Proposed label: billing, Snapshot: version 4. New: Snapshot: version 4.
- Attempt update. Expected 3 differs from current 4; reject without writing. Active: Record t1, Snapshot: version 3, Proposed label: billing, Snapshot: version 4, Conditional update, Conflict; no write. New: Conditional update, Conflict; no write.
Checking before saving is not always enough. Two workers can read the same state, independently produce reasonable changes, and then overwrite each other. The second save loses the first worker’s update. Coordination belongs at the shared mutation boundary; independent reads and other work can remain concurrent.
Optimistic concurrency control checks that state is still current when applying a change. For one PostgreSQL row, an UPDATE matches the record identifier and the proposal’s expected version, then increments the version in the same atomic operation. A mismatch rejects the write. Every relevant writer must advance the version; multi-row invariants need additional protection.
Track effects and unknown outcomes
A side effect is an observable change beyond a function’s returned value, such as updating a database or sending a message. A satisfactory final answer says little about whether the operations used to produce it were acceptable. Separating a proposed change from the code that applies it creates a place to inspect the proposal before effects occur.
A local database transaction groups changes so they commit together or are discarded together. An application can therefore commit an operation result and its associated status in one transaction when both live in that database. An unrelated remote API call does not automatically join the transaction. Rolling back local records cannot establish that the remote system changed nothing.
One timeout, two possible histories
ExampleThe same client observation can conceal different remote states.
Read the diagram as text
- Remote mutation requested.
- Remote state: unchanged.
- Remote state: changed.
- Completion reply lost.
- Client: outcome unknown.
- Remote mutation requested → Remote state: unchanged: request never executes.
- Remote mutation requested → Remote state: changed: request executes.
- Remote state: unchanged → Client: outcome unknown: deadline expires without reply.
- Remote state: changed → Completion reply lost: effect precedes acknowledgment.
- Completion reply lost → Client: outcome unknown: deadline expires without reply.
The remote effect and the caller’s knowledge are separate facts. A server may complete an operation while its response arrives after the caller’s deadline. Cancellation also does not roll back changes already made. Preserve an unknown outcome until authoritative status or reconciliation resolves it; neither a timeout nor a cancellation request establishes that repeating the action is safe.
Safe repetition
Idempotency means repeated application of the same logical operation has the same intended effect. It is a property the receiving implementation must enforce. A genuinely new operation with similar arguments may legitimately cause another effect. Recording a workflow step does not itself prevent duplicate external execution when an attempt completes its effect but fails to report its result.
Stripe’s idempotency contract illustrates why details matter. It saves the first executing request’s status and body under a key, including server failures, and rejects changed parameters. Validation failures and concurrent-execution conflicts before execution do not create saved results. Keys may be removed once at least 24 hours old; reuse after removal starts a new request. These cases require different handling, not one blanket “retry errors” rule.
Keep the operation identity while resolving an ambiguous result. Durable execution provides machinery for preserving work and continuing after interruption, but does not supply universal exactly-once external effects. Agent Runtimes and Harness Engineering develops persistence, retries, and recovery beyond this application boundary.
Evidence from tests
Test application behavior
A unit test checks a focused unit of behavior against specified expectations. A fixture supplies the preparation and cleanup needed for the test, such as an initial database. A test suite groups checks for execution. These terms describe how evidence is organized; suite membership does not establish that every requirement is covered.
A test double is a controlled substitute for a dependency. At the dependency seam, it can supply a fixed model response or raise an error. The test then establishes how the application handles that supplied behavior. It does not establish that the real model will produce the response, or how often. The substitute’s fidelity—its resemblance to the real dependency—limits the conclusion.
| Responsibility | Controlled condition | Observable assertion |
|---|---|---|
| Parsing | Malformed response | An explicit failure reaches the caller. |
| Validation | Unknown label | The candidate is rejected. |
| Policy | Record is ineligible | No accepted change is produced. |
| State transition | Acceptance condition fails | Stored state remains unchanged. |
| Effect handling | Remote acknowledgment is missing | The outcome remains unresolved, not failed. |
Assert public behavior rather than private storage layouts or helper calls. A test that breaks when an internal representation changes can obstruct a correct refactoring. Conversely, a test that never checks the promised state change can pass while the application is broken.
Cases and properties
Table-driven testing runs the same check over explicit input-and-expectation pairs. pytest parametrization provides this mechanism directly. It is useful when valid, invalid, and boundary cases share one assertion structure.
Property-based testing checks an executable property over generated inputs. For example, generate proposals that violate an acceptance condition and assert that rejection leaves stored state unchanged. QuickCheck separates the property from the generator, whose admissible values and distribution need deliberate choices. Both can be wrong: if the test copies a validator’s implementation, it may repeat the same defect instead of detecting it. Finite passing cases provide evidence, not a proof over every input.
Check integrations and model usefulness
Integration tests exercise real components together. Contract tests check their described boundary agreement. Pact, for example, first tests a consumer against supplied responses and records its expected interactions. Provider verification then sends those requests to the provider and checks actual responses, with required starting conditions established for each interaction. This tests whether substitute-based assumptions agree with the provider; it does not establish complete workflow correctness or imply that a particular model provider supports Pact.
A test oracle is the rule or reference used to decide whether a result satisfies a criterion. Model evaluation gathers evidence about intended task performance. Its oracle can be an exact database assertion, a model grader, or human judgment; evaluation does not necessarily mean asking another model to score an answer. Evals and Benchmarks covers task selection, grading, and comparisons.
| Check | Question answered | Still unproven |
|---|---|---|
| Unit test with a double | Does local code handle supplied behavior? | Actual dependency behavior. |
| Integration or contract check | Do these components honor tested interactions? | Untested workflows and task meaning. |
| Task evaluation | Does the system meet these success criteria? | Behavior outside the examined tasks and conditions. |
Keep live checks identifiable separately from repeatable local tests. They consume requests, depend on service availability, and may return varying results. Sampling more responses costs more work; provider errors and changed configuration defaults can also invalidate comparisons. A failed live run should distinguish an unavailable dependency from a completed trial with a poor answer.
Literal text equality is useful when literal text is the interface. The llmeval demonstration showed a mathematical answer failing because additional explanation violated an answer-only requirement. Where equivalent wording is acceptable, an exact snapshot instead makes wording changes look like defects. Assert the property callers need, and normalize whitespace only when the contract makes it irrelevant.
Failure investigation
Reproduce at a useful boundary
Reproduction recreates the conditions needed to observe a failure. A minimal reproducer removes circumstances while retaining that failure. Its purpose is to make the problem inspectable, not to preserve every detail of the original environment.
Preserve what explains the affected boundary: relevant input and initial state, code and configuration identities, dependency responses, and operation order. For consequential actions, retain the operation identity, attempts, and external evidence as well. A final conversational answer cannot reconstruct which operation was authorized or what the receiver confirmed.
Freeze the response, not the world
ExampleReplay fixes one dependency result while downstream state remains a separate input.
Read the diagram as text
- Model request.
- Live model call. Produces a new response.
- Saved response. Fixed recorded value.
- Validation and policy. The boundary under investigation.
- State and configuration. Must be arranged separately.
- Observed application behavior.
- Model request → Live model call: alternative: fresh inference.
- Model request → Saved response: alternative: select recording.
- Live model call → Validation and policy: new response data.
- Saved response → Validation and policy: recorded response data.
- State and configuration → Validation and policy: execution inputs.
- Validation and policy → Observed application behavior: result or failure.
Replay and fresh inference
VCR.py records HTTP interactions and supplies saved responses on later matching requests. Its strict replay mode rejects new requests rather than contacting the service. Applied to a model client, this freezes the response entering downstream code. Re-recording obtains a new response and changes the fixture.
Rerunning inference asks the model to generate a new response. In OpenAI’s archived November 2023 example, fixed seeds and request parameters improved repeatability but did not guarantee identical responses, even when the server fingerprint—a marker for model weights, infrastructure, and other server configuration—also matched. Saving the returned response preserves the historical value; saving those settings alone does not.
Saved output also does not restore database contents, clock values, or concurrent scheduling. Choose the boundary according to the failure: replay can isolate a parser defect while leaving a timing-dependent race unreproduced.
Preserve only necessary diagnostic data. VCR.py supports configured filtering of headers, request fields, and response bodies; those controls do not automatically make a recording safe. Removing credentials is useful, but removing failure-relevant content can destroy the reproducer. Check that the filtered fixture still exhibits the intended failure.
Observability supplies recorded signals for understanding execution, including calls, timing, and state transitions. Observability develops the instrumentation and retention infrastructure. Here, the goal is narrower: preserve enough evidence to run a useful diagnostic experiment.
Test explanations of a failure
A symptom is an observation; its cause is an explanation to test. State expected and actual behavior, propose competing explanations, and choose an intervention that distinguishes them. Recent changes suggest hypotheses, but timing alone does not establish causation. Keep unrelated conditions fixed where possible: changing code, model settings, and data together makes the result difficult to interpret.
| Hypothesis | Controlled experiment | Supported conclusion |
|---|---|---|
| The reply violates the input contract. | Validate the captured reply. | Rejection establishes nonconformance, not its cause. |
| The application mishandles invalid input. | Inject that reply through its client interface. | A crash violates the promised rejection behavior. |
| The repair fixes rejection behavior. | Compare old and repaired code using that reply. | An explicit failure supports the repair for this case. |
An upstream violation and a downstream defect can coexist. Locating the first observed violation narrows the investigation; it need not explain every failure. A fresh model response that happens to be valid does not exercise the broken rejection path. Repeated calls can characterize variability, but one successful rerun cannot establish that the application was repaired.
Preserve the finding
Delta debugging makes difficult cases smaller through repeated tests. Zeller and Hildebrandt’s February 2002 paper automated removing input fragments while retaining a failure, motivated partly by the effort of simplifying Mozilla bug reports. Its ddmin procedure reaches a case where no single remaining element can be removed successfully. That is neither necessarily the globally smallest case nor an explanation of the defective code. The failure check must keep identifying the same problem.
A regression is previously working behavior that breaks after a change. Turn the confirmed defect into a test of the missing obligation, so later changes retain the finding instead of requiring someone to rediscover it.
Controlled change and release
Make changes reviewable
Version control records revisions and exposes differences for inspection. A focused change has one coherent purpose, with related tests and enough context to assess its consequences. Small does not mean a universal line limit. Keep each submitted change working; separate substantial refactoring from behavior changes where practical, but include mutually dependent edits needed to make one change understandable.
Refactoring changes internal structure while preserving observable behavior, including relevant state effects. A deliberate behavior change makes a different claim and needs evidence for the new obligations.
| Record | Content |
|---|---|
| Intent | The behavior to preserve or change. |
| Difference | Changed code, instructions, model settings, dependencies, or relevant data artifacts. |
| Affected obligations | Changed assumptions, interfaces, state transitions, and effects. |
| Evidence | Checks performed, versions exercised, and remaining uncertainty. |
Continuous integration automatically builds and checks integrated changes. Its evidence extends to the checks actually run. Prompt edits and schema changes need appropriate regression and boundary checks too, even when they happen outside application code. Scheduled failures also need to become visible rather than silently leaving old output in place.
Review examines whether assertions would fail for broken behavior, whether concurrency assumptions hold, and whether complexity serves a purpose. Passing tests do not answer those questions automatically. User-facing changes may also need a demonstration because their consequences are difficult to infer from a diff.
Review has a collaborative role as well. How to Kill the Code Review argues that architectural discussion, mentorship, and shared understanding must survive changes to implementation verification. Automated evidence can support that discussion; it does not establish that the team agrees on the intended behavior.
Identify the tested release
A release artifact is the packaged output deployed to an environment. Its identity differs from a source revision, a configuration revision, or a moving label such as “canary.” Verification should identify the exact release it exercised, including relevant configuration.
| Identity | What it establishes |
|---|---|
| Source revision | Which source was selected. |
| Build inputs | Dependencies, environment, and instructions used. |
| Artifact identifier or hash | Which packaged output is being tested or deployed. |
| Configuration revision | Settings paired with that artifact; include instructions and model settings where relevant. |
| Verification record | Which checks exercised that release and with what outcomes. |
A reproducible build recreates identical specified artifacts from the same source, environment, and instructions. Comparing bytes or cryptographic hashes checks that property. A source revision alone is insufficient. Retaining a deployable artifact and being able to rebuild it are distinct capabilities; neither reconstructs mutable runtime state.
Mainline tests may not cover a release assembled with selected fixes. Check the actual release revision and packaged artifact where their behavior matters.
External dependencies may not remain available. Anthropic’s model lifecycle documentation distinguishes deprecated models, which still function pending retirement, from retired models, whose requests fail. Partner schedules can differ. Saving an identifier therefore does not preserve the service behind it; availability must be checked when planning restoration.
Bound exposure and verify recovery
A canary release exposes a change to limited use before expansion. Compare candidate and control metrics separately: aggregate results can hide candidate failures. Choose representative traffic and an observation period that exercises relevant conditions, with acceptance and stop criteria set beforehand. There is no universal percentage or duration. Limited exposure reduces affected use; it does not make each harmful outcome acceptable.
A feature flag is runtime configuration that controls behavior. Disabling a capability differs from restoring an earlier configuration. Agents Need Feature Flags emphasizes checking controls during active work, at the next decision point, rather than only when a session starts. That can prevent further actions; it does not establish interruption of an executing tool or reversal of completed effects.
Restoration has prerequisites
Restoring software and remediating completed effects are separate operations.
Read the diagram as text
- Prior release.
- Current state.
- Available and compatible?.
- Restore and verify.
- Contain and investigate.
- Completed effects.
- Separate remediation.
- Prior release → Available and compatible?: input: availability.
- Current state → Available and compatible?: input: compatibility.
- Available and compatible? → Restore and verify: control: yes.
- Available and compatible? → Contain and investigate: control: no.
- Completed effects → Separate remediation: requires remedy.
Restoration and remediation
Rollback restores a previously deployed version or configuration. Retain a concrete restoration target and its required configuration. After switching, verify the selected artifact, settings, and relevant observable behavior; a completed deployment command alone is not evidence that the intended service has recovered.
An older executable must also work with current state. Amazon’s rollback-safety account illustrates a new version writing compressed data its predecessor cannot read. Restoring that predecessor creates another failure. Establish compatibility before treating an available old artifact as a usable recovery target.
Completed effects require separate treatment. Compensation performs a new, business-specific action to counter earlier work. It may incur costs, preserve intervening changes, or produce a different final state. It can itself fail. Some effects require an acceptable remedy or human intervention because no operation can erase what already happened.
The release decision therefore depends on consequences as well as evidence. Identify what live exposure can safely teach, what must be established before exposure, and what remains after stopping. A clinical utterance, for example, cannot be withdrawn from the patient’s experience by rolling back software. Better offline task results and passing application tests are valuable, but neither supplies a recovery path for an irreversible action.
Open questions
Behavioral compatibility across model replacements remains difficult to specify. Stable interfaces contain implementation changes, while learned behavior can shift across inputs and downstream consumers. Progress would make those behavioral dependencies easier to identify and test without promising complete isolation.
Evidence of task usefulness remains bounded by its success criteria. Precise checks miss omitted requirements; open-ended graders need calibration. Progress would improve agreement with meaningful human outcomes while making each check’s limits explicit.
Recovery becomes harder when dependencies cannot be retained and effects cannot be undone. Progress would provide tested restoration targets and explicit remedies that preserve service continuity without misrepresenting mitigation as reversal.
