AI Engineering 201: The Rest of the Owl

How do you get from a model that runs inference to a product that delivers value? Charles Frye opens with the drawing-an-owl problem: two simple outlines provide the starting geometry, but leave almost all the work of producing the finished animal unexplained. Inference is that starting geometry. The surrounding application—and the process for improving it—is the rest of the owl.

Language user interfaces, or LUIs, give that application layer a useful organizing idea. Terminals required users to learn a computer’s special language. Graphical interfaces recruited spatial intuition and vision, making computers approachable beyond specialist settings. Language models offer another change: users describe what they want, and the system translates that intention into something the machine can do.

Frye cites Sam Altman’s enthusiasm for language interfaces, but the ambition long predates foundation models. ELIZA provided a conversational psychotherapy setting in the 1960s. SHRDLU accepted instructions such as “Pick up a big red block” inside a graphical blocks world. Ask Jeeves promised a language interface to the internet, and Alexa brought spoken requests into everyday devices. The opportunity with foundation models is to make this interaction work across many domains, rather than engineering each narrow environment separately.

Sequoia’s Generative AI’s Act Two describes foundation models becoming components of comprehensive solutions. Honeycomb’s query assistant makes that concrete. Instead of assembling a query in an unfamiliar interface, a user can ask for slow requests, errors, or latency distribution by status code. SQL itself once carried the aspiration that ordinary business users could write queries; a request like “Can you show me slow requests?” comes closer to the interaction people actually want.

Adding language to an existing feature is the immediate opportunity. Over a longer product horizon, the interface may change the machine around it, just as terminal, desktop, and mobile interfaces supported different forms of computing. Google’s SayCan illustrates the direction: ask for a water bottle instead of opening an app and navigating several dropdowns. Frye notes that the robot demonstration plays at 4× speed, tempering the impression of readiness. The desired interaction is simple; delivering it still requires substantial engineering.

Retrieval-augmented generation chatbots have become the to-do-list application of language interfaces: a common first project. Their underlying requirement is durable. A generally knowledgeable model does not automatically know the user, the organization, or the documents relevant to a particular question. Retrieval supplies that missing context before generation.

The familiar implementation follows a short sequence:

1. Collect and store the relevant material.
2. Build an index, often using embeddings.
3. Retrieve material relevant to the incoming question.
4. Insert that material into the model’s prompt.

Frye’s Full Stack Deep Learning Discord bot uses lectures, selected papers, and the course website. Its purpose is to answer from those materials and their authors’ perspectives, rather than offer another generic answer about language models.

Vector search became the default partly through convenience: developers already using OpenAI could obtain embeddings through the same library. Dot products and softmax also felt familiar to people working with transformers. That explains the proliferation of chat-with-your-documents examples, but the lasting requirement is broader than vector storage: find useful information and place it where the model can use it.

Frye describes Redis vector search as awkward to use but workable. The larger point is that fast retrieval from a large store is already a database problem; adopting RAG does not automatically require a separate vector database.

Free-text questions are heterogeneous, so one retrieval method may not cover them well. A useful combination is keyword and vector search plus metadata extracted by a language model, which can then drive direct filters. Frye recommends The Data Quarry’s database-oriented writing for this perspective. In the audience clarification, he returns to the invariant: search saved information from the outside world and put the result into the prompt. Ordinary database techniques and specialized vector stores are both implementations of that pattern.

How does retrieval differ from the history in a conversation with GPT-4? At the API boundary, the application constructs the message history. Earlier user and assistant messages are supplied inputs, not necessarily an immutable record of events. Frye points out that fabricated assistant history can also become a jailbreak vector, while warning against violating service terms.

In the implementations he describes, retrieved information often enters the system prompt, either once or refreshed for every user turn. He attributes the model’s attention to that message to its training. An application could instead pretend that retrieved facts were earlier user speech, but he considers that unusual. The meaningful distinction is ownership: retrieval lets the programmer supply additional information the model should have; conversational history records what the user and assistant have said.

Another audience question challenges the familiar rule that retrieval supplies knowledge while fine-tuning supplies style: what if the training set contains 100 GB of research papers? Frye treats the rule as conditional on the scale and method of adaptation. In discussing small hosted fine-tunes, an audience member suggests a 10,000-row limit and Frye tentatively agrees; that exchange should not be treated as a current API limit.

LoRA adapts a model through low-rank updates to its layers. Frye’s intuition is that a small adaptation often changes which existing computations matter most. A capable model already knows how Homer Simpson or Rick Sanchez tends to speak; a character chatbot makes that behavior central rather than incidental to its next-token probabilities. This is an explanatory intuition about small adaptations, not a proof that low-rank updates cannot teach facts.

Training on a large textbook collection moves into a different regime. Frye distinguishes it from the small fine-tunes behind the style-versus-knowledge rule and points to coding-oriented Llama derivatives as examples of changes extending beyond style, including programming and library knowledge. The practical question is how much data and what kind of update the model receives, not simply whether the work is called fine-tuning.

A question about Neo4j brings the discussion to graph databases. Frye, who does not claim database specialization, relays specialists’ preference for representing graphs in Postgres. The concrete scaling difficulty is partitioning: an arbitrary graph has no obvious cut, and new edges can change which partition would be best. Rebalancing can therefore resemble a migration rather than a simple redistribution of independent records.

That difficulty may not dominate an application using a graph as external model memory. Frye sketches workloads with megabytes or gigabytes of data and tens or hundreds of thousands of requests per second to ask whether distribution across many machines is even the relevant concern; these are illustrative workloads, not reported capacity measurements. Knowledge graphs have a natural fit with language-model memory, but he has not yet seen a decisive application that settles the choice.

Metadata filtering exposes a more immediate retrieval problem. Suppose the user wants crab restaurants in San Francisco. A Boolean flag, category, or location constraint carries a different meaning from a soft similarity score.

Post-filtering is straightforward, but it can answer a subtly different question from the one the user asked.

Efficient pre-filtering affects index construction: the search system must support restrictions such as location while still finding similar items quickly. Different databases support different kinds of restrictions. Frye mentions Vespa and Weaviate as having good reputations here, without claiming a comprehensive comparison. The filter is part of the retrieval design, not merely a cleanup step after similarity search.

A language model produces text, but an application usually needs something it can act on. Connecting generated text to another system’s inputs turns the model into a language interface for that system. Structured outputs make that connection more robust by narrowing what the model is allowed or encouraged to return.

Several approaches work at different points in generation:

• Prompting: ReAct-style examples encourage the model to request an external API. When the request appears, the application intercepts it, calls the real API, and supplies its result instead of letting the model invent the response. Frye also recounts Riley Goodside’s dramatic prompt threatening that an orphan will die without valid JSON—a memorable example of persuasion, with effectiveness reported anecdotally.
• Fine-tuning: Gorilla, Toolformer, GPT-J, and adapted Llama models illustrate training for tool use or structured generation. Frye is skeptical about how much hosted OpenAI fine-tuning would help with arbitrary structure; OpenAI’s contemporary fine-tuning announcement did list improved formatting as a use case, which is distinct from guaranteed schema compliance.
• Retries: Validate the output, explain the mismatch, and ask for another attempt. Frye highlights Guardrails’ XML-based interface and speculates that it could fit Claude well.

These approaches respectively shape the request, the model’s learned behavior, or the response to a failed validation.

Grammar-constrained sampling intervenes before an invalid token is emitted. In llama.cpp, a grammar can rule out continuations that violate the required format. Forbidden tokens receive zero probability, equivalent to setting their log probability to negative infinity:

Context-free grammars can constrain JSON, code, and other formats that conventional systems know how to parse. Tight decoding control can make a model more useful for a particular integration even when another model has stronger general capabilities.

The audience raises TypeChat, which Frye has not used, then asks whether a GPT response could pass through Gorilla for formatting. He considers structural conversion an easier task than producing the original answer, making it a plausible job for a smaller model. His example pairs GPT-4 for primary generation with GPT-3.5 for error handling. Guardrails-style repair and LangChain-style chaining offer ways to separate those responsibilities.

An audience correction sharpens the distinction between observing log probabilities and changing token probabilities through logit bias. Those are separate capabilities. Frye agrees that token biasing may still be available through the API, then identifies the operational problem: a grammar’s valid next tokens change as generation proceeds. A client enforcing those changes through successive requests would need a network round trip for each token instead of receiving a whole continuation in one request.

He then looks beyond one-token decisions. A richer decoder might explore several grammar-valid continuations and choose among them, using something like Monte Carlo tree search. Frye presents this as a prospective direction likely to emerge first with open models, where developers control the inference loop. The historical API discussion here predates OpenAI’s later Structured Outputs, which introduced server-side schema-constrained decoding in August 2024. That later capability changes the implementation options, while still leaving correctness of the generated values as a separate problem.

Training for particular tools creates a generalization problem. The Gorilla example targets APIs from Torch Hub, TensorFlow Hub, and Hugging Face; knowing those tools does not imply knowing every possible tool. A shared interface changes the problem. OpenAI’s function-calling interface uses JSON Schema to describe arguments, allowing a model trained on the common format to connect to many tools through small application adapters. Grammar constraints can further enforce the format. The integration starts to resemble ordinary web development: pass a JSON object to code that knows what to do with it.

The function need not actually execute. Suppose the application needs to classify a message as spam or not spam so it can render an appropriate interface. Describing a fictional function whose argument is that classification makes the model produce the decision in a machine-readable form. The application can consume the arguments directly.

Here is a JavaScript representation of that boundary. The schema describes the requested result; the consumer validates and uses the returned arguments without invoking a classifier function:

const classificationFunction = {
  name: "report_spam_classification",
  description: "Report whether the input message is spam.",
  parameters: {
    type: "object",
    properties: {
      label: {
        type: "string",
        enum: ["spam", "not_spam"]
      }
    },
    required: ["label"],
    additionalProperties: false
  }
};

function readClassification(argumentsJson) {
  const result = JSON.parse(argumentsJson);
  if (
    result === null ||
    typeof result !== "object" ||
    Array.isArray(result) ||
    Object.keys(result).length !== 1 ||
    !["spam", "not_spam"].includes(result.label)
  ) {
    throw new Error("Invalid spam classification");
  }
  return result.label;
}

Jason Liu’s Instructor packages this broader pattern of extracting structured results. The useful artifact is the argument object, whether or not there is a downstream function to call.

The request still has to fit the function-call interface and its learned conventions. Frye uses functional programming to explain the flexibility: even a constant can be represented as a function, and optional results can have a structured representation such as a maybe. He also describes a DAG-constructor schema that yields a graph of proposed function calls rather than one call. Producing that graph is a planning result; executing it remains an application responsibility.

Structured results complicate streaming. Frye describes the function-call pattern as better suited to back-office data processing than immediate user-visible text. A pipeline could begin downstream work as soon as enough relevant upstream information arrives, but it needs a reliable boundary for deciding what is complete. Unix pipes have newline-delimited records; partial function-call arguments do not automatically offer an equivalent. Recovering useful incremental behavior is therefore a pipeline-design problem, not something solved merely by enabling a token stream.

Retrieval and structured extraction are recognizable natural-language-processing tasks. Agents add something closer to the everyday image of artificial intelligence: behavior that develops over time. Generative Agents combines streams of memories with reasoning flows inside a simulated environment, producing characters whose personalities and interactions evolve.

Voyager makes the accumulation particularly concrete. Its Minecraft agent writes JavaScript against an environment API, builds routines for activities such as mining wood or attacking zombies, and retains those routines as reusable skills. It also develops its own curriculum. Frye highlights its progress on mining diamonds, a task that had been a significant reinforcement-learning challenge. Memory here includes executable procedures, not just a transcript of previous observations.

Accumulating information, skills, and tools does not remove the central obstacle: reliability. Structured outputs can help make individual interactions less fragile. Frye places that suggestion in the research landscape of the talk, when relatively little published agent work had yet incorporated the improved function-calling capabilities he was discussing.

Cognitive Architectures for Language Agents, or CoALA, provides a vocabulary for comparing the design choices scattered across Voyager, ReAct, and Generative Agents. The work from Tom Griffiths’ group at Princeton revisits production systems and cognitive architectures associated with earlier AI research, including procedural, semantic, and episodic memory.

Those older systems supplied explicit organization but struggled with general knowledge and common sense. Language models supply much of that knowledge while lacking persistent memory and an explicit organization for cognition. Combining them lets the architecture define how the agent observes, remembers, reasons, selects an action, and interacts with the world.

Internal actions are an especially useful distinction. An agent can spend time reasoning, update long-term memory, or revise its decision procedure instead of immediately acting on the external environment. Making those choices explicit helps separate memory design, external grounding, and action selection. CoALA also presents a research agenda built from combinations of language-model techniques and cognitive-architecture ideas. Frye closes the architecture discussion by recommending Eugene Yan’s writing on patterns and anti-patterns.

An architecture explains how to build the system. Engineering also needs a process for improving it. Frye describes the prevailing approach as shipping to learn: the data engine or data flywheel emphasized in Full Stack Deep Learning and by Andrej Karpathy. Collect real-world data, find failures, improve the system, and repeat.

Charity Majors’ comparison with conventional software testing makes the consequence clear. Software teams describe tests as the gate to production; ML teams often need production observations to discover what their tests should contain. Frye jokes that this becomes “oops, all regression tests.” Monitoring therefore belongs at the beginning, across user behavior, performance, cost, and bugs.

The Discord bot supplies a small, concrete example. Users asked whether emoji reactions provided feedback, whether its dataset included its own source code, what it did, and even who was a good bot. These meta-questions had not seemed important while building the application. Seeing them in real conversations led Frye to add prompt handling for that class of input. He logged the interactions to Gantry, examined thumbs-up and thumbs-down examples, and read the entire collection because it was only a couple hundred rows.

Latency quantiles matter because a user experiences a sequence of requests. A system that is fast for most individual requests can still repeatedly expose engaged users to its slow tail. Frye emphasizes watching extreme quantiles such as p99 rather than stopping at a satisfactory p90. Throughput is a separate signal: how much work the system completes is not the same as how long an individual request takes.

Randomly sampled profiles and traces help explain the aggregate numbers and locate throughput problems. When serving inference yourself, maintaining that throughput becomes a direct responsibility requiring more model- and hardware-specific investigation.

The tooling landscape spans several established categories:

• General observability: Datadog, Sentry, New Relic, Honeycomb, or custom OpenTelemetry-compatible instrumentation.
• ML tooling: Weights & Biases, where Frye previously worked, alongside Fiddler, Arize, and Gantry. Their emphases differ between training management and production monitoring.
• LLM application tooling: LangSmith from LangChain and Langfuse.

Frye does not identify a settled winner. His preference is for tools that make unusual queries over unstructured data easy; he cites Weights & Biases and Gantry, while noting that general-purpose tools may require custom analysis or notebooks.

Langfuse’s documentation chatbot offers an appealing way to inspect the product: use the chatbot, then examine the records it creates in the monitoring interface. Frye recommends that live-demo pattern but skips his planned walkthrough for time. Collecting those records is still only the beginning; the next question is whether they let you explain and improve the application’s behavior.

Observability asks whether external observations reveal what is happening inside a system. In software terms, can you debug from logs without entering a live debugger? This becomes essential when the system’s complexity exceeds your ability to anticipate its failures. For a language-model application, dropping a breakpoint deep inside GPT-3 or GPT-4 is not a practical way to diagnose a bad answer. The application must expose enough evidence to locate and fix problems.

The harder obstacle is recognizing that an answer is wrong. If correctness is unclear, both failure diagnosis and verification of a fix become unclear. Frye points to evaluation concerns raised by Anthropic and AI Snake Oil authors Arvind Narayanan and Sayash Kapoor, and interprets OpenAI’s open-sourcing of its eval framework as another sign of the problem’s difficulty. The False Promise of Imitating Proprietary LLMs provides a concrete warning: convincing response style can create impressions of capability that targeted evaluations do not support. That finding concerns the imitation methods studied, rather than every possible form of distillation.

Evaluation begins with sustained attention to data. Frye cites Stella Biderman and Jason Wei on the value of inspecting examples, understanding evaluations, and building internal tools for that work. Open-ended generation becomes especially difficult when neither inputs nor outputs have much structure. Josh Tobin’s flowchart from the Full Stack LLM Bootcamp offers a way to avoid treating every task as that hardest case.

A multiple-choice answer can have a definite correct label. A short written answer may instead need reference matching. Even without either, a previous output supports a narrower question—whether the new answer is better—and explicit feedback supports checking whether the requested change happened. Fully unstructured evaluation is the residual case after these sources of evidence are exhausted.

Elicit’s scientific-paper extraction work illustrates iterated decomposition:

1. Start with a task that works end to end.
2. Inspect a failure.
3. Split the task so that the failure occurs inside a simpler subtask.
4. Optimize and evaluate that subtask.

This can turn an opaque generation problem into smaller questions with clearer answers. The tradeoffs remain real: chained calls add latency, and a conversational response does not always admit a useful decomposition.

When natural-language evaluation remains unavoidable, a few trusted cases are a reasonable beginning. Passing them does not explain what to do when another example fails, but the collection can grow: add production failures, preserve them as regression cases, and run the suite in a GitHub Action. This is how a small manual check develops into the data engine discussed earlier.

User feedback is more useful when it fits naturally into the product. Frye points to Midjourney as an example of learning from revealed preferences: what users choose to do can carry more information than a compulsory feedback form. Honeycomb takes the idea further by connecting its query assistant to downstream business objectives. The measurement chain runs from the feature, through user behavior, to organizational goals.

Paid annotation teams provide another source of feedback. They can perform deliberate reviews that ordinary users will not reliably stop to provide, but annotation is itself a task whose cost and quality need attention.

OpenAI’s use of paid human feedback provides a point of comparison, but language models can also help annotate and improve data. Frye characterizes GPT-3.5 Turbo as roughly a median crowdworker and GPT-4 as roughly a 90th-percentile crowdworker on textual annotation tasks, with lower cost at equal performance. He supplies no benchmark conditions for those comparisons, so they are a reason to investigate model-assisted annotation on the actual task, not staffing estimates. A hybrid arrangement could have a smaller human team supervising language-model annotators.

The talk ends with the unfinished product problem still open. The right interfaces and user experiences are not settled, and neither are the methods for making these systems reliably correct. Filling in the owl means discovering those missing steps through working applications: try an approach, observe what happens, improve it, and share what works so the next builder starts with more than two circles.