Compilers in the Age of LLMs

An OpenAI client in the codebase, Hugging Face tabs in the browser, several playground repositories, and an agent that strings together HTTP calls: this is a familiar starting point for an AI application. Adding a new open-source model often means writing another Dockerfile, running another container, and finding infrastructure to host it. Making that model available to an agent adds another tool to its context, perhaps exposed through MCP. The model may be easy to download; adopting it still expands the application’s infrastructure.

The desired interface is much smaller: an OpenAI-style client that accepts a model name and works with minimal application changes. Execution might be local or remote, backed by llama.cpp or TensorRT. Yusuf Olokoba’s proposed route to that experience is a Python compiler: take plain inference code and turn it into a self-contained native artifact for targets ranging from cloud servers to Apple Silicon. His introduction also frames LLM-generated code as something to constrain with verification and testing.

Two requirements motivate that choice. First, developers should be able to bring an internal model or one found on Hugging Face or GitHub and obtain something their application can execute. Changing the model argument when a new OpenAI model appears provides the interaction to emulate. A compiler supplies the missing transformation: Python inference code goes in; an artifact suited to the user’s execution environment comes out.

Second, Olokoba expects hybrid inference to become common: small models on devices or at edge locations cooperate with larger cloud models that offer stronger reasoning. That forecast changes the deployment target. A server-side Python process in a Docker container is no longer sufficient for every location where inference should happen. Native execution offers a route closer to the hardware, with fewer runtime dependencies and greater control over responsiveness.

The running example is Google’s EmbeddingGemma. A Python function takes a list of sentences and returns an embedding vector for each sentence—together, an embedding matrix. Those vectors support text search, retrieval-augmented generation, and retrieval of individual document passages. Olokoba describes the model as having 270 million parameters; Google’s release specifies 308 million. His reason for choosing it is its suitability for both cloud GPUs and consumer devices, rather than a measured speed comparison in this demonstration.

The intended journey has three stages: translate the Python function into equivalent C++ or Rust, compile it with the model and its dependencies, and expose inference through client.embeddings.create. Before generating native code, however, the compiler needs a graph describing what the function does. That graph becomes the basis for translating the function operation by operation.

The first prototypes used PyTorch’s compiler tooling, including torch.compile and torch.fx. Two details help separate the tools: FX arrived in PyTorch 1.8, before PyTorch 2, and its basic symbolic tracer records operations through Proxy objects. Fake tensors, which carry tensor metadata without allocating the represented data, belong to related compiler analysis machinery. Olokoba describes his prototype in terms of running with fake inputs to obtain an operation graph.

That approach exposed two problems for the team’s broader Python target:

• Library coverage: inference functions may include NumPy operations, OpenCV calls, and other non-PyTorch data types. Supporting those paths would require extending the tracing machinery.
• Input representation: a data-free tensor can retain a tensor’s shape and other metadata, but constructing suitable stand-ins for images, dictionaries, and arbitrary objects was harder in their prototype.

These were obstacles to the team’s chosen implementation, rather than a universal requirement that every FX trace use fake inputs.

The next experiment asked an LLM to produce the trace directly. Structured outputs supplied the constraint: give the model source code and a response schema, then ask for a graph in that schema. Olokoba reports almost 100% accuracy for LLM trace generation in internal testing, but says it took too long. He supplies neither the test size and correctness criterion nor a latency measurement. The limiting factor was therefore the reported cost in time, even when the generated structure looked promising.

The eventual tracer instead analyzes Python’s abstract syntax tree, using internal heuristics to construct an intermediate representation, or IR. For the embedding example, the relevant graph contains the string-list input, a tokenizer call, a model call, and the returned embedding vectors. This gives the compiler a structured description of the computation without making an LLM request the tracing mechanism.

An operation graph is not yet enough to emit native code. Python can bind x to an integer and then rebind it to a string. An individual variable in the generated C++ or Rust has a fixed type, whether that type is written explicitly or inferred. The compiler must therefore determine native types for the intermediate values in the Python function.

The example begins with a list comprehension that prepends a task-specific prefix to each input sentence. The essential operation can be isolated in Python like this:

def make_prompts(text: list[str], prefix: str) -> list[str]:
    return [prefix + sentence for sentence in text]

In the demonstrated function, the prefix comes from a global task-prefix map. The function’s annotation establishes that each member of text is a string, and the map supplies string-valued prefixes. The remaining question is the native type of prefix + sentence.

Type propagation answers that question using an implementation of the operation for the known operand types. Here, Python’s operator.add means string concatenation. A corresponding C++ operation can have this signature and body:

#include <string>

std::string add_strings(
    const std::string& prefix,
    const std::string& sentence
) {
    return prefix + sentence;
}

The function takes two strings and returns a string. Its return type supplies the missing type information for the graph node.

The compiler combines information from the Python function signature with native type information for constants and operations. Once the concatenation result is known, that information can flow onward through the comprehension and subsequent operations. Repeating the process determines types for intermediate values across the IR.

Type propagation moves a substantial requirement into view: every supported Python operation or function call needs a corresponding native implementation. Knowing how string addition behaves does not supply the implementations for other operand types, library functions, or tensor operations. Olokoba accepts that requirement; the compiler needs those building blocks.

His first reason for considering it tractable is that source-code variety comes largely from combinations of operations. Many different functions reuse the same elementary operations in different sequences. The slide illustrates this with operator.add, operator.pow, and operator.matmul: Function A and Function B contain the same operations in different orders. Native implementations can likewise be composed into different generated functions.

The elementary set is still large. Addition, subtraction, and exponentiation are only the beginning; useful inference functions also rely on NumPy and PyTorch operations. This is where LLMs return to the compiler design. Instead of requiring engineers to handwrite each native equivalent, the team uses LLMs to generate C++ and Rust implementations. The role is narrower than generating a fresh trace for every user function: produce reusable implementations of operations that the compiler can compose.

With native operation implementations available and types propagated through the graph, the compiler has the information needed to emit typed C++ source. Olokoba presents this as the point where the IR can become compilable native code. The side-by-side walkthrough follows the original computation: build the prefixed prompts, tokenize them into IDs, run the model, and return the embedding matrix. The deployment language changes while the sequence of inference operations remains recognizable.

The next step uses an ordinary native compiler. C and C++ toolchains exist across a broad range of devices, which makes native source a useful portability layer. The result in this example is a dynamic library, or shared object, that another process can load and execute. Portability here means building for supported targets, not shipping one native binary that executes on every architecture; Olokoba’s companion article also describes target-specific builds and incomplete Python language coverage.

The host application in the demonstration is JavaScript running on Node.js. A foreign-function interface, or FFI, bridges JavaScript and the compiled library. The binding must identify the library built for the host’s operating system and architecture, name the exported function, and declare its native signature. Those declarations tell the binding how to call the function across the language boundary.

The invocation procedure is then straightforward:

1. Locate the compiled library.
2. Load it through the FFI bindings.
3. Invoke the exported inference function with the expected inputs.
4. Receive the embedding matrix.

The team standardizes this scaffolding across compiled functions. The particular binding implementation remains open-ended in the explanation; the demonstrated result is a JavaScript call that obtains embeddings from the native library.

The final layer restores the interface that motivated the compiler. A Client class contains an Embeddings class whose create method exposes the familiar client.embeddings.create path. That method receives the model name and resolves it to the path of the compiled binary. The FFI machinery already built for Node.js loads the library and executes its inference function.

One step remains after execution: reshape the embedding matrix into the result structure expected from the OpenAI-style client. That separates three responsibilities cleanly: model-name resolution chooses an artifact, FFI invokes native inference, and response conversion presents the result to the application. The final slide brings these layers together with an embedding request containing a question about the capital of France.

The application can now request embeddings through a familiar method while the implementation runs a compiled open-source model inside the host process. Olokoba closes with the ambition of extending this access to any open-source model expressible through a Python function. Within the compiler’s supported operations and build targets, the completed example shows what that means concretely: a model name selects a native library, the library returns embeddings, and the client presents them in the expected shape.