# Substrate Launch: the API for modular AI

Rob Cheung

AI Engineer World's Fair 2024

[Talk page](https://ai.engineer/talks/x8HbIJh2wpQ-substrate-launch-api-modular-ai) · [Video](https://www.youtube.com/watch?v=x8HbIJh2wpQ)

## Substrate Launch: the API for modular AI

Substrate couples a graph SDK with a coordinated inference engine, making dependencies explicit and reducing the overhead of moving between model calls.

Basic familiarity with model API calls and Python is enough to follow the graph example.

### When one inference run is not enough

[0:00](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=0s)

What does it take to turn a model into a product that reliably summarizes documents or solves coding problems end to end? Substrate entered this problem space after about a year working with private clients. At his [June 2024 AI Engineer World’s Fair session](https://ai.engineer/worldsfair/2024/schedule), Rob Cheung described the launch the previous week as the beginning of the team’s public introduction.

The starting observation is that successful AI products often use multiple inference runs, sometimes combining different types of models to perform a specific task. Even document summarization can benefit from a system of calls; end-to-end coding makes the need for a logical structure more apparent. The foundation model supplies a capability, while the surrounding system organizes how that capability gets used.

![Cream slide with centered text stating “More than one inference run to accomplish a specific kind of task really well” and a Substrate logo.](https://ai.engineer/api/talk-essay-media/fc30515077418a68d5787b5d91599749d398aa6c471571680fc3c304a075b6ac)

More than one inference run to accomplish a specific kind of task really well.

**An explicit structure makes intermediate behavior inspectable.** Substrate’s premise is that modular intelligence will be more effective than monolithic intelligence because developers can understand the system’s parts and relationships. That makes it easier to debug a failing step, extend the workflow, and evaluate intermediate decisions. When the decision tree is explicit, evaluation can ask where an error entered the process instead of judging only the final answer.

### Describe the graph, then execute it together

[1:50](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=110s)

Substrate’s proposed combination is speed, flexibility across AI products, and scale by default. Its architecture couples two components: a **developer SDK** for describing a computation graph and an **inference engine** built to execute that graph. Nodes cover image generation, speech transcription, text generation, JSON, embeddings, and code execution. The graph captures both the tasks and their dependencies.

A document-summary workflow illustrates the dependency model. One node generates a summary; another turns that summary into a title. In the [Substrate Python SDK](https://github.com/zocomputer/substrate-python), a future output can become part of a downstream prompt:

```python
import os
from substrate import Substrate, ComputeText, sb

substrate = Substrate(api_key=os.environ["SUBSTRATE_API_KEY"])

document = (
    "The team moved the release to Friday. "
    "The extra time will be used to test the payment flow."
)
summary = ComputeText(
    prompt=f"Summarize this document in one sentence:\n{document}"
)
title = ComputeText(
    prompt=sb.concat(
        "Write a short title for this summary:\n",
        summary.future.text,
    )
)

response = substrate.run(summary, title)
print(response.get(summary).text)
print(response.get(title).text)
```

Here, `summary.future.text` expresses a dependency before the summary exists. `sb.concat` constructs a prompt containing that future value, and `substrate.run` submits the connected work. This example shows the graph-authoring pattern: the application describes how outputs will be consumed rather than retrieving each result and manually dispatching the next call.

The execution engine is the other half of the design. Because it receives tasks and their relationships, a coordinated compute cluster can optimize **batching, caching, networking, concurrency, and physical placement**, both statically and dynamically. Dependencies identify work that must wait; independent branches expose opportunities for concurrency. The graph therefore serves both as a readable application structure and as information the execution system can use to schedule work.

### The cost between model calls

[3:14](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=194s)

A workflow assembled from separately dispatched API calls repeatedly crosses service boundaries. Cheung identifies DNS resolution, proxies, authentication, and balance checks as work incurred along that path. Coordinated execution aims to keep intermediate transfers inside the compute system instead.

| Execution pattern | Inter-step path |
| --- | --- |
| Separately dispatched APIs | Repeated service entry and request checks |
| Substrate graph | Internal node-to-node, process-to-process transfer |

Cheung reports hundreds of milliseconds of overhead per separately dispatched API step, versus microsecond-scale internal transfers in Substrate—a roughly 10,000× difference in inter-step transfer overhead. This is a comparison of handoff costs, not total model inference time. The recording supplies no payload sizes, hardware configuration, concurrency level, or latency statistic; the [launch-era investor account](https://lsvp.com/stories/substrate-building-compound-ai-systems/) describes output handoffs in a few milliseconds without enough detail to reconcile the measurement scopes.

The practical ambition is to make online applications with dozens of nodes feasible. If each handoff is expensive, adding useful intermediate steps can make an application too slow. Reducing that overhead gives developers more room to decompose the task without spending the interaction budget on communication between steps.

### Structured outputs make the connections useful

[3:58](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=238s)

Fast transfers address the cost of connecting nodes; **JSON decoding** addresses a useful format for those connections. Cheung identifies JSON as a particularly valuable pattern in workflows involving multiple inference runs and says Substrate invested heavily in its JSON mode. He positions its reliability and speed as best in class, though the recording provides no supporting measurements.

The concluding product claim ties these mechanisms to higher-quality outcomes: a flexible graph lets developers compose capabilities, a legible structure exposes how the system works, and intermediate outputs make individual steps verifiable. The intended benefit is the ability to build and improve a system of inference runs whose behavior can be examined along the way.

The launch closes with an invitation to obtain credits through a QR code, visit `substrate.run`, or contact Cheung by email. Treat that as the historical access invitation: the [Python package](https://pypi.org/project/substrate/220240617.0.0/) is now marked archived, and the [former documentation destination](https://docs.substrate.run/) redirects to Zo. The SDK example above illustrates the documented graph pattern, rather than establishing that hosted access or launch credits remain available.

## Resources

- [Substrate Python SDK](https://github.com/zocomputer/substrate-python) — Source and examples for connecting inference nodes through future outputs and executing computation graphs. Hosted-service availability is unverified.
- [Substrate: Building Compound AI Systems](https://lsvp.com/stories/substrate-building-compound-ai-systems/) — Launch-era investor account of Substrate’s graph SDK, atomic compute nodes and coordinated execution.
- [Substrate SDK: June 2024 release](https://pypi.org/project/substrate/220240617.0.0/) — Historical Python package release with usage examples and release metadata. PyPI now marks the project archived.

## Slides

### Multiple inference runs

The speaker argues that successful AI products typically use multiple inference runs, often combining different types of models to perform a specific task well.

[0:35](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=35s) [More than one inference run to accomplish a specific kind of task really well.](https://assets.ai.engineer/sites/aiecode2025/sha256/831f1a940b5fa8067b7e28f3fb4d7b4268e6c79f7b82f80eccb9e0f8370a501e.webp)

More than one inference run to accomplish a specific kind of task really well

Substrate

### The foundation model is not enough

The speaker argues that the best AI products right now use logically structured systems of inference runs, from document summarization to end-to-end coding.

[0:55](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=55s) [Slide stating that the foundation model is not enough, with the speaker shown in an inset.](https://assets.ai.engineer/sites/aiecode2025/sha256/2b871317bd8f63df1735d9852fe0e8bc6e95fe169f6fa7703be56d38216e276f.webp)

The foundation model is not enough.

Substrate

Interface text:
AIE
AI Engineer World's Fair
DATASTAX
Galileo
aws

[1:07](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=67s) [Slide reads: The foundation model is not enough.](https://assets.ai.engineer/sites/aiecode2025/sha256/1af9c5a874ce4c25cdd46bf8114d36822e7d107efb06df34e7743c8ed1c72f75.webp)

The foundation model is not enough.

Substrate

Interface text:
AIE
AI Engineer World's Fair
DATASTAX
Galileo
Microsoft
smol ai

### Modular intelligence

Substrate argues that modular systems are easier to understand, debug and extend, while explicit decision trees make it easier to evaluate each step.

[1:19](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=79s) [Slide says building with modular intelligence is more effective than building with monolithic intelligence.](https://assets.ai.engineer/sites/aiecode2025/sha256/422ccafb2dedd78db7ff9ef504dc17e029dc7a1306d527ba929ca8eae33e26d6.webp)

Building with modular intelligence is more effective than building with monolithic intelligence

Substrate

Interface text:
AIE
AI Engineer World's Fair
Google Cloud
DATASTAX
Galileo
AI Engineer World's Fair

### Voice note processing workflow

Substrate expresses voice-note processing as a graph: transcribe speech, extract JSON, then embed each note in parallel. The diagram places the work on one machine with zero roundtrips.

[2:11](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=131s) [Code and a diagram show speech transcription, JSON extraction, and parallel embeddings on one machine.](https://assets.ai.engineer/sites/aiecode2025/sha256/f96430a8d4a16007a247b0e7a3f418b45d0c75096bafbcd882ea0fb2b5a0a110.webp)

async function processVoiceNote() {
  // transcribe
  const transcript = new TranscribeSpeech({ audio_uri: "..." });
  // extract
  const notes = new ComputeJSON({
    prompt: interpolate`Extract notes from: ${transcript.future.text}`,
    json_schema: { ... },
  });
  // embed
  const embeddings = notes.future.json_object.notes.map(
    (n) => new EmbedText({ text: n.content })
  )

  const output = new Box({
    transcript: transcript.future.text,
    notes = embeddings.map(e => e.text),
    vectors: embeddings.map(e => e.vector),
  });
  return substrate.run(output);
}

1 machine
Transcribe Speech
Extract JSON
Embed
Embed
Embed
Zero roundtrips

Substrate

## Full transcript

[0:00](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=0s) [on-hold electronic music] Yeah, it's really good to be here. Um, this is a particularly exciting talk for us because we've been working with private clients for about a year now, but this is the first time we've really talked about it in public, um, since our

[0:25](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=25s) launch last week. Um, I'm incredibly proud of the work we've done so far and, um, excited to take a few minutes to tell you about it. Um,

[0:36](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=36s) so if you look at the products out there that have really successfully leveraged this generation of AI, I think one thing is true about nearly all of them is that they're using more than one inference run, often many different types of models in tandem to accomplish a specific kind of task really well.

[0:54](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=54s) And I think people really quickly realize that the foundation model is not enough, and even very simple tasks like summarizing a document to much, much more complex tasks like solving coding problems end to end, I think the best products right now are all using systems of inference runs in a logical structure.

[1:17](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=77s) So I think at Substrate we believe that building with modular intelligence is always going to be more effective than building with a monolithic intelligence. Um, these systems are inherently more legible, which means you can understand them structurally, which means that they're debuggable and they're extensible.

[1:38](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=98s) And evals become a lot easier because the decision trees are explicit, and you can sort of verify at every step what's going on and what's going wrong. Um,

[1:50](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=110s) so Substrate, I think, is a, is a sort of new way, a new approach to this. Um, I think our model is sort of fast in ways that other paradigms can't be.

[2:01](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=121s) It's sort of flexible enough to build any s- AI product out there, and it works at scale by default. So what is it? Um, I think at its core, Substrate is a coupling of two things.

[2:14](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=134s) First, I think it's a really elegant developer SDK that lets you describe a computation graph over any number of nodes. Um, and the abstractions here are, are really general, and so we have, we have a bunch of intelligence nodes across all the modalities that you might care about, which is like generating images, transcribing speech, generating text, JSON,

[2:39](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=159s) embeddings, executing code. Um, but second, Substrate is also an inference engine specifically built to run these computation graphs as efficiently as possible. Um, so these graph representations here, um, are ...

[2:57](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=177s) It's a representation of many tasks and their relationships, and since we run a very coordinated compute cluster, um, we can statically and dynamically optimize things like batching, caching, sort of networking, concurrency, physical placement, um, which really makes a big difference.

[3:14](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=194s) Uh, and if you look at most frameworks out there, um, they're typically involving dispatching a bunch of API calls separately, and if you look at what happens mechanically when you do that, it's every step means you've gotta resolve DNS, you've gotta go through proxies, you've gotta go through authentication, like balance checks.

[3:34](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=214s) Um, and all of that sort of adds hundreds of milliseconds of latency on every single step. And if you contrast that with Substrate, we, we transfer data from node to node, process to process, on the order of microseconds versus some 10,000 times faster, meaning that it's actually feasible now to run online applications that involve dozens of nodes.

[3:58](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=238s) Um, we've also noticed that JSON decoding is, is one of the most useful patterns for multi-inference runs, and I think we've invested a lot into offering a, a best in class, um, JSON mode, both in terms of reliability and speed.

[4:15](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=255s) And if you look at all of this together, I think what it means is that Substrate is, is, is really a way that, way to enable higher quality outcomes with AI, letting you work in a system that's more flexible, it's more legible, it's more verifiable than any of the current paradigms that sort of exist now.

[4:36](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=276s) Um, I think there's a lot more to say. That's all the time I really have today. It's only five minutes. Um, but if you're curious, um, please come out and say hi on the expo floor.

[4:48](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=288s) Um, you can scan this QR code we ... and get some credits, um, and go to the website substrate.run, um, or give me an email at, uh, [REDACTED:email_address]. [on-hold electronic music]
