Substrate Launch: the API for modular AI

What does it take to turn a model into a product that reliably summarizes documents or solves coding problems end to end? Substrate entered this problem space after about a year working with private clients. At his June 2024 AI Engineer World’s Fair session, Rob Cheung described the launch the previous week as the beginning of the team’s public introduction.

The starting observation is that successful AI products often use multiple inference runs, sometimes combining different types of models to perform a specific task. Even document summarization can benefit from a system of calls; end-to-end coding makes the need for a logical structure more apparent. The foundation model supplies a capability, while the surrounding system organizes how that capability gets used.

An explicit structure makes intermediate behavior inspectable. Substrate’s premise is that modular intelligence will be more effective than monolithic intelligence because developers can understand the system’s parts and relationships. That makes it easier to debug a failing step, extend the workflow, and evaluate intermediate decisions. When the decision tree is explicit, evaluation can ask where an error entered the process instead of judging only the final answer.

Substrate’s proposed combination is speed, flexibility across AI products, and scale by default. Its architecture couples two components: a developer SDK for describing a computation graph and an inference engine built to execute that graph. Nodes cover image generation, speech transcription, text generation, JSON, embeddings, and code execution. The graph captures both the tasks and their dependencies.

A document-summary workflow illustrates the dependency model. One node generates a summary; another turns that summary into a title. In the Substrate Python SDK, a future output can become part of a downstream prompt:

import os
from substrate import Substrate, ComputeText, sb

substrate = Substrate(api_key=os.environ["SUBSTRATE_API_KEY"])

document = (
    "The team moved the release to Friday. "
    "The extra time will be used to test the payment flow."
)
summary = ComputeText(
    prompt=f"Summarize this document in one sentence:\n{document}"
)
title = ComputeText(
    prompt=sb.concat(
        "Write a short title for this summary:\n",
        summary.future.text,
    )
)

response = substrate.run(summary, title)
print(response.get(summary).text)
print(response.get(title).text)

Here, summary.future.text expresses a dependency before the summary exists. sb.concat constructs a prompt containing that future value, and substrate.run submits the connected work. This example shows the graph-authoring pattern: the application describes how outputs will be consumed rather than retrieving each result and manually dispatching the next call.

The execution engine is the other half of the design. Because it receives tasks and their relationships, a coordinated compute cluster can optimize batching, caching, networking, concurrency, and physical placement, both statically and dynamically. Dependencies identify work that must wait; independent branches expose opportunities for concurrency. The graph therefore serves both as a readable application structure and as information the execution system can use to schedule work.

A workflow assembled from separately dispatched API calls repeatedly crosses service boundaries. Cheung identifies DNS resolution, proxies, authentication, and balance checks as work incurred along that path. Coordinated execution aims to keep intermediate transfers inside the compute system instead.

Cheung reports hundreds of milliseconds of overhead per separately dispatched API step, versus microsecond-scale internal transfers in Substrate—a roughly 10,000× difference in inter-step transfer overhead. This is a comparison of handoff costs, not total model inference time. The recording supplies no payload sizes, hardware configuration, concurrency level, or latency statistic; the launch-era investor account describes output handoffs in a few milliseconds without enough detail to reconcile the measurement scopes.

The practical ambition is to make online applications with dozens of nodes feasible. If each handoff is expensive, adding useful intermediate steps can make an application too slow. Reducing that overhead gives developers more room to decompose the task without spending the interaction budget on communication between steps.

Fast transfers address the cost of connecting nodes; JSON decoding addresses a useful format for those connections. Cheung identifies JSON as a particularly valuable pattern in workflows involving multiple inference runs and says Substrate invested heavily in its JSON mode. He positions its reliability and speed as best in class, though the recording provides no supporting measurements.

The concluding product claim ties these mechanisms to higher-quality outcomes: a flexible graph lets developers compose capabilities, a legible structure exposes how the system works, and intermediate outputs make individual steps verifiable. The intended benefit is the ability to build and improve a system of inference runs whose behavior can be examined along the way.

The launch closes with an invitation to obtain credits through a QR code, visit substrate.run, or contact Cheung by email. Treat that as the historical access invitation: the Python package is now marked archived, and the former documentation destination redirects to Zo. The SDK example above illustrates the documented graph pattern, rather than establishing that hosted access or launch credits remain available.