# How agents broke app-level infrastructure

Evan Boyle

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/_-oIuRH4oGA-agents-broke-app-level-infrastructure) · [Video](https://www.youtube.com/watch?v=_-oIuRH4oGA)

## How agents broke app-level infrastructure

As prompts become long-running workflows, compute must outlive browser connections, preserve progress, and recover from failures without making users start over.

Familiarity with TypeScript, HTTP APIs, and background jobs will help; the article explains the Redis Streams mechanism as it appears.

### From prompts to data pipelines

[0:00](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=0s)

What happens to the compute layer when a simple prompt grows into a data pipeline? Evan Boyle approaches that question as GenSX’s founder and CEO and an early Pulumi employee. His diagrams were generated with **GPT-4o**, accompanied by a warning about its spelling. The infrastructure problem starts with something equally familiar: a prompt and a few tool calls—enough, he jokes, to raise a ten-million-dollar seed round.

Turning that prototype into a reliable product means controlling more of its behavior. Instead of leaving everything to one nondeterministic interaction, developers chain tailored prompts, write evals against them, and manage the context each receives. Boyle illustrates the resulting runtime growth as thirty seconds becoming a couple of minutes. Eventually, context becomes the larger engineering problem: crawling a user’s inbox or ingesting GitHub code creates substantial additional LLM processing. Improving the agent has also turned the application into a data pipeline.

![Three illustrated stages labeled Simple prompt, Build a workflow, and Process large amounts of data, connected by arrows.](https://ai.engineer/api/talk-essay-media/d8c2e1665cefe6779a7997dea69143ca767923f684f19dd4a9af00561ab06ecd)

From a simple prompt to workflows and large-scale data processing.

### Longer requests, unreliable dependencies

[1:22](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=82s)

The old request model assumed a short trip through an API and database. Boyle contrasts a conventional web response taking a few tens of milliseconds with an AI response taking a couple of seconds at best, conditional on a fast model or prompt-cache hit. He calls the latter “P1,” without defining the statistic. A delay that once might have paged the on-call engineer is now part of the expected product experience. Infrastructure designed around short requests inherits the wrong assumptions about how long work stays active.

Longer execution also leaves more opportunities to encounter dependency failures. Boyle walks through a Slack channel subscribed to provider incidents: one outage lasts roughly an hour and resolves by 10 PM, then trouble returns at 1:20 AM and remains at 4 AM. He challenges a displayed 99.9% uptime figure by pointing to yellow and red status-history bars, and reports that a Gemini outage overlapped the illustrated OpenAI outage. The example makes failover less reassuring: another provider is useful only if it is available when needed. The displayed history does not establish a comparable availability measurement across providers.

![OpenAI status page with a fully operational banner above API and ChatGPT history bars containing green, yellow, and red segments.](https://ai.engineer/api/talk-essay-media/1f318867b8f19df41cce20d82d17e2c692dff446d8c8d350c55204b2d35a833b)

OpenAI’s status page shows interruptions across its API and ChatGPT history.

Even when providers are healthy, onboarding customers or processing document batches produces bursty demand. Boyle describes high usage tiers as a spending barrier to experimentation, saying Tier 5 required thousands of dollars. That historical figure should not be used as a current requirement: the [OpenAI rate-limit documentation](https://developers.openai.com/api/docs/guides/rate-limits), checked in August 2026, lists $1,000 paid for Tier 5 qualification. Qualification spending, monthly usage ceilings, and model-specific request or token throughput are separate constraints. The engineering problem remains absorbing bursts without treating every rate-limit response as a failed user operation.

### The gap between a chatbot template and a workflow engine

[3:14](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=194s)

A Next.js chatbot template makes the first version easy. Once its work lasts minutes or hours, the full-stack engineer inherits problems traditionally handled by data infrastructure. Existing tools offer several starting points:

| Tool | Starting point |
|---|---|
| SQS | Queues around which to assemble orchestration |
| Airflow | Batch-processing workflows |
| Temporal | Durable execution |

Boyle’s objection is partly about developer experience. As a TypeScript and full-stack developer, he would rather retain the simplicity of serverless than assemble a queue-based orchestration system. His joke about avoiding tools designed by Java engineers expresses that preference, rather than a technical evaluation of those tools.

Serverless introduces its own constraints. Boyle describes five-minute timeouts and, on some platforms, limits on outgoing HTTP requests. He also describes streaming as work bolted onto the application. These are not universal platform properties: [Lambda supports execution timeouts up to fifteen minutes](https://docs.aws.amazon.com/lambda/latest/dg/configuration-timeout.html), and [native Lambda response streaming](https://aws.amazon.com/blogs/compute/introducing-aws-lambda-response-streaming/) arrived in 2023 through function URLs and its invocation API, with integration restrictions. **Streaming a response and preserving a reconnectable history are different capabilities.** A user refreshing a page during a long operation needs the latter, even if the platform already supports the former.

### Let users start before ingestion finishes

[4:35](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=275s)

Website onboarding makes the lifecycle problem concrete. The user supplies a URL, and an initial LLM call extracts information from that page and identifies other pages to scrape. That first interaction starts a much larger background operation. In Boyle’s example, scraping and enrichment take multiple minutes and involve hundreds of LLM calls.

The product should become usable while that work continues. Boyle presents a ten-minute onboarding wait as a risk to funnel completion, not as a measured conversion result. Moving ingestion into the background removes the wait, but creates a communication requirement: show its status inside the product. Users need to understand that more data is arriving and that the experience will improve as enrichment progresses. Background execution and visible progress are therefore parts of the same onboarding design.

### A writing agent makes progress part of the product

[5:42](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=342s)

The second product example generates a blog post about building high-performance web applications with Vercel and Next.js. It tells the user up front that the operation will take a couple of minutes. A progress panel then exposes the work: research, outline generation, and writing individual sections. Those intermediate steps give the user something meaningful to follow, but they must also survive navigation away from the page.

![Generating your article message beside a stack of Research Agent and Content Writer Agent status cards describing research and individual article sections.](https://ai.engineer/api/talk-essay-media/fc25a9baf19c78f53ef2c5c1dc4e895237f0c7a47cfa01de1596c419ff8a4cd2)

Article generation displays research and section-writing progress while the user waits.

Boyle reports that this workflow in a previous product took three to four minutes. Experimenting with workflows beyond five minutes would have required substantial changes to its serverless infrastructure, so the team delayed those experiments. They eventually migrated, but the work involved had already constrained product development. A runtime ceiling had become a limit on what the team was willing to try.

The application needs to stream both **intermediate status** and **final content**, then restore both after a refresh, as users expect from deep-research experiences. Recovery matters more as the work gets longer: Boyle describes an intermediate error potentially making a user repeat five minutes of work. That is a different cost from refreshing a quick webpage or resubmitting a form. Preserving progress is part of making the operation tolerable, not merely an observability feature for developers.

### Infrastructure-aware components

[7:18](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=438s)

The infrastructure built for that earlier product became the open-source [GenSX library](https://github.com/gensx-inc/gensx). Its central design couples the component model to the execution environment: the framework knows about the infrastructure, and the infrastructure knows about the framework. That relationship supports resumable streams for intermediate status and final output.

Boyle describes an unopinionated anti-framework inspired by React’s component model, applied to backend execution. The aim is to share and compose code without hiding the decisions that matter when building agents. **Components** are reusable, independently testable steps, described as idempotent; **workflows** collect those steps into a larger operation. Treating a step as idempotent still requires its implementation to tolerate repetition—especially if it changes an external system.

![Slide listing an anti-framework approach, React-inspired composition, reusable components, workflows, plain functions, resumable streams, tracing, retries, caching, and durability.](https://ai.engineer/api/talk-essay-media/b58c114d8b2981dc3ee6c8c2d53a4c2104a4c755c152444de2763bfbba2ccda5)

An infrastructure-aware component model combines reusable steps, workflows, and resumable streams.

The demonstrated component uses a wrapped OpenAI SDK. The wrapper retains the familiar SDK surface and access to model parameters while adding tracing and retry tooling. The component itself remains a small function: accept a prompt and context, call the model, and return a response. It can be invoked or tested independently before becoming part of a larger workflow.

### Compose the work, expose its boundaries

[8:43](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=523s)

The demonstrated workflow fetches the Hacker News front page, analyzes its posts, generates reports, writes a tweet, and then edits the report. Ordinary TypeScript can express that sequence while leaving each operation independently replaceable and testable:

```typescript
type Post = { title: string; url: string };
type Analysis = { summary: string; sources: Post[] };

type Steps = {
  fetchFrontPage(): Promise<Post[]>;
  analyzePosts(posts: Post[]): Promise<Analysis>;
  generateReports(analysis: Analysis): Promise<string>;
  writeTweet(report: string): Promise<string>;
  editReport(report: string): Promise<string>;
};

async function runHackerNewsWorkflow(steps: Steps) {
  const posts = await steps.fetchFrontPage();
  const analysis = await steps.analyzePosts(posts);
  const report = await steps.generateReports(analysis);
  const tweet = await steps.writeTweet(report);
  const editedReport = await steps.editReport(report);

  return { report: editedReport, tweet };
}
```

This expresses the composition without depending on a particular version of the GenSX API. The infrastructure-aware component model adds execution behavior around the individual calls.

Each component supplies a retry boundary and an error boundary. Tracing covers both the enclosing workflow and the components nested inside it. The framework also turns workflows into REST APIs for synchronous invocation, asynchronous invocation, and retrieval of intermediate and final output streams. The same composed operation can therefore serve an immediate request or run as a background job.

The trace view exposes nested components alongside token information, OpenAI call details, and user and system messages. This makes it possible to investigate a particular model interaction within the larger workflow. For a troublesome component, the demonstrated fluent configuration API supports retries and caching, including an exponential retry policy with a configurable retry count. The policy belongs to the problematic step rather than being applied indiscriminately to the entire workflow.

### Separate execution from the API connection

[10:01](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=601s)

The deployment platform separates the **API layer** from the **compute layer**. Rather than keeping a request handler attached to a worker for the lifetime of a job, it uses a Redis Stream as the communication path:

1. The API invokes compute and passes it a Redis Stream ID.
2. The executing sandbox writes status and output to that stream.
3. The API obtains subsequent status and output through the stream rather than continued direct communication with compute.
4. The sandbox emits heartbeats that background processes monitor. If execution dies or crashes, the system can restart it or notify the user.

The browser connection no longer has to be the channel that holds the operation together.

This separation also creates independent scaling boundaries. The API can scale with client connections while compute scales with workflow execution. Boyle describes ECS as the default compute backend, with the option to bring another compute layer. Workers are stateless with respect to stored output: they communicate with the surrounding platform and write results to Redis Streams, rather than requiring the API to retrieve output from a particular worker’s memory.

### Reconnect to stored progress

[11:23](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=683s)

Both synchronous and asynchronous workflows write their output to Redis Streams. Because the API reads that stored output instead of reading directly from compute, a replacement browser connection can retrieve the history of status messages and generated content. Refreshing the page, navigating away, or terminating the browser’s connection to the API does not itself cancel the work.

For a concrete illustration, suppose the writing agent has stored `Research complete` at entry `1000-0` and `Outline complete` at `1001-0`. The browser has displayed both and saved `1001-0` as its cursor. While it is disconnected, the worker adds `Section 1 draft` at `1002-0`. On reconnect, reading after `1001-0` retrieves the missing draft; the earlier entries remain stored. [Redis `XREAD`](https://redis.io/docs/latest/commands/xread/) supports this last-seen-ID pattern. A refreshed client that has lost its local content instead needs to rebuild from retained history. **Replaying output is distinct from resuming failed computation**: retention and Redis persistence determine what remains available, while execution recovery and safe repetition require their own mechanisms.

Retrieve the draft produced while the browser was disconnected
Reconnection retrieves stored output without asking the worker to generate it again.
Constructed example: The entry IDs, message text, saved browser cursor, and retained browser state are teaching values extending the writing-agent example. The comparison illustrates output replay, not recovery of failed computation.
Reconnect request — unchanged: Writing-agent stream; last-seen entry ID: 1001-0
Operation: Read stored entries after 1001-0, display the missing draft, and advance the browser cursor.
- Redis entry 1000-0
  - Before: Disconnected browser; draft already stored: Research complete
  - After: Browser has replayed the missing entry: Research complete
- Redis entry 1001-0
  - Before: Disconnected browser; draft already stored: Outline complete
  - After: Browser has replayed the missing entry: Outline complete
- Redis entry 1002-0
  - Before: Disconnected browser; draft already stored: Section 1 draft
  - After: Browser has replayed the missing entry: Section 1 draft
- Browser's retained research status
  - Before: Disconnected browser; draft already stored: Research complete
  - After: Browser has replayed the missing entry: Research complete
- Browser's retained outline status
  - Before: Disconnected browser; draft already stored: Outline complete
  - After: Browser has replayed the missing entry: Outline complete
- Browser's displayed draft
  - Before: Disconnected browser; draft already stored: Not present
  - After: Browser has replayed the missing entry: Section 1 draft
Reconnection retrieves stored output without asking the worker to generate it again.

### Start small, but leave room for long-running work

[12:07](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=727s)

There is no need to build for an hour-long workflow on day one. Start with the operation the product actually needs, while leaving room for longer execution. Coding agents and deep-research experiences motivate Boyle’s expectation that more work will become delegated: give an agent instructions, let it operate independently, and have it communicate when needed. Separating compute from the API and retaining progress in a stream allows that work to continue without requiring the user to keep a page open.

That independence also changes deployment. Safe navigation and transparent error handling are incomplete if a routine release kills active work. Boyle uses sixty-minute workflows to emphasize the need for careful worker draining and a blue-green deployment pattern. A deployment must account for workers that are still busy, not just whether replacement instances are healthy. Getting the first workflow running is easy compared with getting its lifecycle right.

Boyle closes by offering GenSX as an implementation of much of this infrastructure for developers who do not want to build it themselves. The linked repository was archived on October 2, 2025; it remains a reference for the design, rather than evidence that the hosted platform is currently available.

## Resources

- [GenSX on GitHub](https://github.com/gensx-inc/gensx) — TypeScript workflow framework with composable components, tracing, and agent examples. The repository was archived in October 2025.
- [GenSX Cloud launch](https://www.gensx.com/blog/gensx-cloud) — Evan Boyle's subsequent launch article explains the 60-minute runtime, background invocation, storage, tracing, and durable progress streams.
- [GenSX Quickstart](https://www.gensx.com/docs/quickstart) — Instructions for creating a TypeScript workflow and integrating model SDKs.
- [Reading Redis Streams with XREAD](https://redis.io/docs/latest/commands/xread/) — Explains how to read stream entries after a saved ID and wait for new entries.
- [AWS Lambda response streaming](https://aws.amazon.com/blogs/compute/introducing-aws-lambda-response-streaming/) — The 2023 announcement and examples for streaming Lambda responses through function URLs and the invocation API.
- [OpenAI API rate limits](https://developers.openai.com/api/docs/guides/rate-limits) — Current usage-tier qualifications, request and token limits, response headers, and retry guidance.

## Slides

### How Agents Broke Infra

Evan Boyle introduces a talk on how agents broke infrastructure, focusing on the compute layer. The title image depicts a robot smashing computers with a hammer.

[0:00](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=0s) [An orange robot smashes computer infrastructure with a hammer.](https://assets.ai.engineer/sites/aiecode2025/sha256/1163eb9ac9b8232f2af7ed1b5dac59869dd9798ef44dd5736b33d44e6e61b15c.webp)

HOW AGENTS
BROKE INFRA

A

### Evan Boyle

GenSX founder and CEO Evan Boyle led and scaled Pulumi’s engineering team through 50+ and previously worked at Azure and Amazon.

[0:09](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=9s) [Evan Boyle’s background, with a GenSX logo and portrait.](https://assets.ai.engineer/sites/aiecode2025/sha256/a1d8ce830eaaac74981e93d9629357b584ffe91cc2138aa865c7ac5f108bc7f0.webp)

Evan Boyle

- Founder & CEO @ https://github.com/gensx-inc/gensx
- Early employee @ Pulumi - led and scaled the engineering team through 50+
- Early career at Azure & Amazon (search, cloud, etc.)

GenSX

### Typo Warning

Boyle warns that the talk’s AI-generated diagrams may contain spelling errors because 4o still struggles with spelling.

[0:18](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=18s) [A warning that AI-generated diagrams may contain spelling errors.](https://assets.ai.engineer/sites/aiecode2025/sha256/c9722bc3a42a18c18e415a40bd973b35c6eeb6b81ecca63b876602e5d7732cf7.webp)

Typo warning!
- In the spirit embracing AI, all diagrams are AI generated
- 4o still has issues spelling
- Thank you for your patience =D

### Changing Assumptions Around Infra

The slide contrasts Web 2.0’s 500 ms P99 latency with AI applications’ P1 of a couple of seconds at best. The speaker says even that assumes a fast model or a prompt cache hit.

[1:20](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=80s) [Web 2.0’s 500 ms P99 latency is contrasted with AI applications taking a couple of seconds at best.](https://assets.ai.engineer/sites/aiecode2025/sha256/af3293281088aaa59895659914abc34e2fff21865b29673a73ce87a7eacf5f16.webp)

CHANGING ASSUMPTIONS
AROUND INFRA

WEB 2.0
P99 LATENCY
500 MS

AI APPLICATIONS
p1
A COUPLE OF
SECONDS AT BEST

### Service Availability Issues

A worried builder stacks an LLM application on cracked bricks, illustrating how dependency outages make reliable applications hard to build.

[2:02](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=122s) [A worried builder places an LLM application on a cracked foundation labeled service availability issues.](https://assets.ai.engineer/sites/aiecode2025/sha256/0dae097ebb9ed3d89af82d0448824b3f0a4de3b0fe4c0fab7621db0fa17677e2.webp)

LLM APPLICATION

SERVICE AVAILABILITY ISSUES

### Service Incident Updates

Anthropic’s Slack alerts track elevated Claude 3.7 Sonnet errors from investigation at 9:11 PM to a fix under monitoring at 9:44 PM and resolution at 10:00 PM.

[2:10](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=130s) [Incident notifications show elevated model errors being investigated, monitored, and resolved.](https://assets.ai.engineer/sites/aiecode2025/sha256/6671d1818b1e7c018006b903d81b58eb67e3758f247aec858392f0625008630c.webp)

Anthropic Status APP 9:11 PM
Incident: Investigating
Elevated errors for requests to Claude 3.7 Sonnet
We are currently investigating elevated errors on requests to Claude 3.7 Sonnet on the API, Claude.ai, and the Anthropic Console.
claude.ai
console.anthropic.com
api.anthropic.com

Anthropic Status APP 9:44 PM
Incident: Monitoring
Elevated errors for requests to Claude 3.7 Sonnet
A fix has been implemented and we are monitoring the results.
claude.ai
console.anthropic.com
api.anthropic.com

Anthropic Status APP 10:00 PM
Incident: Resolved
Elevated errors for requests to Claude 3.7 Sonnet
This incident has been resolved.
claude.ai
console.anthropic.com
api.anthropic.com

Interface text:
# incidents
Messages
Add canvas
Files
Sunday, May 11th
View all updates
Powered by Atlassian Statuspage | Manage notifications
View all updates
Powered by Atlassian Statuspage | Manage notifications
View all updates
Powered by Atlassian Statuspage | Manage notifications

### Anthropic incident updates

On May 12, Claude 3.7 Sonnet errors returned at 1:20 AM. Updates continued until 4:06 AM, when Anthropic identified the issue and began implementing a fix.

[2:22](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=142s) [Slack channel showing repeated investigations and an identified issue affecting Claude 3.7 Sonnet.](https://assets.ai.engineer/sites/aiecode2025/sha256/cabd84ac4941c079051d5dbfbd1c4a68a643098a65573093eb5d00d8d361df9e.webp)

Anthropic Status APP 1:20 AM
Incident: Investigating
Elevated errors for requests to Claude 3.7 Sonnet
We are currently investigating this issue.
claude.ai
console.anthropic.com
api.anthropic.com
View all updates
Powered by Atlassian Statuspage | Manage notifications

Anthropic Status APP 2:13 AM
Incident: Investigating
Elevated errors for requests to Claude 3.7 Sonnet
We are continuing to investigate this issue.
claude.ai
console.anthropic.com
api.anthropic.com
View all updates
Powered by Atlassian Statuspage | Manage notifications

Anthropic Status APP 3:41 AM
Incident: Investigating
Elevated errors for requests to Claude 3.7 Sonnet
We are continuing to investigate this issue.
claude.ai
console.anthropic.com
api.anthropic.com
View all updates
Powered by Atlassian Statuspage | Manage notifications

Anthropic Status APP 4:06 AM
Incident: Identified
Elevated errors for requests to Claude 3.7 Sonnet
The issue has been identified and a fix is being implemented.
claude.ai

Interface text:
# incidents
Messages
Add canvas
Files
Monday, May 12th

### OpenAI system status

OpenAI’s February–May 2025 status history reports 99.91% API uptime and 99.55% ChatGPT uptime. The speaker questions the uptime figure, pointing to the yellow and red bars.

[2:32](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=152s) [OpenAI status page showing operational status and API and ChatGPT uptime histories.](https://assets.ai.engineer/sites/aiecode2025/sha256/52b612c077286a00fab8b763297138de09cbb5ee055220dd9f553a8e10449f86.webp)

OpenAI

We’re fully operational
We’re not aware of any issues affecting our systems.

System status
Feb 2025 - May 2025

APIs
15 components
99.91% uptime

ChatGPT
20 components
99.55% uptime

Interface text:
Subscribe to updates

### Gemini past incidents

Gemini 2.5 Pro was unavailable for some users on May 11 and mitigated by May 12. The speaker says the outage coincided with another outage, complicating failover.

[2:46](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=166s) [Past incidents listing Gemini API unavailability in May 2025.](https://assets.ai.engineer/sites/aiecode2025/sha256/c7bff0fc61c9eeedc65386a8b455d40d05d7232cad13e52686b612727d541df8.webp)

Past Incidents

May 12, 2025

Intermittent unavailability of Gemini API 2.5 Pro

Resolved - The issue is mitigated. We are keeping the incident open to monitor the situation.
May 12, 03:15

May 11, 2025

Intermittent unavailability of Gemini API 2.5 Pro

Detected - Gemini 2.5 Pro is unavailable for some users.
May 11, 23:15

May 9, 2025

Intermittent unavailability of Gemini API

Resolved - Mitigations were deployed, and error levels are back to nominal levels.
May 9, 07:30

Detected - Some requests across all models were failing with 500 or 503 errors.
May 9, 04:55

### How agents broke infrastructure

A happy prototype gives way to a worried engineer as agent workflows stretch from seconds to minutes and hours. The speaker says full-stack AI engineers accidentally become data engineers.

[3:12](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=192s) [Illustrated curve rising as agent workflow runtime grows from seconds to hours.](https://assets.ai.engineer/sites/aiecode2025/sha256/52ea40f1d473541a806b5a4ce52db4114ef460dbcf17f76889af2c58c759db34.webp)

HOW AGENTS BROKE INFRA

Full-stack LLM engineer becomes data engineer

Easy!

X-axis: Runtime of Agent Workflow Using LLMs
X-axis labels: seconds, minutes, hours

### Workflow tooling complexity

Long-running workflows can become Rube Goldberg machines built from queues like SQS, batch-processing tools like Airflow, and execution engines like Temporal.

[3:37](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=217s) [A worried engineer beside a tangled machine labeled SQS, Airflow, and Temporal.](https://assets.ai.engineer/sites/aiecode2025/sha256/1f22db3feddfd85f9eb7c332ea7f1660315d726c3f8a4e331f32b38e3948b04a.webp)

SQS
AIRFLOW
TEMPORAL

### Common product experiences

The talk turns to LLM-assisted onboarding and keeping users engaged while an agent runs for multiple minutes.

[4:33](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=273s) [Section slide titled Common product experiences.](https://assets.ai.engineer/sites/aiecode2025/sha256/eec434373dba96e61ede1ee944075e024b5c5b5267991516286189d98fdf2bbe.webp)

Common product experiences

### Configure an organization

After a user enters a URL, one LLM call gathers page information and detects pages to scrape. The form shows an organization description and two selected sitemaps, with an option to add another.

[4:56](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=296s) [Organization configuration form with a description and two detected sitemaps.](https://assets.ai.engineer/sites/aiecode2025/sha256/2473dd027c3eb23427381b5a36adb398f99e3047ce8d2cbcaabeeec437c8a0e4.webp)

Org Name
workos

Org Description
WorkOS provides a suite of tools for developers to quickly integrate enterprise features into their applications. Customers can easily add Single Sign-On (SSO), user management, and directory sync capabilities with minimal effort. The platform supports various authentication methods and identity providers, making it a versatile solution for enterprise needs. WorkOS aims to simplify the process of becoming enterprise-ready, allowing businesses to focus on their core products while meeting IT requirements.

Sitemaps Detected
✓ https://workos.com/sitemap_website.xml (348 pages)
✓ https://workos.com/docs/sitemap.xml (149 pages)

(Optional) Enter a sitemap URL to add
Add
Continue

Interface text:
WorkOS

### Starting ingestion

The demo starts ingesting 497 WorkOS pages. The speaker says background scraping and enrichment take minutes and hundreds of LLM calls, while users are intended to start using the app immediately.

[4:59](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=299s) [Onboarding progress screen showing organization creation and ingestion of 497 pages.](https://assets.ai.engineer/sites/aiecode2025/sha256/d76ed75aa80f236ea63e9087dac915643d8936e8aa8838a644b21c8f1ceb91df.webp)

Select Website
Configure Org
3 Create Cortex

Creating the org...

Kicking off ingestion of 497 pages from https://workos.com/...

Optimizing configurations...

Interface text:
WorkOS

### SSO integration answer

The demo shows a chat answer about WorkOS SSO as the speaker describes letting users start using the app while website ingestion runs in the background.

[5:12](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=312s) [Chat response describing initial steps for integrating single sign-on with WorkOS.](https://assets.ai.engineer/sites/aiecode2025/sha256/0f72e15b30491d8c5069f4e97f3759896836a05c9f349d6fafb90eaa2acb3d21.webp)

How can I integrate Single Sign-On (SSO) into my app using WorkOS?

Integrating Single Sign-On (SSO) into your app using WorkOS involves a few key steps. Here's a general guide to get you started:

Step 1: Set Up Your WorkOS Account
1. Sign Up: Create an account on the WorkOS dashboard.
2. Create a Project: Once logged in, create a new project. This will give you access to the necessary API keys and configuration settings.

Step 2: Configure SSO in the WorkOS Dashboard
1. Add an SSO Connection: In the WorkOS dashboard, navigate to the “SSO” section and add a new connection. You can choose from a variety of identity providers such as Okta, Azure AD, Google, etc.
2. Configure the Identity Provider: Follow the specific instructions for the identity

Interface text:
WorkOS
Dashboard  Content  Chats  Catalogs  Cortexes  Settings
Type your message...

### Problems

Site ingestion takes one to 10 minutes depending on site size. To avoid the increased funnel falloff associated with a 10-minute wait, the team wants users to start immediately and see ingestion status as data is enriched.

[5:16](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=316s) [Slide describing ingestion delays, immediate product access, and showing ingestion status.](https://assets.ai.engineer/sites/aiecode2025/sha256/8d3669073d97d75638e36e350a29e7fd5f48e3c6b7231ea9cf0d2cfaa3bc1239.webp)

Problems

1. Ingestion takes anywhere from one to 10 minutes depending on the size of the site.
2. We want users to be able to use the app immediately! Waiting 10 minutes means increased falloff in the funnel
3. Show users status of ingestion as they are using the product

### Article creation prompt

The demo prepares an article about performance in LLM-powered web apps, focusing on streaming across the client-server boundary with Next.js. The form also offers file attachments and an assistant selector.

[5:40](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=340s) [Article creation form requesting a post about performance and streaming in LLM-powered web apps.](https://assets.ai.engineer/sites/aiecode2025/sha256/603d9f232a1a75d424e3ed2670f33ca33b99fd8823ac182e72c958fde23f55d2.webp)

What can I help you create?

Building high performance web apps on top of LLMs

Write an article about performance concerns when building LLM powered apps. Focus on streaming across the client server boundary with Next.JS

Attach files
Vercel-assistant
Generate Article

For best results, be specific about what you want to create. Include key points, target audience, tone, and desired length. For example: “Write a technical blog post about implementing WebSockets in Next.js, aimed at intermediate developers, with code examples and best practices. Around 1500 words.”

Upload supporting documents (.docx, .md, .mdx, .json, .txt, and code files) to provide additional context, research, or reference materials that should be incorporated into the article.

### Article generation research

The article generator warns that the task could take a couple of minutes while status cards show its research and searches in progress.

[5:49](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=349s) [Article generation screen showing research and search status cards.](https://assets.ai.engineer/sites/aiecode2025/sha256/e7814efbeb47aeb75a4aa918c6408b5f4890cb554686c925e761b461ba34f6b2.webp)

Generating your article...

Sit back and relax, this could take a couple minutes

Research Agent
Downloading and reading 1 web pages.

Working...
Searching through the catalogs for "performance optimization" and "Next.js streaming"

Working...
Researching on the web for "performance optimization techniques for LLM powered web apps" and "streaming data handling in Next.js for high performance applications"

### Article generation writing

The demo shows research and individual section-writing updates while an article generates. The speaker stresses keeping that context when users navigate away.

[6:12](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=372s) [Article generation screen showing research and content writer status cards for several sections.](https://assets.ai.engineer/sites/aiecode2025/sha256/e8a3cb88ccbf220d39c750e093170bec849043959c9dcba31f0a03ce4b09d1da.webp)

Generating your article...
Sit back and relax, this could take a couple minutes

Research Agent
Downloading and reading 1 web pages.

Content Writer Agent
Writing a section about “The importance of performance in LLM-powered apps”.

Content Writer Agent
Writing a section about “Understanding streaming in Next.js”.

Content Writer Agent
Writing a section about “Implementing streaming for LLM responses”.

Content Writer Agent
Writing a section about “Performance considerations and trade-offs”.

Content Writer Agent
Writing a section about “Building the future of LLM-powered applications”.

### Content generation problems

Content generation took 3-4 minutes; workflows of 5+ minutes required deep infrastructure changes under the serverless provider. The team also needed streamed output and progress that survived navigation, plus recovery from errors.

[6:15](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=375s) [Slide lists workflow duration, infrastructure limits, streaming, persistent status, and error recovery challenges.](https://assets.ai.engineer/sites/aiecode2025/sha256/f4cb29cd86be441ba556dfc95805f93db19cceca049b7f443b4df9ffd7754056.webp)

Problems

1. Content generation takes 3-4 minutes.
2. Experimenting with longer workflows (5+ minutes) means deep infra changes due to limitation from our serverless provider.
3. We need to stream both the final content, as well as intermediate status.
4. Should be able to navigate away from the page, come back, and still see the intermediate status.
5. Users get frustrated if there is an intermediate error (unreliable LLM providers) and we force them to regenerate.

### Infrastructure-aware component model

React-inspired components compose into workflows using plain functions. Infrastructure awareness supports resumable streams for status and output, alongside tracing and retries.

[7:28](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=448s) [Slide describes reusable components, workflows, resumable streams, tracing, retries, caching, and durability.](https://assets.ai.engineer/sites/aiecode2025/sha256/72a164fb5fedd89658229b4f5c158e667d26c9fe6a14a62a8e8d8736f8f555f8.webp)

A simple component model that is infrastructure-aware

- Anti-framework: unopinionated and focused on building blocks
- Inspiration from React’s component model applied to the backend. Focus on composition over abstraction.
- Components: reusable, idempotent, independently testable steps
- Workflows: collections of components that run together
- Everything is just a plain function, use plain if statements, while loops, etc.
- Built in resumable streams for intermediate status and final output.
- Provides built-in tracing, retry policies and error handling, and other tools like caching and durability.

### Tweet-writing component

WriteTweet combines context with a prompt and returns a model response. The speaker says the wrapped OpenAI SDK adds tooling such as retries and tracing, while the component can be run and tested independently.

[8:14](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=494s) [TypeScript example defines a tweet-writing component that prompts a language model and returns its response.](https://assets.ai.engineer/sites/aiecode2025/sha256/245546c24bda393b36f02f40d75acb29541f438a79e091daefa99c55ee2d49d9.webp)

import * as gensx from "@gensx/core";
import { OpenAI } from "@gensx/openai";

interface WriteTweetProps {
  context: string;
  userPrompt: string;
}

const WriteTweet = gensx.Component(
  "WriteTweet",
  async ({ context, userPrompt }: WriteTweetProps): Promise<string> => {
    const PROMPT = `
    You are Paul Graham composing a tweet. Given a longer analysis, distill
    it into a single tweet that captures the most interesting insight.

    Focus on the most surprising or counterintuitive point rather than
    trying to summarize everything.`;

    const response = await openai.chat.completions.create({
      messages: [
        { role: "system", content: PROMPT },
        {
          role: "user",
          content: context + '\n' + prompt,
        },
      ],
      model: "gpt-4o",
      temperature: 0.7,
    });
    return response.choices[0].message.content ?? "";
  },
);

### Hacker News analysis workflow

The workflow fetches and analyzes Hacker News posts, generates and edits a report, then uses it to write a tweet. The speaker says each component gets retry and error boundaries and traces.

[8:43](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=523s) [Code defining a workflow that fetches and analyzes Hacker News posts, edits a report, and writes a tweet.](https://assets.ai.engineer/sites/aiecode2025/sha256/f854e2379a7604cbbdc80c9fc25b6fd96b865ddb775f6e3fc05015938ff7f715.webp)

interface AnalyzeHackerNewsTrendsProps {
  postCount: number;
}

export const AnalyzeHackerNewsTrends = gensx.Workflow(
  "AnalyzeHackerNewsTrends",
  async ({ postCount }: AnalyzeHackerNewsTrendsProps) => {
    const stories = await FetchHNPosts({ limit: postCount });
    const { analyses } = await AnalyzeHNPosts({ stories });
    const report = await GenerateReport({ analyses });
    const editedReport = await EditReport({ content: report });
    const tweet = await WriteTweet({
      context: editedReport,
      prompt: "Summarize the HN trends in a tweet",
    });

    return { report: editedReport, tweet };
  },
);

### Workflow execution trace

The workflow trace nests individual steps on a timeline. An expanded OpenAI call shows its input, output and token usage, giving the developer details to inspect when debugging.

[9:26](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=566s) [Execution timeline with nested workflow steps and an expanded model call showing input, output, and token usage.](https://assets.ai.engineer/sites/aiecode2025/sha256/9a93284934091e43512163da0fe1a8b4a4d66f85cb87e917f8e09a802bef8ee7.webp)

Timeline: 0s, 10s, 20s, 30s, 40s, 50s, 60s

306 AnalyzeHackerNewsTrends 57.31s
1 OpenAIProvider 0.00s
FetchHNPosts 3.05s
294 AnalyzeHNPosts 13.53s
2 SummarizePost 6.83s
1 ChatCompletion 6.83s

OpenAIChatCompletion
Input
messages:
  0:
  1:
    role: user
    content:
      Title: Show HN: CSV GB+ by Data.ollo – Open and Process CSVs Locally URL: https://news.ycombinator.com/item?id=43985527 Text: I built CSV GB+ by Data.ollo, a local data tool that lets you open, clean, and export gigabyte-sized CSVs (even billions of rows) without writing code.<p>Most spreadsheet apps choke on big files. Coding in pandas or Polars works—but not everyone wants to write scripts just to filter or merge CSVs. CSV GB+ gives you a fast, point-and-click interface built on dual backends (memory-optimized or disk-backed) so you can process huge datasets offline.<p>Key Features: Handles massive CSVs with ease — merge, split, dedup, filter, batch export<p>Smart engine switch: disk-based "V Core" or RAM-based "P Core"<p>All processing is offline — no data upload or telemetry<p>Supports CSV, XLSX, JSON, DBF, Parquet and more<p>Designed for data pros, students, and privacy-conscious users<p>Register for 7-days free to pro try, pro versions remove row limits and unlock full features. I'm a solo dev building Data.ollo as a serious alternative to heavy coding or bloated enterprise tools.<p>Download for Windows: <a href="https://apps.microsoft.com/detail/[illegible]" rel="nofollow">https://apps.microsoft.com/detail/[illegible]</a><p>User Guide: <a href="[illegible]" rel="nofollow">[illegible]</a><p>Would love feedback! I'm actively improving it based on real use cases. Score: 3 points
      Comments (sorted by score):
model: gpt-4o
temperature: 0.7
stream: false

Output
id: [illegible]
object: chat.completion
created: 1747235864
model: gpt-4o-2024-08-06
choices:
usage:
  prompt_tokens: 687
  completion_tokens: 315
  total_tokens: 1002
  prompt_tokens_details:
  completion_tokens_details:
service_tier: default
system_fingerprint: [illegible]

Interface text:
EXECUTION DETAILS
AnalyzeHackerNewsWorkflow
[illegible]
Remote
default/default
8:17:41 AM
57.00s
307 steps
logs
Completed
Show JSON
6.82s

### Serverless platform architecture

For executions up to 60 minutes, the API invokes an ECS sandbox with a Redis Stream ID, then reads status and output from the stream. Heartbeats help the platform detect failures and restart the workflow or notify users.

[9:59](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=599s) [Diagram showing API endpoints, an ECS sandboxed program, and a Redis stream for long-running executions.](https://assets.ai.engineer/sites/aiecode2025/sha256/b089236c6c465be4234a124eb6793dbbb277a0be7f19acf92951ebadd7a6d856.webp)

SERVERLESS PLATFORM SUPPORTS
60-MINUTE EXECUTIONS

API LAYER
Invoke
Status API
Output API
invoke
output

COMPU LAYER
ECS
Sandboxxed Program
status
output
Redis Stream

## Full transcript

[0:00](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=0s) [notification sound] How agents broke infrastructure. Particularly, we're going to focus on the compute layer today. [mouse clicking]

[0:11](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=11s) My name's Evan Boyle. I'm the founder and CEO in GenSX. I was an early employee at Pulumi, and I've spent my entire career working on cloud and developer tools. [keyboard clicking]

[0:20](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=20s) Typo warning. In the spirit of embracing AI for this talk, all diagrams are AI-generated, but 4o still has some spelling issues, so thank you for your patience. We all know this journey.

[0:31](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=31s) We start off as AI engineers with a simple prompt and a couple of tool calls, and typically, this is enough to raise our ten million dollar seed round. But from there, when rubber re-meets the road, we end up building workflows, right?

[0:43](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=43s) We take that fully non-deterministic code and make as much of it deterministic as possible. That might mean chaining a couple of very specific tailored prompts together that we've written evals against, uh, and more carefully controlling the context.

[0:57](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=57s) This takes the runtime of our workflows from thirty seconds to a couple of minutes, but eventually, we all end up becoming data engineers because the most challenging part of the problem is getting the right context into the prompts.

[1:12](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=72s) And that means that we might have to crawl a user's inbox, ingest code from GitHub, or many other artifacts that require lots and lots of LLM processing.

[1:22](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=82s) But our assumptions around infra have changed. In Web 2.0, you would build a simple web service where you'd make an API request, maybe talk to a database to fetch some data, and return all that back in a few tens of milliseconds.

[1:38](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=98s) In AI applications, our P1 is a couple of seconds at best, and that's only if you're using a fast model or if you hit a prompt cache. In the previous era of infrastructure, if you had a request that took a couple of seconds, your on-call was getting paged.

[1:54](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=114s) Um, and so the infrastructure that was designed for the last ten years of the web is not really designed for the applications that we're building today.

[2:04](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=124s) LLM applications are really built on shoddy, uh, foundations. It's really hard to build reliable apps today. And if you're like us and you run a tight ship, you might have a Slack channel that looks like this, where you subscribe to, you know, issues and outages from your dependencies.

[2:19](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=139s) But luckily, this one only lasts, you know, an hour. By ten PM, it's resolved. [keyboard clicking]

[2:24](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=144s) But oh, at one twenty AM, we're back at it, and four AM, it's still going on. But surely, you know, ChatGPT and OpenAI have, have better availability, right? Well, I wouldn't really trust this ninety-nine point nine percent uptime number, and no, it's not.

[2:40](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=160s) Lots of yellow and red in this graph. But maybe we could have failed over to, you know, Gemini during this time period. Nope, we have an outage during the exact same time period.

[2:51](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=171s) And even if you don't hit an outage, you still have rate limits to contend with. A lot of these traffic patterns are gonna be extremely bursty as you batch process documents from a user or you onboard new customers, and not everybody has tier five rate limits that require spending thousands of dollars just to get to that point

[3:09](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=189s) that you can start experimenting with high levels of traffic.

[3:14](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=194s) So we start off nice and happy with our prototype. Maybe we built something on top of the Next.js chatbot template. That's easy, right? But as our workflows start running for minutes and hours, us full stack AI engineers accidentally become data engineers. [keyboard clicking]

[3:30](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=210s) So how do people solve this today? You know, there-- this used to be something that was rather niche, ro-long-running workflows, data engineering. So there is an existing set of tools.

[3:40](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=220s) People build these Rube Goldberg machines on top of queues like SQS. They might use batch processing data engineering tools like Airflow or durable execution engines like Temporal. But me personally, I'm a TypeScript developer.

[3:52](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=232s) I'm a full stack engineer, and frankly, I don't wanna use anything that was designed by a Java engineer. No offense. [keyboard clicking]

[4:00](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=240s) I'd much rather use serverless. What about existing serverless providers? They aren't really well suited for these kinds of long-running workflows. Uh, most of them time out after five minutes, and some of them even limit outgoing HTTP requests.

[4:14](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=254s) And for all of them, they don't really have native streaming support. Streaming is something that you have to bolt on at the application layer and not something that's a native part of the infrastructure that you're building on top of.

[4:27](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=267s) Something that's really important when you're building an application that might run for multiple minutes, users might refresh the page, so you want resumability.

[4:35](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=275s) And so let's look at some of these common product experiences. We're gonna look at both onboarding, where we might ingest a bunch of data from the user with an LLM up front, and then also a product experience where we're gonna run an agent that's gonna take multiple minutes, and we have to keep the user engaged during that

[4:51](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=291s) time period. So during this onboarding experience, the user puts in their URL, right? And we use a single LLM call to grab some information from that web page and then detect all the pages that we're gonna scrape, right?

[5:04](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=304s) And now we've kicked off that scraping job in the background, and it's gonna take multiple minutes. Uh, and we're gonna be making hundreds of LLM calls to extract content from those web pages and enrich them.

[5:14](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=314s) And we want the user to be able to start using the product right away. So this ingestion can take, you know, a varying amount of time, but we want users to use the app immediately.

[5:25](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=325s) If we wait ten minutes, that means we're gonna have increased fall off in the funnel, so we have to run this in the background, right? And we wanna show users the status of this ingestion as they're using the product.

[5:35](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=335s) We want them to know, "Hey, as time goes on, we're gonna enrich more data, and your experience is gonna get better." [keyboard clicking]

[5:42](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=342s) Next, this is an example of building a content generation app, right? We're gonna write a blog post, uh, about building high-performance web apps with Vercel and Next.js. And this agent, we tell the user up front, "Sit back and relax.

[5:56](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=356s) This is gonna take a couple of minutes." And on the right, we're showing them all of the different steps, right? We're running research. We're writing an outline. And we're writing each section step by step.

[6:07](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=367s) Since this takes multiple minutes, it's really important that if the user leaves the page or navigates away, that they don't lose this context, right?

[6:16](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=376s) And so that workflow that we built when working on a previous product took three to four minutes, and we really felt a lot of friction experimenting with longer workflows.

[6:26](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=386s) Like, we were deterred from making our workflows longer than five-plus minutes because that meant that we would have to make deep infra changes due to limitations from our serverless provider.

[6:37](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=397s) Eventually, we did rip it out, and it was a bunch of work, but it took us much longer than we should have to run those experiments, uh, because of the amount of work that was required to m-migrate off of, you know, our easy-to-use serverless offering.

[6:52](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=412s) Also, we need to stream both the final content as well as the intermediate status, and if the user refreshes the page and comes back, we want them to see that same thing, just like you get with deep research, right?

[7:04](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=424s) And if there's an intermediate error, users are gonna be way more frustrated if it takes them five minutes to get back to that same point, uh, versus if it's just, you know, a simple webpage that you have to refresh or a form that you have to resubmit.

[7:18](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=438s) So we built a lot to make this easy in our core product, uh, when we were working on, you know, agentic workflows, uh, and we ended up turning a lot of this into an open source library.

[7:30](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=450s) So we built a simple component model that's infra-infrastructure aware, right? Uh, the, the framework itself is very aware of the infrastructure it's running on, and the infrastructure is aware of the framework.

[7:42](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=462s) And that's so that we can provide things like resumable streams for intermediate status, uh, and final output, and we call this a bit of an anti-framework. It's unopinionated and focused on building blocks.

[7:54](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=474s) Uh, it's-- it takes inspiration from React's component model and applies it to the back end 'cause we know that abstraction is bad when building agents. You don't want to abstract.

[8:03](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=483s) You want to share. You want to compose. You want to reuse code. So we have components that are reusable, idempotent, independently testable steps, and then workflows that are collections of components that run together.

[8:16](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=496s) So this is a simple component that uses a wrapped version of the OpenAI SDK. It's wrapped so that we can get a bunch of other tooling for retries and tracing and things like that, but it's sa- the same exact OpenAI surface area that you would expect.

[8:29](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=509s) We give you full access to the model, so it's just a prompt, a little bit of context that's injects-ingested, and it returns a response. So a simple function that takes in some prompts and returns some output, and it can be independently run or tested.

[8:43](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=523s) We can also combine it together into what we call a workflow. This workflow fetches the front page of Hacker News, analyzes those posts, generates reports, writes a tweet, and then edits the report.

[8:55](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=535s) And with this, every single step, we're gonna get a retry boundary, an error boundary off each component, and we're gonna get traces, both at the top level of the workflow, but also all of the components that run within it.

[9:09](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=549s) Each workflow can be turned into a set of REST APIs automatically by the framework that support synchronous invocation as well as asynchronous invocation and the relevant API so that we can retrieve the intermediate stream and the final output stream if needed.

[9:27](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=567s) When we run this workflow, we see that we get a trace. This trace shows us all of the nested components that run inside the workflow, including all the tokens, all the details on the OpenAI call, user messages, system messages, so it makes it super easy to debug.

[9:42](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=582s) And we have built-in retries. Every component has a fluent API hung off of it that allows you to configure a component with things like retries and a cache. So if I have one component that's particularly problematic, I can add an exponential retry policy, uh, with...

[9:57](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=597s) and, and configure the number of retries that run.

[10:01](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=601s) And so this all gets deployed to a serverless platform that we built that's tailor-made for long-running workflows and building agentic UIs, uh, for processes that run in the background.

[10:14](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=614s) So we have an API layer and a compute layer that are completely separate, and this is really important for a lot of reasons. The API, the API layer invokes the compute layer and passes it in a Redis Stream ID.

[10:27](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=627s) That's the last time that the API layer is gonna have to talk to the compute layer. The rest of it is going to happen via communication with this Redis Stream.

[10:34](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=634s) Status is gonna come from the Redis Stream. The output is gonna come from the Redis Stream. And as the sandbox program executes, it's gonna emit heartbeats so that the background processes within the system can monitor and make sure that a workflow runs to completion and doesn't die or doesn't crash.

[10:51](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=651s) And if it does, we can automatically restart it or notify a user.

[10:57](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=657s) So key architectural points of this is this separation between the API and the compute layer. It allows us to scale things independently, scale the API layer independent of the compute layer.

[11:07](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=667s) It also allows us to plug that compute layer, so we can provide, uh, a compute layer by default on top of ECS, but we can allow users to bring their own compute layer as well since it's completely stateless and just talks to the API layer and the Redis Streams to store output.

[11:23](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=683s) All that output goes to the Redis Stream, both for sync and async workflows, which means that we get resumability, right? The API layer only reads from the Redis Stream, not the compute layer directly, so that means that we can build UIs that allow users to refresh the page, navigate away, transparently handle errors, uh, and they still get

[11:41](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=701s) the full history of status messages and all of the output, um, something that you wouldn't get from a typical serverless like infra platform.

[11:52](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=712s) Like we said, key point here is being able to refresh the page, resume directly from the Redis Stream, and get all of your intermediate status messages. None of the work is lost if the new user navigates away or if the web browser terminates the connection to the API server.

[12:07](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=727s) So things and lessons to consider when you're building your own infrastructure for agentic workflows. Always start simple. You're not gonna write a workflow that runs for an hour on day one, so don't build for it.

[12:19](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=739s) But do plan for a future that's long-running. In my opinion, this is where agents are going, and I think that we can see this with all of the recent coding agents that have come around, experiences like deep research.

[12:30](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=750s) Like, the future is giving your agent instructions and letting it go off and do work for you and communicate with you when you need it. And to make that happen, think about keeping your compute and API plane separate and lean on Redis Streams for resumability.

[12:45](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=765s) Make it easy to u- for users to navigate away from the page, not lose progress, and handle errors transparently. All this is also to say you gotta take care when deploying.

[12:55](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=775s) If you're gonna let your workflows run for sixty minutes, you have to be very, very careful about how you drain workers and how you, uh, do kind of a blue-green deployment pattern.

[13:04](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=784s) And overall, the devil is really in the details. Easy to get started, but very hard to get this right.

[13:11](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=791s) Now, it's really, really fun to build all of this infrastructure yourself. But if you don't wanna build it yourself, I'd encourage you to check out GenSX on GitHub, where we've implemented a lot of this for you.

[13:23](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=803s) Again, my name's Evan. Thanks for having me at this talk, uh, and happy building.
