AI Engineer Code 2025
Infra that fixes itself, thanks to coding agents — Mahmoud Abdelwahab, Railway
Read the talk
From unhealthy infrastructure to a reviewable pull request
Railway Autofix connects infrastructure metrics, durable workflows and a remotely controlled coding agent to turn a suspected service problem into a pull request for human review.
From a talk by Mahmoud Abdelwahab
Before you start: Familiarity with HTTP status codes, GitHub pull requests and basic service metrics will help you follow the workflow.
A memory leak is obvious; a slow query may not be
The Railway dashboard opens on several deliberately unhealthy services. In the first, memory usage keeps climbing, suggesting a leak that could eventually crash the process. Its request charts also show many HTTP 500 responses. Mahmoud Abdelwahab reports a 94% request error rate and responses taking multiple seconds for this demo service. These are dashboard readings from the demonstration; the request denominator and measurement window are not specified.
In production, that combination would likely mean pages and urgent investigation. But the next service illustrates a less obvious failure: it queries Postgres, its CPU usage looks ordinary, and its memory chart is somewhat spiky without immediately looking disastrous. There are failed requests and an elevated error rate, but the strongest signal is extremely high response latency. Slow database queries make the experience painful even when resource charts look relatively normal. Abdelwahab illustrates the user impact with a roughly 30-second page load.
The usual response is to configure thresholds for CPU, memory or request errors, receive an alert, and then assemble the explanation yourself from logs, metrics and traces. An alert identifies a symptom; a useful repair needs the surrounding evidence. The proposed change is to let a coding agent monitor that evidence and prepare a fix, so the operator’s next task becomes reviewing and shipping a pull request. Human approval remains part of the workflow.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Monitor a time window, then investigate affected services
The path from a Railway incident to a GitHub pull request begins with scheduled project-health monitoring. Suggested intervals are every 10, 15 or 30 minutes. Each run performs a small sequence:
- Fetch the application architecture: deployed frontends, backends, crons and queues.
- Fetch each service’s CPU and memory utilization.
- Fetch HTTP metrics, including error rates and counts of 400 and 500 responses.
- Compare the measurements with thresholds and return the affected services.
The output is a list of candidates for investigation, not yet a diagnosis.
Why poll instead of starting the workflow from an alert webhook? Abdelwahab prefers analyzing a slice of time because individual threshold crossings can be noisy. An illustrative spike to 80% CPU utilization warrants investigation, but does not by itself establish a fault. A spiky workload may still be operating correctly when viewed alongside the rest of its behavior.
The next workflow gathers more context for suspicious services. High resource utilization with clean logs may simply mean the application is getting more use. A proposed extension would scan repository code to identify upstream providers and then check their status pages. If a payment processor is down, the appropriate recommendation could be to wait for that provider to recover rather than modify application code.
When the evidence does point toward an application defect, combine the signals: many HTTP 500s, high memory utilization and errors identifying a failing endpoint. Add the application architecture and affected-service details, then turn that context into a repair plan. The coding agent can clone the repository, derive a to-do list, implement the changes and open a pull request. This separates broad monitoring, focused investigation and code editing into distinct stages.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resume failed workflows without repeating successful steps
These workflows depend on several external services, so failure recovery is part of the design. Abdelwahab introduces durable execution through Inngest, using a separate processVideoUpload example. A video-upload event starts three operations: call a transcription API, ask an LLM to summarize the transcript, and store both results in a database. Each operation can fail independently.
Durable steps provide automatic retries, with retry behavior and failure handling configurable; the walkthrough mentions exponential backoff as an option. Successful step outputs are persisted and reused. If transcription and summarization succeed but the database write fails, recovery can reuse the first two results and retry the write.
The step boundaries in that example can be expressed in TypeScript as follows. The application supplies its transcription, summarization and storage functions; each external operation runs inside its own durable step.
typescript
import { Inngest } from "inngest";
type VideoOperations = {
transcribe: (videoUrl: string) => Promise<string>;
summarize: (transcript: string) => Promise<string>;
save: (record: {
videoUrl: string;
transcript: string;
summary: string;
}) => Promise<void>;
};
export function createVideoWorkflow(operations: VideoOperations) {
const inngest = new Inngest({ id: "video-processing" });
return inngest.createFunction(
{ id: "process-video-upload" },
{ event: "video/uploaded" },
async ({ event, step }) => {
const videoUrl = String(event.data.videoUrl);
const transcript = await step.run("transcribe", () =>
operations.transcribe(videoUrl),
);
const summary = await step.run("summarize", () =>
operations.summarize(transcript),
);
await step.run("save", () =>
operations.save({ videoUrl, transcript, summary }),
);
return { videoUrl, stored: true };
},
);
}
| Step | State when the write fails | Recovery |
|---|---|---|
| Transcribe | Result persisted | Reuse transcript |
| Summarize | Result persisted | Reuse summary |
| Save | Failed | Retry database operation |
The handler can execute again; successfully persisted steps do not need to repeat their external work. That is why API and database calls belong inside durable steps. Abdelwahab attributes the speed and cost benefits to avoiding repeated work, then applies the same pattern to Railway API calls for architecture, resource metrics and HTTP metrics.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Run the coding agent behind an API
For the coding agent, the demo uses OpenCode, an open-source terminal agent presented as an alternative to Claude Code, with a choice of model providers. The architectural feature that matters here is its client-server separation: running opencode starts a terminal UI and a server. The terminal is one client of that server, so the infrastructure workflow can become another client.
A headless OpenCode server hosted on Railway exposes the agent through an API. Its environment supplies the tools the agent needs, including filesystem access and configured Git. The workflow service can therefore request repository work remotely while the agent performs edits and prepares pull requests in its own server environment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Package the agent and its tools
The Railway Autofix repository separates the orchestration API and OpenCode into two directories. In the recording, Bun runs a small server wrapper that Abdelwahab calls createOpenCodeServer, listening on port 4096. The current companion Dockerfile instead launches opencode serve directly; the recorded wrapper’s exact historical SDK identifier is not established by the current code.
The container defines the agent’s working environment. It installs curl, jq, Bash, Git and the GitHub CLI, followed by OpenCode. Git is configured, the server port is exposed, and GitHub CLI authentication enables pull-request creation. These tools are what let the remote agent move from a natural-language repair plan to concrete repository operations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Inspect a monitoring run
With OpenCode running, the orchestration API starts at localhost:3000. Inngest’s debugging UI lists functions, each representing a workflow with inspectable steps. The intended live sequence is scheduled monitoring followed, when an issue is detected, by service-context collection and fix generation:
monitor project health → pull service context → generate fix
The demo starts this chain manually and shows the downstream workflows being invoked. Railway-specific environment variables provide deployment context; Abdelwahab notes that Railway supplies them automatically on the platform.
The first monitoring step retrieves the project architecture. Its output identifies the database, deployed services and their configurations, repository locations, and volumes. This gives the later agent more than an isolated error message: it knows which pieces of the application exist and where their code lives.
Independent metric-fetching steps then run in parallel. Their outputs include readable summaries for the coding agent. In the displayed database summary, Abdelwahab reports average CPU usage of 0.93 vCPU and memory usage of 31.96 GB against a 32 GB maximum. The aggregation window is not specified, but the memory reading supplies a concrete reason to investigate pressure near the displayed ceiling.
HTTP metrics are collected for each of the three deployed services. They include separate error percentages for 400 and 500 responses, latency, and status counts. Formatting these measurements into summaries gives the next workflow a consistent account of how each service is behaving, alongside the resource measurements and architecture.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn service context into a repair plan
The pull service context workflow receives the monitoring payload and fetches HTTP logs, build logs and deployment logs for affected services. It also carries an architecture summary describing three services, one database, volumes and an environment labeled production. Although the debug UI displays this text awkwardly on one line, the agent receives it as Markdown. The collected context then passes to generate fix.
Fix generation begins with an AI analysis step. Architecture and performance evidence become a plan containing debugging steps and recommendations. One displayed recommendation is to reproduce the problem locally under the same load, run the application and investigate errors encountered along the way. That is a proposed debugging action in the plan, rather than a demonstrated reproduction result.
The workflow passes the recommendations to the coding agent and creates a session. A session acts like a separate chat context: with multiple repositories, each repository can have its own session. The agent works from that context toward the expected completion condition—an open pull request containing its changes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The demonstrated endpoint is a pull request
The final screen shows an open pull request. Its conversation includes a change summary, an analysis summary, identified root causes and descriptions of what was fixed. This is the concrete handoff: the operator can review a proposed repair together with the agent’s explanation, rather than begin with only an alert.
Review and merge remain human decisions. Abdelwahab’s next step is to merge if the changes look good; the recording does not show a merge, redeployment or post-fix health measurement. The demonstrated result is therefore the path from detected infrastructure trouble to a reviewable code change. He closes by inviting questions on X/Twitter and pointing viewers to the companion repository.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Companion source code for the infrastructure-monitoring and coding-agent demo, organized into API and OpenCode services.
Documentation for running a headless OpenCode server and controlling it through an HTTP API.
Further reading
- Inngest durable executionDocumentation
Explains how persisted step results and retries let workflows resume after failures.
Updates since the talk
Current JavaScript and TypeScript interfaces for starting OpenCode or connecting to an existing server.
Current CLI commands for retrieving resource and HTTP metrics with time filters and JSON output.
Read the complete timestamped transcript
- 0:00
Your app's infrastructure should fix itself. Let me show you. So right now, I'm on the Railway dashboard, and I have a bunch of services that are deployed, and all of these services have one thing in common.
- 0:09
They all have bugs and problems. So, for example, this service has a memory leak. If I click on it, go to Metrics, we can just see memory utilization keeps growing high and very quickly.
- 0:22
This is just a sign of a memory leak, and pretty sure the service would eventually crash. If I look over at the amount of requests, we have a high number of 500s, so the server is failing to respond.
- 0:34
We have a high request error rate of 94%, and we also have an extremely high response time of, like, multiple seconds, uh, for, like, the service to respond, which is not ideal.
- 0:47
Like, if this was a service running in production, everything would be on fire, you'd be getting paged, and you'd just try to bring back the service, uh, back quickly.
- 0:57
But the thing is, not all problems are this obvious. For example, this service, all it does is just queries a Postgres database, and if we go to Metrics, we'll just see that, well, CPU utilization seems fine.
- 1:11
Memory usage is also fine. Sure, it's a bit spiky, but okay, whatever. We have some fails, but okay, nothing too alarming. Request error rate is somewhat high, so that also should make us kinda, like, wanna investigate, but the response time is extremely high.
- 1:31
The thing is, this is because the service makes queries that are super slow, and the thing is, if you're an end user that's trying to use this experience, you would just suffer.
- 1:41
You would need, like, 30 seconds for, like, a page to load, which would be a nightmare. So, the thing is, when you deploy your app to production, maybe some, you know, bugs or issues make their way to production.
- 1:54
Things happen. And kinda like the typical way of dealing with these things is maybe you set up a bunch of thresholds, and when these thresholds are met for, let's say, CPU or memory utilization, maybe, uh, you wanna have a threshold for the request error rate that shouldn't exceed a certain amount.
- 2:14
Well, what will happen is you're gonna get alerted, and you'll be aware that there's an issue, but you still have to do the investigation yourself. You have to dig through logs, metrics, and traces to try to paint a picture in your head and try to piece things together so that you can ship a fix.
- 2:31
Now, what I'm proposing is you should have a coding agent that monitors the state of your project and your application's infrastructure, and if any issue is detected, so, you know, any of the thresholds we define are met, we should just have a fix shipped, right?
- 2:48
So, like, instead of, you know, getting the alert and investigating, you just review a pull request, and you're like, "Oh, looks good to me." You ship it, and then everything is good and crisis averted.
- 3:01
So today, I'm gonna show you what I have in terms of demo that kind of paints a picture of how this could be achieved. So at a high level, I wanna have a series of workflows that will kick in that will help me go from issue detected in, on Railway, my deployment provider, to a pull request being
- 3:21
open in my GitHub repo, and this is what I have in mind. Uh, the first workflow that I wanna have is a workflow that runs on a schedule. So let's say it runs every 10 minutes, 15 minutes, 30 minutes, and what this workflow will do is, one, it should fetch the application's architecture.
- 3:37
We should have an understanding of what services are deployed, which, you know, like, front ends, back ends, crons, queues are live in my project. And I then want to fetch each service's resource metrics, so CPU and memory utilization, and I also wanna fetch each service's HTTP metrics.
- 3:57
So I wanna see the request error rate, the number of failed requests for, you know, 500, 400 errors. And once that's done, I will want to then see which services have exceeded which thresholds, and then I just wanna return a list of the affected services.
- 4:15
So this would be essentially the goal. Now, you might be wondering, "Well, why not make this an alert-based system?" So maybe we configure something like webhooks for alerts, and then that would kick off, uh, essentially this workflow instead.
- 4:30
I would argue that it's probably better to be able to analyze a slice of time rather than just having a threshold being met, because it can get pretty noisy.
- 4:40
Like, imagine you have a spiky workload, uh, and, you know, you reach that 80% resource utilization for, like, your CPU, but things are still fine, and that's not... Like, in my mind, this is enough to be investigate, but it might not...
- 4:57
Like, it might mean that there just aren't issues when we try to look at, uh, like, the bigger picture and all of the details.
- 5:06
Now, once we have this list of impacted s- impact services, we essentially wanna pull in even more context for them. So, like, at a high level, we wanna see project health, all of the services.
- 5:17
Is everything operating as expected? Oh, we have this thing that we're suspicious about. Let's actually pull all of, you know, additional context for the service. Because imagine, again, you have, like, high resource utilization.
- 5:29
Maybe you're just successful. [laughs] You have high usage. Uh, but then when you pull the logs, it's like, oh, everything seems fine. There aren't any errors. Well, you're good. And you can imagine that we can even pull even more context.
- 5:42
Like, imagine maybe we scan the code in the repo, and based on that, we infer the upstream providers that the repo relies on, and then we can automatically check the status pages of these services.
- 5:53
Imagine, like, a payment processor goes down. Well, that's kinda how you can know, and then the coding agent will be able to maybe tell you, like, "Hey, you should just, like, wait out this issue."
- 6:06
And once we have all this information, we can just write a detailed plan. So, like, we can look at, oh, we have a high number of 500 requests. We see that we have very high resource utilization for memory, and we see that we have, you know, um, just errors specifying that a specific endpoint is failing.
- 6:28
Well, this is enough information that we can write a detailed plan of, "Hey, this is my application's architecture. These are the affected services." We just then give this plan to an agent, and then the agent will just follow the process of, "Hey, let me clone this repo.
- 6:43
I'll just create a to-do list based on the plan you gave me. I'll implement all the fixes, and I'll just create a pull request." And this is kinda how we go from issue detected to an open pull request.
- 6:55
So let's actually sh- see this in practice. So because we have the idea of workflows, what I wanna do is actually use what is known as durable execution. So the idea of durable workflows has been around for a while, and it's really one of my favorite abstractions because it can help you simplify complex logic while making it
- 7:14
more reliable. So for example, here, we have this workflow. So this actually is Inngest, but there are lots of solutions out there that pretty much do the same thing.
- 7:24
And we have this function that, you know, called processVideoUpload. It listens on an event of video uploaded, and we essentially wanna do three things. We first want to generate a transcript, and we do this by making an API call to a third-party API.
- 7:40
Once we get that transcript, we wanna generate a summary by also making a request to an LLM provider. And once we have the transcript and the summary, we wanna store them in a database.
- 7:50
The thing is, all of these steps, they are not 100% guaranteed to work. Uh, they are prone to failure. And what's neat about this pattern is, by default, these steps will be automatically retried.
- 8:04
You don't even have to think about it. But if you, let's say, want to modify this behavior, maybe you want the retry to happen, uh, like, on a certain schedule, like, you know, exponential backoff, uh, maybe you want to define another thing that should happen in the case of failure, you'll be able to do it.
- 8:23
But what's neat is each step, when it succeeds, uh, the result is cached. So if, for example, we are able to transcribe the video correctly, we summarize the transcript correctly, but we failed to write to the database, if we were to retry this workflow, we just continue where we left off.
- 8:40
Uh, we don't... We won't really repeat any work, which is, one, awesome 'cause it's faster, but also it's more cost effective. So at a high level, this is the thing that I'll be relying on in my code because I'll be making API calls to the Railway API to be able to fetch the project architecture, all of the
- 8:56
resource metrics, um, as well as, you know, the HTTP metrics and whatnot. So yeah, uh, this is kinda like the first thing that, um, we need to talk about.
- 9:08
The second thing is the coding agent. And for the coding agent, I'll be using OpenCode. OpenCode is an AI agent that's built for the terminal. You can think of it as an alternative to something like Claude Code.
- 9:20
But the main difference is OpenCode is fully open source, and you can choose any LLM provider or, uh, you know, model that you like, which is pretty nice. Uh, you have this nice terminal UI, but honestly, what's so cool about the project is how it's architected.
- 9:36
So if you go to their docs, they actually have a server implementation. You can have a, a, a headless server that runs, that exposes an API for you to essentially interact with an agent.
- 9:50
So the way it works is when you run the command OpenCode, which is what starts up the agent in your terminal, it doesn't just run a single app. It actually starts a terminal UI and a server.
- 10:02
And because the terminal UI here is the client, we can essentially bring our own client and talk to the server, which is awesome, uh, because now we can run OpenCode on a server, in this case, it would be on Railway, and we can just have this server have all the tools that the agent would need.
- 10:20
So we'd install all of the necessary, you know, tools. We can configure Git, and then the agent will be able to open pull requests and, you know, go through the file system and do everything.
- 10:32
Let me show you what... how easy it is to essentially have this deployed on Railway. So if we go to the code, uh, here right now, this is my project.
- 10:40
It's called railway-autofix. I know, great name. Uh, I have essentially two directories. One is for my API. The other one is for OpenCode. And OpenCode, really, we just have a single server running using Bun, and all we're doing is we're just calling a function, uh, that is called createOpenCodeServer.
- 11:01
So if I actually stop this here, you can see it runs on port 4900, uh, 4096, and this is pretty much all we need. And I have a Docker file, and in this Docker file, we're essentially defining the environment.
- 11:16
So we're installing a bunch of tools. You can see we're installing curl, jq, Bash, all the other tools, even Git. Uh, we're installing the GitHub CLI, which is what will allow us to open pull requests against a given repo.
- 11:28
We're then installing OpenCode in the environment. We're configuring Git, and at the end, we're just exposing the port, and we're just authenticating the GitHub CLI, which is pretty neat.
- 11:39
Uh, by the way, the code will be linked somewhere down below. But that's really it for OpenCode. And when it comes to the actual API, let me actually run it.
- 11:49
So now the... This is the OpenCode server that's running. And if I go here, I have my actual API running on localhost:3000, and I have a UI that is provided by Inngest, which is very useful for debugging.
- 12:04
So if I go here And I go to functions. Essentially, each function here is a workflow, and it has a bunch of steps. So let's actually try to run it to see what happens.
- 12:16
Uh, now in production, when this is live, this monitor project health workflow should run on a schedule, and if an issue is detected, we will call the pull service context, and then pull service context will call the workflow for generating a fix.
- 12:33
So if we actually just kick things off, this is how the flow of things will happen. So if I actually have now, I have this function run, we called monitor project health, then we call pull service context, and now we're actually calling generate fix because we detected an issue.
- 12:48
And we're just setting, um, like the Railway-specific variables as environment variables, and all of these are actually available, uh, on Railway. They're just set automatically, which is pretty neat.
- 13:00
So if I actually go to monitor project health, you'll see we have a bunch of steps. Uh, the first one is getting the project architecture, and this step right here, this is...
- 13:10
We can actually see its output. So we can see all of the databases that I have in my project. I just have one. Uh, we can see also a list of all the services as well as their configuration.
- 13:21
We can see which, like, where is the repo for them, and we just now have a high-level overview of our application's infrastructure. Uh, we also see that we have any kinda like volumes that are there, which is cool.
- 13:36
And then we have a series of steps that are actually running in parallel, so like, you know, things are efficient. So we're getting the database resources. We can see on average, well, what's the max CPU?
- 13:47
Uh, and it's like 0.9 CPU. Okay. Same thing for memory. And we actually have a summary, and this summary essentially is us formatting these results so that we can then pass it to the coding agent.
- 13:59
So you can see CPU usage, average 0.93 vCPU, um, and you know, this is the max, and memory usage as well. Now, this is actually high, uh, and we'll be able to kinda understand that because it's like, oh, memory usage here is 31.96 gigabyte out of a max, which is 32 gigs.
- 14:18
Uh, and then we just pull even more, um, like resources. So, like, because we have multiple services, we will call each step for it. So, like, we will pull the HTTP metrics for each of the three services that we have deployed, for example.
- 14:32
But also, for this one, for the HTTP metrics, we can see the error rate percentage for 400s, for 500s. We see, like, the latency, um, and we just have like a status count.
- 14:44
So we can also have a summary, and then we can say, "Hey, these, this is the rate of, um, like request error rates. This, these are the latencies." And this way, when we actually, at the end of, like, this workflow, so if I go to runs, go here again, towards the end, we will actually give this, uh,
- 15:06
pull service context function just all of this information in a nicely formatted way. So if I actually go now to this function run, we will see here that we're fetching the HTTP logs, the build logs, the deployment logs for, like, all the services that are affected.
- 15:23
And we can see here, like this is the function payload, uh, so this is the stuff that we passed from the other function. And we can see we just have all this info.
- 15:33
We also have an architecture summary. So this, actually, we can expand this. Uh, the architecture summary is just a nicely formatted, uh, text saying, like, "This is the project architecture.
- 15:43
We have three services. We are running in the production environment. We have one database. We have all these volumes," and we just have all of this information. It's just harder to read 'cause it's like in one line.
- 15:55
But for the, um, coding agent, we'll just give it to it as, like, markdown. So now that we have that, I'm just go to runs again. Now that we have that, we are just going to make a call to another workflow, which is generate fix.
- 16:11
And for this one, what it does is, one, it will analyze with AI. So this is the actual output in terms of, like, the input. It's a bit large to render here.
- 16:22
Uh, but we analyze it with AI. So, like, you can imagine we give a large language model saying like, "Hey, this is my project architecture. This is the data.
- 16:31
This is how things are performing." And then we take all of this information, and now we actually come up with a plan. So you can see here debugging steps.
- 16:41
We want to see reproduce locally with the same load. Maybe we want to run it. We want to see what will happen. If we see that the agent is like, "Oh, I ran into an error," then it's going to fix it.
- 16:53
And then we have like recommendations. So, like, this is the plan that we'll then just pass to our coding agent, and then we have a step to create a session.
- 17:01
So on the coding agent, you can imagine each session being its own chat. So this will run, like imagine you have multiple repos. Each repo will have its own session.
- 17:11
The coding agent will work, and then at the end, it should, you know, if, as expected, it should open a pull request. So yeah, that's pretty much it. This is how it works.
- 17:22
Now, if everything works as expected, we should see a pull request on the project. And here we go. We have a pull request that is open with all of our changes.
- 17:33
If we go to the conversation, we'll actually be able to see that we have a summary of all the changes, uh, an analysis summary, the root causes, what was fixed.
- 17:42
So we should be able to just review this. If everything looks good, we merge, and we're good to go. And that's it. Hope you enjoyed this talk as much as I enjoyed making it.
- 17:52
If you have any questions, feel free to reach out to me on X or Twitter. This is where I mostly hang out. Also, the repo for this project will be available somewhere down below, so make sure to check it out.
- 18:03
And with that, thank you so much for watching, and I'll see you in the next one.