AI Writes the Code. So What's Left for Engineers? - Laurie Voss, Arize AI
Read the talk
AI Writes the Code. So What’s Left for Engineers?
Laurie Voss follows the bottleneck from code generation to review, operations, and the harder task of defining what people actually need—then asks how engineers should work and learn when agents do the typing.
From a talk by Laurie Voss
At a glance
Ideas worth remembering
Cheap code generation moves the delivery constraint into verification. More code produced is useful only insofar as it becomes software worth shipping.
A maintainer’s merge decision includes context beyond passing tests. Specs, rules, rubrics, and evals can make more of that judgment available to agents.
“Keep track of my orders” leaves the essential requirements unresolved. Product engineering discovers what that request means for a particular customer and defines a useful result.
Replacing junior implementation tasks also weakens an apprenticeship. Teams need deliberate training in customer conversations, specification, and judgment.
Measure shipped work, useful review findings, maintainer acceptance, and production behavior; concentrate human attention on consequential changes.
Eight times the code does not mean eight times the shipping
Writing plausible code has become dramatically cheaper. Establishing whether to trust it still takes work. That mismatch drives Laurie Voss’s forecast for engineering: accelerate implementation, and the work piles up at review; accelerate review, and another constraint takes over. Voss, head of developer relations at Arize AI and co-founder of npm Inc., extends the rise of the AI engineer into a question about every software engineer’s job: what remains when agents can handle more of the delivery process?
The opening discrepancy is between code produced and software shipped. Voss cites developers switching on autonomous coding agents producing 741% more code, roughly eight times as much, while shipping only 30% faster. Those are reported study figures, rather than a general productivity guarantee; the recording does not establish the study’s conditions. Even so, the comparison frames the right question: how much of the extra implementation actually makes it through delivery? Thirty percent is a substantial gain. The much larger increase in code suggests that another part of the process is absorbing the load.
Human review cannot simply expand to match that output. Voss cites an older Cisco study in which review accuracy fell sharply beyond about 450 lines per hour. His concrete warning is a 10,000-line agent-generated diff: inspecting it carefully becomes a substantial block of work, which he describes as about a week. Generating the diff quickly has transferred the effort to the person who must understand it. Larger output alone has not removed that effort.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Passing tests leaves out the maintainer’s decision
Automated review is the obvious response, and Voss describes it already taking a significant share of the work. He reports that GitHub Copilot performed more than one in five GitHub reviews in the preceding month and describes OpenAI’s internal workflow as permitting pull requests without mandatory human review. These examples establish the direction of adoption. They do not establish that machine review catches everything that matters.
A comparison of four AI reviewers exposes a useful distinction between finding bugs and providing dependable coverage. In Voss’s account, all four found real issues, but they found different issues and agreed on no individual bug. Each reviewer contributed something useful; any single reviewer left findings uncovered that another could detect. That is his reason for recommending multiple reviewers to teams taking automation seriously.
Skipping review altogether has produced painful reversals in the cases Voss recounts. Dexter Horthy abandoned advice to stop reading generated code after systems needed to be ripped out and replaced. Voss also describes Amazon responding to agent-related outages by requiring senior engineering sign-off, and an Anthropic postmortem about bugs that escaped both human and machine reviewers. The failures differ, but they all challenge the assumption that an agent has completed a change correctly on its first attempt.
Cognition’s benchmark asks a more demanding question than whether a patch passes tests: would the repository’s maintainer merge this pull request? In the comparison Voss cites, the pass rate falls from 87% to 54% when judged by that standard. A merge decision includes code style, side effects, interactions with external systems, and project history that may live only in the maintainer’s head. Tests cover some requirements; the maintainer brings additional requirements and context. The gap is a concrete example of information the agent has not been given.
That missing information is Voss’s first version of “what good means.” A change can satisfy an executable test suite and still be wrong for the system it must join. Improving review therefore requires more than a stronger bug detector: it requires making the reasons for accepting or rejecting a change available to the reviewer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
After review, the constraint moves again
Eliyahu Goldratt’s theory of constraints supplies the model for what happens next. In Voss’s account of The Goal, improving a part of a system that is not its bottleneck does little to improve total throughput. Once the bottleneck is relieved, another part becomes limiting. Faster code generation has brought review into focus; successful automated review would move that pressure elsewhere.
The proposed next constraints concern running the software: deploying it, scaling it, and making it secure and private at scale. Voss explicitly makes a leap here. He assumes progressively better agents will eventually handle these tasks because humans have accumulated substantial knowledge about them. This is a forecast with thin evidence for some stages, not an account of a completed autonomous delivery system. He expects progress on review over the following year, while placing operations, scaling, and security on a five-to-ten-year horizon.
A second assumption determines whether cheaper software supports more work for humans: demand must remain effectively unlimited. The everyday examples are bad websites at a hairdresser, a bank, or a government department. Better versions are possible, but the people who need them cannot always afford the engineering effort, and capable developers are occupied elsewhere. If production gets cheap enough, previously uneconomic improvements become possible. Voss’s prediction of a larger profession depends on that unmet demand turning into more software.
His resulting ordering separates implementation work from decisions about the product. Writing code has already become cheap in his framing; review and maintenance are improving; deployment and scaling come later. The work he expects to remain with people is deciding what to build, defining success, and making the result delightful. The rest of the talk explains why those decisions might resist the same economies of scale.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
“Keep track of my orders” is not a specification
A customer says, “I need to keep track of my orders.” Thousands of products fit that sentence. Choosing or building the right one requires discovering what the sentence leaves out. A bakery, an auto parts store, and a large manufacturing business all track orders, but they need different software. The initial request names a category of activity; it does not describe how that activity works in this particular business.
The next step is conversation: what does an order mean here, what must the system do, and how does the business operate? The necessary knowledge belongs to the person running that business. A model’s general knowledge of order management cannot supply facts that this customer has never expressed. Voss’s deliberately bodily description—“flapping your meat parts”—lands on a practical task: talk to the customer until the request becomes specific enough to build.
That discovery work transfers poorly between products. A calendar app and a scheduling app address closely related needs, yet their requirements and definitions of success differ. Adding configuration options does not guarantee a fit either: Voss points at Jira as the familiar experience of software that can be configured endlessly and still fails to do what someone needs. Cheap implementation makes this specificity more consequential. Producing another system is easy only after someone has described the right system.
Fred Brooks’s distinction helps name the remaining difficulty:
- Accidental complexity: Difficulty introduced by tools, languages, syntax, and the mechanics of implementation.
- Essential complexity: Difficulty in the problem itself—understanding what must happen and what a satisfactory solution means.
Voss uses the modest shipping gain beside the large code-output gain to illustrate why reducing implementation effort does not remove the whole problem.
Sean Grove’s description of code as a “lossy projection of the specification” pushes the value upstream. The specification contains intent that may not survive fully in the implementation. Whoever can express that intent precisely—a product manager, engineer, lawmaker, or marketer—can direct what gets built. In Voss’s forecast, writing down what good means becomes a central engineering skill.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Written judgment becomes something agents can use
There is a tension in treating judgment as the remaining human advantage: once people express it clearly enough, automation can use it. Cognition’s mergeability benchmark attempts to capture more of the maintainer’s acceptance criteria so models can train against them. Voss also describes Cursor’s Bugbot turning team thumbs-down feedback into rules, reporting roughly 44,000 rules. In both examples, knowledge moves from an individual’s judgment into a reusable representation.
This is why Voss expects operations and security to become more automatable: their requirements can increasingly be written down, checked, and used to improve agents. His proposed human role keeps producing those definitions. The baker’s unstated needs must first be discovered; after they become a specification, agents have something concrete to implement. The scarce work lies in getting from a person’s situation to a useful definition of success.
That shift also changes the division between product management and engineering. Voss cites Andrew Ng describing a team’s proposal for twice as many product managers as engineers, then favoring a closer combination of the jobs: an engineer who understands users can decide what to build and build it directly. The historical comparison is the systems analyst, whose job was to talk to the business and turn its problem into requirements. Product engineering recombines that discovery work with delivery.
Voss calls the combined role the product engineer: someone who finds out what people need, defines what good looks like, and ships it with agents doing the typing. Gergely Orosz’s earlier description of the product-minded engineer supplies a recognizable predecessor. Voss’s stronger claim is that product judgment moves from a desirable trait, often associated with senior engineers, to an essential skill for engineers generally.
Forward deployed engineers provide his market example of a related role. He reports postings rising 800% over nine months of 2025 and average compensation of $240,000, interpreting that pay as recognition of senior engineering judgment. The labels remain unsettled: product engineer is his chosen name for a cluster of responsibilities, rather than a standardized title shared across employers.
This forecast makes software development a broader activity than programming. It also carries an objection Voss raises himself: people discussing AI often conclude that everything can be automated except the part they personally enjoy. Discovering the problem is his favorite part of software work. His bet that it will remain central deserves that qualification; it is a reasoned forecast about missing customer knowledge, not proof that agents can never participate in discovery.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The old training task is becoming the agent’s task
The traditional junior assignment was a well-specified ticket: implement this behavior, then have a senior engineer explain what needs correcting. Repeated implementation and review helped beginners acquire judgment. Agents now receive that same kind of assignment. If employers stop hiring people for those tickets, they also remove a route by which people learned to become the senior engineers they still need.
Voss presents an employment graph with a sharp divergence: developers aged 22–25 lose ground while those aged 41–49 gain. The comparison makes visible a problem that an overall employment average can hide. Age is an imperfect proxy for experience, as his companion labor-market analysis acknowledges, and the chart alone does not establish AI as the cause. His professional concern is the shrinking entry route while experienced judgment remains valuable.
Existing product training programs suggest a possible shape for the replacement, but Voss argues that their scale is inadequate. He describes Google’s Associate Product Manager program as producing about 50 people a year, estimates about 1,000 annually across comparable programs, and calls for at least 10,000. The demand figure is his estimate of what the transition requires. The mismatch he wants leaders to address is deliberate training in customer understanding versus the much larger pipeline that previously learned judgment through programming work.
His proposed exercise returns to the bakery: sit with a baker for a week and come back with a specification. Data structures and React teach useful implementation skills, but that assignment trains someone to discover and express a business problem. Voss points to IBM redesigning entry-level roles around customer conversations and specifications as an example of the direction. Engineering leaders should build such learning into real jobs, rather than wait for product judgment to appear as an accidental byproduct of the old apprenticeship.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Loving the craft does not mean loving the replacement job
Some programmers chose the profession for a clean abstraction, a stubborn bug finally yielding, or the pleasure of making a machine obey. Interviewing a baker is a different attraction. A forecast of more software and more developers does not compensate someone whose preferred work becomes a smaller part of the paid job. Voss treats that as a real loss and offers no easy consolation.
For some people, though, the satisfying moment may have come just before the typing: when a vague problem resolved into a repeatable system in their head. That activity remains central to his product engineer. Programming as a craft can also survive independently of its market position, as woodworking survived furniture factories. His analogy separates the desire to keep making things from the question of which tasks employers will pay large salaries to perform.
The transition could still be ugly. Tools are arriving after the load they must handle, and replacement training is arriving after the junior ladder has weakened. Voss looks to compilers, frameworks, and open source as earlier changes that removed large amounts of typing while expanding what software developers could accomplish. His ten-year prediction is a larger profession with comparable pay, organized mostly around product engineering. It remains conditional on the automation and demand assumptions that opened the argument.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make quality explicit, then measure what ships
The immediate recommendations concern where to spend human attention and how to make automated work verifiable:
- Identify code with a high blast radius. Write down which changes can cause widespread harm and concentrate human review there as automated review improves. This is the direction Voss recommends preparing for; his earlier failure examples explain why it cannot rest on assuming current agents always get changes right.
- Delegate work that is cheap to verify. Ask how to determine automatically whether an output is good, then give that check to the agent. A clear success test can support repeated automated attempts without requiring a person to inspect each one from scratch.
- Express expert judgment in specs, rubrics, and evals. Turn criteria that live in experienced people’s heads into artifacts that agents can use to build and assess the result.
The measurements should follow the outcome rather than celebrate generation. Track how much software actually ships, how often review comments lead to action, and the gap between passing tests and earning a maintainer’s approval. Each answers a different question: is work getting delivered, are review findings useful enough to change the code, and are automated checks capturing the acceptance criteria that matter? A 741% increase in code output cannot answer those questions.
“The last reviewer is production.” Deployed behavior encounters conditions that earlier checks missed. Voss describes another model grading that behavior as an eval, extending quality assessment beyond the pull request. This makes the definition of good operational: it must be useful for judging what the system actually does, as well as what its tests say it can do.
The product engineer’s work is to turn a sense of the problem into a specification precise enough for agents to build from, while keeping the result connected to the people who need it. Finding those needs, writing down what good means, and making something people like are the responsibilities Voss expects to spread across engineering. His closing line gives the forecast its bluntest form: “We are all product engineers now.”
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
Voss’s companion analysis develops the age-based employment comparison, discusses alternative explanations, and links its labor-market sources and data.
- LLM Evaluation BasicsReference
A practical introduction to deterministic checks, LLM judges, tracing, and explicit evaluation criteria for variable model outputs.
A worked financial-agent example covering trace inspection, custom rubrics, judge calibration, and comparisons between agent versions.
Related talks
- The Death of the Code Review: What the Data Actually Says — Laurie Voss, Arize AI
Expands the review bottleneck discussed here, including mergeability, reviewer evaluation, and the context that tests miss.
- Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI
Follows a concrete budget-filter failure through traces, a proposed repair, and verification against explicit product criteria.
Read the complete timestamped transcript
- 0:12
My name is Laurie. I'm Head of Developer Relations at Arize AI. Uh, in a former life, I co-founded npm Inc., so some of may, some of you may remember me from JavaScript days. Uh, but these days, I think about AI agents and, uh, in this case, what they are doing to our jobs. Uh, so a little bit over three years ago, um, uh, swyx wrote an essay called The Rise of the AI Engineer. Uh, and his argument was that when it comes to shipping AI products, you want, uh, engineers, not researchers. I think that
- 0:42
is a message that resonated with everybody, especially the people who come to this conference. Uh, and so he created the job title of the AI engineer, and companies since then have been hiring for it. Um, and part of what I'm here to do, today to do is to convince you, uh, that he did not in fact go far enough. Uh, there's not just an AI engineer revolution happening. There is a revolution happening to every kind of engineer. Um, all engineering jobs are being replaced
- 1:12
by a new type of job, and not everybody is going to be happy to get that news. Um, the underlying change that kicked all of this off is that agents have made writing code basically free. Like you pay for the tokens, but relative to the cost of the time it used to take you, uh, it is basically free. Um, everything that comes after writing the code did not become free at the same time. So the, the bottleneck moved from writing code is really slow and all of this other stuff happens at roughly the same speed
- 1:42
to writing code is super fast, uh, and now we get stuck behind new things. So the first... The thing it's stuck behind right this second, like this month, is code review. Um, uh, but my thesis is that it's not going to stop there. We are going to figure out how to automate code review. We are going to make, uh, code review move at the speed of agents, and then we are going to run into another bottleneck and another bottleneck and another bottleneck, and at some point, we're going to stop. We're going to hit one that we can't get
- 2:12
past. Uh, and that is what is going to be left for the humans to do. Whatever the bottleneck is that we can't get agents past. So I'm making a bet that I can think of a thing that agents reliably are not going to be able to do. Uh, and then once we've come to that bottleneck, we're going to have to figure out how do we train the next generation of engineers to be able to do that job, because we're not training them to do that job right now. So a quick warning/apology. I have 18
- 2:41
minutes because that is how long swyx gives for talks. Uh, and I have a really big pile of data. I did a ton of research for this, uh, and I don't have time to give it all to you. I don't have time to, like, quote my sources and do all of that stuff. So I'm just gonna throw numbers at you with a bunch of citations on the slides, and I'm just going to have to sort of ask you to trust me, bro.
- 3:03
Uh, uh, so developers who switch on autonomous coding agents write seven hundred and forty-one percent more code, eight times as much, uh, and but they ship only thirty percent faster. Like thirty percent faster is not nothing. Thirty percent faster is amazing. We are shipping a lot more code than we used to. But given that we're writing eight times more code, only a third more actual shipping is kind of bad, right? It means that we're stuck somewhere. Uh, and where we're stuck is code review. Um, and the reason that we're stuck is because
- 3:33
humans can't just review faster. Uh, there is an enormous study done by Cisco twenty years ago about how fast you can do code review and how accurately you can do code review. Um, and it turns out that humans' ability to, like, parse what's going on in code falls off radically after about four hundred and fifty lines of code per hour. So if you've produced a ten thousand line diff with your agent, it would take you like a week to
- 4:03
reasonably review that code. And the answer is that nobody's actually doing that, right? 'Cause we're all throwing ten thousand diffs around willy-nilly these days. Um, so I did a whole talk about this phenomenon at a previous AIE, and that talk is called The Death of the Code Review. And if you want all of the information about the death of the code review, you can watch that talk. Um, but the problem facing us right now, uh, which is not the same as the problem I'm trying to t-solve with this, overall with this talk, is that the cost of producing plausible
- 4:32
code has collapsed while the cost of knowing whether to trust that code has stayed exactly the same. Um, so the obvious review... move right now is to make the machines do the review, like I said, and that is already happening. Um, GitHub's Copilot, uh, does more than one in five of the code reviews on GitHub as of last month. Uh, you can, if you want to, opt in to let, uh, Copilot be the final reviewer before something goes to, uh, production. You can make it have the approval
- 5:02
power if you want to, which is crazy. I don't know who's turning on that option, but you can do it if you want to. Um, OpenAI says that at, at OpenAI, humans may review their pull requests but aren't required to. Um, but the problem is that the machine reviewers aren't very good yet. Uh, the data says that there's, uh, the data that I found put four AI reviewers through their paces, and they all found a bunch of bugs, but none of the same bugs. They did not agree on any
- 5:31
individual bug. Uh, they all found actual issues in the code. They all found different issues in the code. Um, so they're worth running, but if you are serious about this, you're never gonna run just one of them, uh, because they are not reliable enough yet. Um, and there are some people who, uh, went all the way. They were like, "I'm just not gonna review the code anymore. I'm just gonna assume my agent has one-shotted this perfectly." Um, Dexter Horthy, who is here, I don't know if he's in the audience, probably not,
- 6:01
um, uh, he spent six months telling people not to read the code, and then after that six months, he got on stage and said, "Nope, I was wrong. You can't actually, uh, skip code review entirely. We tried that. It was a disaster. We had to rip and replace whole systems." Uh, that's not an option. The agents are not good enough. They're not one-shotting the code. Um, Amazon had a run of outages, uh, that they did post-mortems on and decided that it was because their coding agent had produced a bunch of slop and they'd just released it
- 6:31
into the wild without thinking. Uh, and so now they don't let them do that anymore. Uh, senior engineers must sign off on code reviews at Amazon. Um, and Anthropic published a post-mortem, uh, on bugs that had got past every reviewer that they had, both human and machine. So the problem is that the automations that we've built so far to check whether our code is good enough are not themselves good enough. Um, Claude Fable 5 passes the tests, uh, on 80% of a standard coding benchmark. Um, and cog-
- 7:01
so Cognition, the makers of Devin, um, they built a benchmark that asks a different question, which is not, does it pass the tests? The question is, would you, as the reviewer, as the maintainer of this repo, merge this PR? And if you ask that question instead, the pass rate falls from 87%, uh, to 54%, um, because there's a lot of extra information that goes into deciding whether you would merge it. There's code style, there's side effects, there's
- 7:31
external systems, there's history that is not encoded into the code base. It is encoded into the brain of the maintainer. Um, so what the maintainer is adding, what the maintainer is demonstrably adding, is something that we haven't given the agents yet. There is information they don't have. There is a skill they don't have. Um, and it is the sense of what good means. That is what we are... That is what the agents don't have yet. Uh, and that is a critical insight that I'm gonna come back to.
- 8:01
Um, Eliyahu Goldratt, if I'm pronouncing that correctly, was a physicist who became a management guru, and in 1984, he wrote a, a novel about running a factory. Does that sound like an interesting novel to you? Um, it is called The Goal. Um, and his big idea was the theory of constraints. Uh, every system has one bottleneck, and speeding things up does not work unless the thing that you are speeding up is the bottleneck. Uh, that is our problem with code review. Uh, we've sped up the code, and we have
- 8:31
not sped up the review. Um, his second idea is the one that matters today, which is that when you fix the bottleneck, it doesn't go away. It just moves and finds another bottleneck to pile up against. Um, so let's assume, despite everything that I've said about how bad the code reviewers are right now, that we figure that out, that we get automated code review to a higher standard, and we move that bottleneck. What is the next bottleneck, uh, that it is going to run into? Um, if we figure out review,
- 9:01
the next bottleneck is probably gonna be operations. How do I deploy this thing? How do I scale it up? Uh, and then you're probably gonna run into security. How do I make this thing safe at scale? Uh, how do I make sure that this thing is private? How do I make sure that it's not radically going off the rails? Uh, but I'm gonna make... And this is the big leap that I'm making in this talk. I'm going to assume we figure those things out as well, because those things are very heavily solved
- 9:31
by humans. There's an enormous amount of data out there about them. If we can figure out how to do operations, how to do scaling, how to do security using progressively more advanced agents, what is it that we are left to do? Like, what is the role of a human in that situation? Um, it's also... It's a big question, and it's also a long-term question. I'm not telling you what is gonna happen in the next 12 months. In the next 12 months, I think we'll probably make some real progress on code reviews. I do not think 12
- 10:01
months from now we will have sorted out operations and scaling and security. I think that is going to take somewhere between five and 10 years to get done, uh, despite people who are like, "AGI 2027." No, we're gonna slowly iterate on this stuff. Um,
- 10:19
but the other assumption that you need to make for my... for my thesis to be correct is that the amount of software that we need is effectively infinite, and I feel that one is a much easier case to make. Think of every website you've ever run into, uh, in the world in production, and it's probably pretty bad. Uh, and the reason it's bad is not because we don't know how to make really great websites. The problem is that there's not enough people to make all of the websites good. Every time you run into a terrible website at your hairdresser or the government or, like, you know, your
- 10:49
bank, like, it's not because there's not somebody who knows how to make that website good. It's because, A, it's very expensive to make it good, and B, the people who are doing that are busy doing other things, uh, you know, probably shipping AI agents. Um, so assume that we can make as much software as we want basically for free and that the appetite for making software is essentially infinite. That is a wonderful situation in which to be in. We've got infinite demand and also suddenly infinite supply. Uh,
- 11:20
where does the bottleneck end up? What do we think the agents are not going to eat? Um, to try and figure that out, I split up the job of software development into as many pieces as I could think of, and I sorted them by how soon the agents, uh, get to them. Uh, writing code is already collapsed. Reviewing and maintaining code is going soon. Um, shipping to production and scaling are next on the chopping block, although there isn't a lot of data there yet. Uh, and then there's the stuff that I think is probably safe, which is deciding what to build, uh, defining
- 11:50
what good means and finally making it delightful. What's left is finding out what people actually want. Every piece of software is a formalization of something that a human wanted to get done. Uh, so if a customer tells you, "I need to con- to keep track of my orders," there are 10,000 pieces of software that- Keep track of your orders. That's what they do. Uh, but only one of those, if you're lucky, is good if the thing that you're doing
- 12:20
is you're running a bakery. If you're running, like, an auto parts store, that's going to be different. If you are running, uh, you know, a, a giant manufacturing concern, it's going to be completely different. The... That requirement is not sufficient. The whole job is figuring out what exactly do you mean when you say that you want to keep track of your orders. What does that mean for your business? What does it have to do? How does that work? Uh, and that knowledge isn't in the training data. It isn't in the weights. It is in the mind of the person who runs that
- 12:49
business, and the only way to get it out is by flapping your meat parts at them and having them flap their meat parts back. Like, you have to talk to them. That is the only way to get the information out. Um, and that work has a very important property, which is that it doesn't transfer. Uh, if you think about what good means for, like, a calendar app and what good means for a scheduling app, those are solving almost the same problem, right? Like, a calendar app and a scheduling app, but they're completely different pieces of software. They have totally different requirements and a totally
- 13:19
different definition of good even though those two use cases are so close together, and we have a zillion use cases. Um, any time that you've used software that has a zillion configuration options and it still doesn't do what you need... I'm looking at you, Jira. Uh, like, that is the problem you're running into of software that is trying to do everything and still failing at it. Um, so the bottleneck moves to describing the problem and defining what it means to
- 13:49
have solved the problem, and my big bet for this talk and for the next 10 years is that is where it stays. Uh, Fred Brooks, who ran IBM System/360 project and wrote, uh, The Mythical Man-Month, uh, he wrote about this in 1986. He split the difficulty of software into two kinds. Accidental complexity is the difficulty of the tools or the languages, the typing, the syntax, and then there's the essential complexity, which is the pro- difficulty of the problem itself. Uh, and he predicted that no single tool would make
- 14:19
us 10 times more productive because the essential part is the hard part. Uh, and Brooks' prediction is looking pretty solid because we made code eight times faster, and we still didn't get very much faster at shipping because the hard part is the essential part. Um, Sean Grove from OpenAI made the same point, uh, two years ago. Uh... No, sorry, a year ago. Uh, what year is it? Um, he said that code is a lossy projection of the specification, and whoever writes the spec, if it's a PM or a lawmaker or an engineer or a
- 14:49
marketer, they become the programmer. Uh, and so writing down what good means is where the value is now. Uh, I'm beginning to run out of time because I, uh, haven't timed this well. Um,
- 15:08
anything that you can verify cheaply, you can train against until you beat it if you're a model maker. Uh, so, uh, Cognition tried to write down, because of their benchmark, about what, what it counts to merge something. Does it... You know, is, is this code mergeable? They were like, "Let's see if we can figure out a benchmark that defines what mergeable really means. Let's see if we can get all of that other information into the benchmark, and then we can train a model on it." Um, uh, Cursor's reviewer Bugbot
- 15:38
turns every thumbs down that you, that your team gives it into a rule, and they have, like, 44,000 rules now. Uh, so the moment the maintainer takes knowledge out of their brain and puts it somewhere written down that a model can train on it, uh, the bottleneck moves again. That is why I think security can go. That is why I think operations can go. I think we can write those things down eventually even though we haven't done it yet. Uh, Andrew Ng, the co-founder of Google Brain and Corsora-
- 16:08
Coursera, says that one of his teams proposed having twice as many product managers as engineers. Uh, and he said that that was a first in his life, uh, but his answer since then hasn't been hire more PMs. It's to blur the jobs together. H- in his words, "When an engineer understands users and can make decisions on what to build and build it directly, uh, they can execute incredibly quickly," and that is where I think we are beginning to hit on the answer. Uh, in the 1960s, there's w- there was a job called the systems analyst.
- 16:38
Their job was not to write code. Their job was to talk to the business and find out what the problem was. Uh, then that analyst turned into something that we call a product manager today, and we've had two professions, uh, where there used to be one and a half. I think what is happening is they are collapsing back into one. We are going back to the '60s. So, that is not an AI engineer. They're not just building AI. They're not just building with AI. They are building everything. They are building all software.
- 17:08
Uh, and I've decided to call this job the product engineer. This is somebody who finds out what people need, defines what good looks like, and ships it, uh, with agents doing the typing. Um, Gergely Orosz, whose name I have probably butchered, he writes The Pragmatic Engineer newsletter. He described the product-minded engineer back in 2019 as a nice trait for an engineer to have, and what I'm saying now is it has gone from nice to essential. It has gone from the, uh... from
- 17:38
a, a skill that senior engineers used to have to a skill that every engineer has to have if they are going to be an engineer. Um, and you can see the market already paying for this. Forward deployed engineers, who look very much like product engineers, uh, job postings for them rose 800% in nine months of 2025. They are everywhere. Everybody is trying to hire them. Who in this room is trying to hire a forward deployed engineer? Show of hands.
- 18:08
Okay. Well, more... I was expecting more hands. Oh, well.
- 18:13
So, uh, the average comp for a forward deployed, forward deployed engineer, by the way, is $240,000, uh, which is to say that it pays like a senior engineer because it is a senior engineer. It's a very useful kind of engineer. Um, but nobody agrees on what to call it yet. There are like a thousand different job titles for this thing that I am describing and have decided arbitrarily to call the product engineer. Um, and probably at this point, some of you are thinking, "That isn't programming," and you are correct.
- 18:43
It is not programming. The programming is being done by an agent. It is, however, recognizably still software development, and we are going to have to wrap our minds around the idea that being a software developer and being a programmer have become divorced, uh, in 2026. Um, and when I say stuff like, "Well, I think the hard part is deciding what good is," I realize that I'm falling into a trap that lots of people in AI land fall into, which is all of the
- 19:13
parts, everything about everything can be automated by AI except the thing that I'm good at, because that's too hard. I heard a VC say that the other day and I was like, "Buddy." Uh-
- 19:25
Like what?
- 19:26
Like, no, no. Someone is gonna automate your job any day now. Um, uh, so you have to take this with a grain of salt. Am I, am I just thinking that the thing that I enjoy about software development, which is figuring out what the problem is, is the hard part of software? You can tell me if I'm wrong. Um, but say you believe me so far. Programming goes to zero, product engineering goes to infinity. That brings us to a new and important problem, which is junior developers,
- 19:56
because the way we used to train a junior developer is we used to give them very well-specified tickets that said, "Write this software this way in particular and then give it to me so that I can tell you all the ways that you were wrong."
- 20:11
Uh, and that is gone because that's what we do with the, that's what we do with the coding agents now. We say, "Write this thing and then I'll tell you how you're wrong," and then we have a conversation with them. Uh, and there's... That is not a job that we need a human to do anymore. But we still need an enormous number of forward deployed engineers or, uh, product engineers, and we have to get them from somewhere. We can't just abandon the current generation of new developers who've showed up and said, "How do I do this stuff?" And we're like, "I don't know,
- 20:41
because I don't know what your job is anymore." Uh, this is not a theoretical future. This is an actual graph of labor statistics since January 2023. It is sorted by age. The red line is engineers aged 22 to 25. Their employment is collapsing. We are not hiring junior engineers. This is not a thing that I think is gonna happen. This is a thing that's been happening for two years. Job market sucks for them. Meanwhile, the job for en- older engineers, 41 to 49, has gone way up because
- 21:11
senior engineers are still super valuable, more valuable than before. But junior engineers are being screwed, and we need to fix that as a profession. Uh, so if we're going to reap the benefits of a world where we 10Xed code output and figured out the rest of the software supply chain and figured out what humans need to do, uh, then we are gonna need a lot of people to do that. I said that the demand for this software is going to infinity. We need infinity people. Where are we gonna get them? Uh, we've been creating them accidentally so far, and we need to figure
- 21:41
out a way to do them on purpose. Google has a product... Uh, has a program called the Associate Product Manager program, which churns out people that look very much like product engineers. They churn out 50 of them per year. We need 10,000 of them per year. Even if you look at all of the other pro- all of the other companies that have something, a program like Google's APM, it's been extensively copied, they're only churning out about 1,000 people a year, and we need 10 times that number at minimum, uh, to keep up with demand.
- 22:11
Uh, universities teach data structures. Boot camps teach React. Nobody teaches go sit with a baker for a week and come back with a spec, and that is the person that we need. Um, so if you lead an engineering team, you're going to have to grow them on purpose. Uh, IBM is already redesigning its entry-level role around talking to customers and writing specs instead of programming. Um, you're gonna have to figure out how to do that for yourself. You're going to have to figure out how do you train product people. Uh, and you're going to have to do it very quickly
- 22:41
because the time we should have started was two years ago and there's a whole cohort of very sad, unemployed juniors right now who have this problem, and we are not hiring them, and we need to do something about that or we are going to be screwed, and not just them. Uh,
- 22:57
so if all of that happens, there's one big thing that I don't want to just gloss over or hand wave away, which is a lot of people got into programming because they love the craft. They love a clean abstraction, the feeling of a hard bug finally giving up, uh, making a machine do exactly what you told it to do. They did not sign up to interview bakers about what they want in their lives. Like, that is not the job that they signed up for. Um, and they are gonna hate this. They already hate this. They are
- 23:27
complaining about it very loudly. I have no consolation prize for them. Um, but, uh, I do have an observation, which is that for a lot of people who think they love the typing, maybe what you loved was the moment just before you started typing when you figured out the problem. That, that part doesn't go away. That part is still there. That creative spark when you're like, "I know how to make this into a repeatable system that glows in my brain," that's not going away. Uh,
- 23:57
and the craft of programming will survive, uh, in the same way that, like, woodworking survived the furniture factory. Like, programmers program the way that singers sing, right? They s- do it in an empty room with nobody watching just because they love it, and that won't go away. You can still do that. Um, it just won't be the thing that people pay you hundreds of thousands of dollars every year to do. Um, so here's my prediction. The next 10 years are gonna be ugly. The last two years have been ugly. Um, the load of this
- 24:27
stuff is arriving before the tools are ready. The junior le- ladder is gone before we have figured out how to train more juniors. Uh, and a lot of people who love the craft are going to have to decide if they love the job that is replacing it. Um, but this has happened before. Compilers, and then frameworks, and then open source each made a generation's worth of typing before them, uh, unnecessary. And each time, we came out of it with more software developers. We just changed what software development meant. Uh, and I think that's going to happen again.
- 24:58
Uh, in 10 years' time, we will get a profession that is bigger than today's, pays about as well, and is mostly shaped like product engineering. That is the case that I'm making for you today. But that is a whole lot of high-level predictions, and I hate those. What does this mean for real? What should you actually do with this today? Uh, first, be prepared for a world in which humans are barely involved in the production of code. So write down your high blast radius code and make that be the only thing that you are reviewing anymore, because the rest of it, your automatic
- 25:28
reviewers are going to get done. Uh, then delegate that's... everything that's cheap to verify. Spend a lot of time thinking about, "How would I build an automatic way of telling whether this thing, whatever that thing is, is good or not?" And then feed that to your agent instead of trying to figure out the review process. Uh, and finally, put your best people's judgment into specs and rubrics and evals. You knew I, the Arize guy, was gonna say evals at some point. Uh, get stuff that exists only in people's heads into
- 25:58
code. That is your job as a produ- product engineer. It is turning the sense of the problem into a spec of the problem so that the agents can build it for us. Uh, and measure the right thing. 741% more code was never the point. Measure how much stuff you are actually shipping. Uh, measure how often your reviewers' comments get acted on. Measure the gap between the tests passed and a maintainer would merge this. Uh, and remember that the last reviewer is production. Watching what it does there with another model
- 26:28
grading it, that is an eval. I'm not going to pitch Arize. That's just what we do, and that's why we're so popular right now. Uh, and I'll leave it at that. Um, three years ago, swyx wrote about the rise of the AI engineer, which is engineers learning to build with models, and the rise of the product engineer is about what's left when the models do the building. Uh, finding out what people need, writing down what good means, and making it something people like, that is the job now. It is going to be the job for everyone in this room, whether we like it or not.
- 26:58
We are all product engineers now. Thank you so much for your time and attention.