AI Engineer World's Fair 2026
Agent Output Is Not UX: Rendering Layer Your LLM Pipeline Is Missing - Bala Ramdoss, Amazon Lens
Read the talk
Agent Output Is Not UX: Building the Rendering Layer
A capable agent can find the right information and still leave the user doing the work. Typed UI contracts, streaming states and a backend for frontend turn that output into an usable interface.
From a talk by Bala Ramdoss
Before you start: Familiarity with API responses, frontend components and basic LLM application architecture will help.
The reservation is still unfinished
Ask an AI assistant to reserve a restaurant table, and it may return an accurate phone number, opening hours and even details about the walk-in oyster bar. The information is useful, but the reservation remains unfinished. The user still has to turn the answer into a booking.
The opening demonstration replaces that answer with a reservation card: choose a date, party size and available time, then use the booking button. The fictional Zuni Cafe mockup illustrates the difference between supplying information and providing the controls to act on it. It does not establish that a reservation was actually completed. The missing layer sits between the model’s output and the interaction a person needs. An agent may already have the tools; the product still needs to make those capabilities usable.
Bala Ramdoss approaches this problem from more than a decade building customer-facing applications, including six years working on Amazon Lens. Lens supports shopping with images, screenshots and barcodes to discover visually similar products. His experience with applications reaching millions of mobile devices informs the architecture here; he presents these views personally, not on behalf of his employer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Describe the interface as data
Interfaces such as ChatGPT make the design questions visible: should an answer become cards, a carousel or a list? But presentation decisions depend on earlier architectural choices. Will the experience respond promptly? Will information arrive together or incrementally? Can the same output work across mobile app versions and device capabilities? These are delivery problems between model output and the screen, and a renderer alone cannot resolve them after the response has already been generated.
Ramdoss previously solved this layer separately for each system. Generative UI gives those bespoke implementations a shared vocabulary, and Google’s A2UI provides an open specification for one approach: the agent describes components as data instead of returning raw text or HTML, and the client renders that description with its own native widgets. The model can express what interface it intends without supplying the implementation of every control.
A conventional API returns data and leaves presentation to the client. Generative UI gives the model some responsibility for that presentation. Ramdoss uses CopilotKit’s spectrum to distinguish three degrees of freedom:
| Approach | Model’s responsibility | Example |
|---|---|---|
| Controlled | Select a prebuilt component | Product card |
| Declarative | Compose components from a catalog | Date field, time field, submit button |
| Open-ended | Produce a novel interface | Generated UI |
A2UI occupies the declarative middle. Ramdoss associates the open-ended end with MCP Apps, although that protocol supports server-supplied, sandboxed HTML interfaces without requiring a model to generate them at runtime. As freedom increases, so does the client’s trust burden. His focus is the controlled and declarative approaches, which he considers the practical territory for most production mobile apps.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Design for clients you cannot promptly patch
On the web, a renderer fix can reach users quickly. Mobile developers do not control when installed applications update. Ramdoss frames the challenge at the scale of hundreds of millions of installs: an unprepared client encountering an unknown content type can crash, and that failure can persist for days or weeks among users who have not updated. This is a risk to design against, not an inevitable property of mobile rendering. The operational constraint is that you cannot rely on promptly patching every client.
That constraint shapes the whole pipeline:
- Build version-aware context describing the client’s capabilities.
- Let the model produce typed UI intent within those capabilities.
- Pass the intent through a backend for frontend, or BFF.
- Have the client renderer draw the result and fall back safely when it cannot.
The architecture divides into three connected patterns: the rendering contract defines what can be selected, streaming governs how output arrives, and the BFF sits between model output and the client.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Constrain component selection before generation
The rendering contract makes the model aware of the client it is addressing. The client should not have to examine an untyped stream of tokens and infer which interface the model meant to show. Maintain a repository mapping application versions to capabilities, then use that map when constructing model context. If a flight card first ships in app version 2.0, expose it to the model only for clients running 2.0 or later. This is an illustrative application version, not an A2UI protocol version. The model chooses the customer experience from the options that context permits.
The model-facing half of the contract separates conversational blocks—what the agent says—from UI blocks—what the client should render. It also encodes layout rules. In the flight example, one to three flights belong in a swipeable carousel; four or more belong in a vertical list. The following JSON expresses those example rules as application-level contract data:
json
{
"blocks": ["conversation", "ui"],
"components": [
{
"name": "flightCard",
"minimumAppVersion": "2.0"
}
],
"flightLayouts": [
{
"minimumItems": 1,
"maximumItems": 3,
"layout": "carousel"
},
{
"minimumItems": 4,
"layout": "verticalList"
}
]
}
The version map determines whether flightCard enters the model’s available catalog; the layout rules tell the model how to arrange the flight results.
The model never invents a component in this pattern. It selects from a fixed menu supplied by the application, ideally a handful of relevant choices. That restriction is straightforward with a small catalog, but becomes harder as features expand across more surfaces. The continuing context-engineering problem is deciding which supported components and rules to expose so that selection remains reliable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Stream useful component states
Traditional applications often make an API call and wait for a complete response. With an LLM, generation latency is compounded by safety checks and the extra work needed to produce dynamic UI. Streaming lets the client render chunks instead of withholding the entire interface until all of that work finishes.
The flight-card demonstration progresses through three states: a skeleton establishes the component’s shape, partial content fills it in, and the complete card supplies the remaining information. The slide labels the delivery mechanism as chunks over SSE. In the illustrative progression, completing the card may take three to four seconds. That duration is an example of a tolerable progressive wait, not a measured performance guarantee.
This changes what the application should optimize. Time to the first useful chunk captures when the user first receives something meaningful, while total latency describes when everything is finished. Ramdoss advocates prioritizing the former rather than judging the experience only by the latter. A spinner communicates that work is pending, but supplies neither useful content nor an explanation of progress.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the wait a useful interaction
Sometimes the product can make the wait interactive. Amazon Lens Live lets users focus on different things and tap an object of interest while results arrive. The user continues directing the task instead of staring at a loading indicator. This depends on the product: not every feature has an activity that remains useful while processing continues.
When the feature does not control the full screen, task-status feedback can serve a narrower purpose. Ramdoss recommends using thinking indicators sparingly: users increasingly expect to understand what is happening, not merely be told that the system is busy. His Gemini example shows glimpses of the agent’s activity as it works. Ramdoss describes personally accepting a ten-second wait when that progress is visible. This is his tolerance in the example, not a universal usability threshold; the useful mechanism is streaming states that help the user understand the work and assess the eventual output.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn UI intent into actionable native components
The contract determines which UI gets selected; streaming delivers the chunks. The BFF makes those chunks meaningful to the application. Ramdoss places this responsibility within server-driven UI, where increasing server control also increases the implementation the server must own. Here, the BFF centralizes platform-specific rules, including differences between Android and iOS, so each client can make fewer presentation decisions and draw what it receives. Layout ownership is a responsibility of this proposed BFF, not a requirement of every backend-for-frontend architecture.
Formatting is only part of that work. The BFF also hydrates the output with the information needed by the rendered components and adds actions. Each rendered element carries an action payload describing what a tap does, which deep link it opens and potentially which impression metric to log. It also carries conversational context across turns so that the next response knows what came before. The output becomes part of an ongoing interaction rather than an isolated arrangement of cards.
Existing applications can adopt this layer by reusing components already in production: flight rows, product cards and other familiar units. The agent chooses among those units while the application retains its brand, information density and native feel. There is no need to introduce a separate visual identity simply because an agent now helps select the interface.
The responsibilities reinforce one another: a typed, versioned contract gives the model supported choices; streamed components make useful output and agent activity visible; and the BFF absorbs the adaptation work that would otherwise complicate every client. The restaurant example returns the architecture to its purpose. Finding the right information is only part of reserving a table. The delivery layer turns that information into an experience through which the user can finish the task.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Declarative UI descriptions, component catalogs, native rendering and streamed interface updates, with links to implementation guides.
An interactive explanation of controlled components, declarative composition and open-ended interfaces.
How MCP tools supply interactive HTML resources, communicate with hosts and operate within sandbox restrictions.
Amazon's overview of live camera shopping, product matching and tap-to-focus interaction in the Shopping app.
Further reading
- Backends For FrontendsArticle
Sam Newman's architectural explanation of backends tailored to individual frontend experiences and their downstream services.
Examples of local actions, agent events, form state and validation in interactive component interfaces.
The abstract and session details for Ramdoss's November 2025 QCon talk on mobile AI latency, interactivity and architecture.
Read the complete timestamped transcript
- 0:01
I asked my AI assistant to help me reserve a table at a popular restaurant. Here's what it gave me.
- 0:09
Now, it's not wrong. The phone number is there. The hours are right. It even knows the walk-in oyster bar at the front. The model did the real work, but look at the outcome.
- 0:24
I have to do my research and work towards actually booking that table. Like, we have been here for a while. Now, imagine if you were to build something like this for your customers.
- 0:35
What would you want it to do? Same answer rendered like this instead. A date, a time, a couple of taps, and then you're done. The agentic product you're building already has all the tools to support your customers' needs.
- 0:53
The only thing that you need to focus on is the layer between the model and something that a human can interact with.
- 1:00
That layer is what I'm gonna talk about. The models and agents are here to stay, and we should learn how to make it friendly to the humans.
- 1:09
Hello, welcome to my talk. I'm Bala Ramdoss. I have been building customer-facing apps for over a decade, and the past six years I have spent, uh, building Amazon Lens.
- 1:22
Amazon Lens is our suite of camera features powered by AI. It enables you to shop using images, screenshots, and barcodes to discover visually similar products. If you have Amazon app installed, I encourage you to try.
- 1:37
This talk comes from my experience building, um, customer-facing products that are in millions and millions of mobile devices. Uh, quick dis-disclaimer before I go further, I am giving this talk on my own.
- 1:50
The opinions in this talk are mine and not my employer's.
- 1:56
When you think of building UX for an agentic AI, obviously ChatGPT comes into mind. How do you draw these cards? How do you choose carousel versus list? How do you craft this so the human can understand and interact better?
- 2:12
When you attempt to build something like this, you run into a wall. There are problems that need to be solved from the very early stages of your system. Is that experience going to be snappy or slow?
- 2:26
Do we get all the information at first or one thing at a time?
- 2:32
And how do you scale this for mobile apps where versions and device capabilities are fragmented? Like if you notice, none of these problems are due to the model itself.
- 2:42
The model does its job well. Um, these are delivery problems, and they live in between the model output and what's on the screen. That is the layer that decides whether your product succeeds or not.
- 2:56
So, uh, for this delivery problem, if, if you had asked me about, uh, about this layer two years ago, I wouldn't have had a name for it. I'd built a couple of AI features by then, and every single time, I solved this part from scratch.
- 3:12
A bespoke pattern shaped around whatever system I was in. Uh, there was no shared vocabulary for any of it.
- 3:22
What gets me is that this thing now has a name, Generative UI, and there's an open spec for it, A2UI from Google. Instead of the agent handing you a raw text or HTML, it describes the UI as data, a list of components, and the client renders them with its own native widgets.
- 3:43
A problem I used to solve alone is becoming something teams get to start from, and that's exactly what I'm excited to dig into.
- 3:52
Um, normal API returns data, uh, and the client decides how to draw it. Like, your model is good at tool use, uh, coding, and even other complex tasks. It can do UI too, and that is what the Generative UI is about.
- 4:08
And it's, it's a spectrum. Uh, CopilotKit, a company that does this primarily, lays out, um, uh, in three rungs. At the bottom, uh, we have controlled. The model picks a pre-built component, like a product card.
- 4:25
It never invents anything. In the middle, um, declarative. The model composes UI, uh, from a catalog, uh, date field, a time field, or a submit button. Um, that's where A2UI sits.
- 4:41
And at the top, it's fully open-ended. The model generates a novel UI on the fly, like, uh, MCP apps. The higher you go, the more, uh, your client has to trust whatever the model hands it.
- 4:56
Uh, most production mobile apps live in the bottom two rungs because, you know, that's where you stay safe, and that's what we'll focus on.
- 5:06
Um, adding to the complexity of UX building, m-mobile apps play an important part. Um, on the web, if a renderer breaks, you know, you ship a fix, and it's live in minutes.
- 5:18
Uh, mobile apps cannot do that. Like, you're looking at hundreds of millions of installs, and you don't control, uh, when any of them update.
- 5:28
So when a client meets a content type it's never seen, it doesn't gracefully degrade. It crashes, and it keeps crashing for days or weeks, um, in the hands of people who haven't updated.
- 5:41
So one rule holds everything that follows for mobile clients. You cannot meaningfully patch the client.
- 5:48
Um, how does this system look when you zoom out?
- 5:53
Um, here's a whole simplified pipeline. Uh, version-aware context phase that feeds the model. Uh, yes, that is context engineering. The model outputs typed UI intent, uh, that flows through a BFF, Backend for frontend, and to a client renderer that draws it and, uh, falls back safely when it cannot.
- 6:17
We'll break it into three patterns. Pattern one, the rendering contract. Pattern two, streaming. Uh, that's the flow in between them. Pattern three, uh, the BFF that sits in the middle.
- 6:30
Uh, we'll take them one at a time, starting with the contract.
- 6:35
Uh, the contract is all about making the model be aware of what the client capabilities are. You ensure it, it stays true to the client that the model is trying to draw the UI for.
- 6:48
Um, what it ensures is that the onus is not on the client or the rendering layer to infer what to show by looking at the token output.
- 6:59
Um, you also maintain a repository of version map with the capabilities. For example, you introduced a new flight, uh, card UI in version two dot oh. Uh, make sure to surface it to the model only from version two dot oh onwards, um, when you build the context.
- 7:20
The takeaway here is that the model is going to choose the CX, and you provide the right information in its con-context.
- 7:29
Um, let's take a look at this example. And this, this is usually the hard part of building the system.
- 7:36
This is the model-facing half of the contract.
- 7:40
Uh, y-you don't want the model to send back text. Like, you want it to stream, uh, blocks of UI components. Um, a conversation block for what it says in text and a UI block for what, what it wants the client to render.
- 7:58
The contract even encodes layout rules. In this example, o-one to three flights, a swipeable carousel. Four or more, a vertical list. The model picks the intent. And notice what the model never does.
- 8:16
It never invents a component. It chooses from a fixed menu that you provide to it. Obviously, it won't be in the millions. It would be a handful. It turns out that it gets significantly harder as you scale your features to more surfaces.
- 8:34
I'll let you think about how to engineer that context to make sure the model succeeds in, uh, picking one.
- 8:42
So, uh, that's the contract. Now, let's move on to the streaming part. This was eye-opening to me when I first encountered it. Traditionally, apps make an API call and wait for something to happen.
- 8:58
When LLMs are involved, um, this pattern becomes ineffective, as often the latency is higher due to the model themselves. And on top of it, there are tons of additional checks to ensure safety and whatnot.
- 9:12
And we also introduce a new layer that it's gonna be a little bit complex to do the dynamic UI.
- 9:20
Um, streaming helps here, uh, by not making the client wait for the whole response. It renders things in chunks at a time.
- 9:31
Take a look at the illustrative example when you have to surface information one chunk at a time. First, you show a skeleton, then partially filled it, then complete it.
- 9:43
It may take three to four seconds, uh, to get there, but the wait is bearable.
- 9:50
That kind of changes what you measure for an app, right? You stop chasing the total latency, which we have done for over a decade.
- 10:00
You start chasing time to first chunk, you know, the first useful thing that your user sees.
- 10:07
For this reason, the traditional loading spinner won't work for AI features.
- 10:13
Sometimes you have to get creative and design your feature around it. This may not work for all, but here's a product example. When you know it's going to take time to render something, keep the user engaged.
- 10:26
The Lens Live allows the user to focus different things, um, tap on an object that they are interested in while they wait for the results.
- 10:36
Uh, if you don't have the full screen in your control, you can show the thinking CX, but please use it sparingly. The overall AI users have, uh, moved out of the forgiving phase, and now they expect to know what is happening.
- 10:53
Now that you have a way to stream back different states, you know how to show what your agent is doing. Um, here's an example I copied from Gemini. When, when it starts to work on a task, it gives a glimpse into what the agent is doing.
- 11:08
Even though it takes 10 seconds, I'm okay if I know what's happening and, uh, be able to trust the agent's final output.
- 11:18
We have covered streaming. Let's move on to the final and most important part, your new BFF. Pattern one was about how UI gets picked. Pattern two was about how the chunks are delivered.
- 11:33
Pattern three is all about how the chunks become meaningful UI elements.
- 11:39
This is a concept of server-driven UI, and it's a spectrum of server control, and the more the server controls, the more you build. The BFF is a subset of it and decides how to render.
- 11:53
It owns the platform-specific rules like Android versus iOS, that kind of stuff. Uh, it also helps the client to make less decisions and draw what's handed to it. That's the whole idea.
- 12:08
The BF- BFF doesn't just ship layout. It does a little more. In addition to the format and context, it does the hydration and adds actions.
- 12:23
Every rendered element carries an action payload to handle what, what a tap does, what the deep link it opens. It, it could even name the impression metric to log.
- 12:34
It carries this conversational context across turns, so the next response know what came before.
- 12:42
And the good thing about this is that you can actually do this to your existing apps. You can reuse the existing CX units you already have. Uh, the flight row, the product card, the components your app already shipped in production.
- 12:58
You're not building a new agentic look into your app, and that's a good way to build for humans. Same brand, same density, same familiar feel. It looks and feels native.
- 13:13
And that brings to our takeaways. One, the models are highly capable already. You feed a properly typed, versioned contract so it can pick and choose the right CX.
- 13:29
Two, stream into typed components, not text. Let the user know what your agent is doing.
- 13:38
Three, let the BFF absorb the model output so the client can stay dumb and safe.
- 13:47
No-- none of these are about the model. The model was fine.
- 13:52
This layer is what ships the product. Bringing back to where we started, your agent output is not the CX. You build on it. Thanks for watching. If you'd like to learn more or connect with me, here's a QR code.