# Feedback Loops are All You Need

Mehedi Hassan, Granola

AI Engineer Europe 2026

[Talk page](https://ai.engineer/talks/ON5LIT0M4do-feedback-loops-are-all-you-need) · [Video](https://www.youtube.com/watch?v=ON5LIT0M4do)

## Feedback Loops Are All You Need

Granola makes AI features easier to improve by exposing the agent’s decisions and turning its desktop interface into shareable web previews.

Basic familiarity with LLM tool calls, pull requests, and web front ends will help; Electron’s process model is explained in the article.

### A summary that notices what you wrote down

[0:15](https://www.youtube.com/watch?v=ON5LIT0M4do&t=15s)

How do you make meeting notes reflect what a person actually cared about? At [Granola](https://www.granola.ai/), that is a product engineering problem as much as a model problem. Mehedi Hassan approaches it from a career spanning jQuery, React’s reshaping of front-end development, and now LLMs. His focus here is the work around the model: making its behavior useful and making the resulting product easier to test.

Granola sits in the dock and captures both system audio and microphone audio to produce a real-time transcript. In the opening demonstration, Hassan uses a recording of the preceding talk. Alongside the transcript, users can write their own notes, giving the app a signal about what deserves emphasis rather than leaving it to summarize every topic equally.

When Hassan generates the summary, he points to how his note about twenty percent overlap influenced the output’s emphasis. The percentage belongs to the content of the preceding talk; it is not a measurement of Granola’s accuracy. The intended result resembles what the user would have written on a notepad, with the transcript supplying the detail. Hassan describes the ambition as best-in-class meeting notes across roles, delivered without getting in the user’s way.

### A chat box meets specific expectations

[1:52](https://www.youtube.com/watch?v=ON5LIT0M4do&t=112s)

That unobtrusive product philosophy becomes harder to maintain as AI features expand. Granola’s existing chat can answer questions about a single meeting, multiple meetings, or shared context. Hassan imagines building a version in one shot and shipping it in a fake Granola app. Getting the chat interface working is easy; the hypothetical users immediately expose what it does not understand.

![Slide posing the production question beside a Granola chat response listing promised follow-ups.](https://ai.engineer/api/talk-essay-media/1b4dc39703412d0daaa0690a9c1a9811f542a27b6b07a116e9a11664ef053c39)

What happens when you put a simple AI feature into production?

Their complaints span several different problems:

- **Task completion:** It cannot produce the requested to-do list.
- **Latency:** Web search takes too long.
- **Personal style:** Follow-up emails do not sound like the person sending them.
- **Intent:** A request for coaching about meetings produces an answer about a football coach.

These examples make the gap concrete: a general chatbot can respond fluently while still failing the particular job the product promises to do. Molding the model to that job becomes the difficult part.

### The operational cost of one-line web search

[2:56](https://www.youtube.com/watch?v=ON5LIT0M4do&t=176s)

Web search illustrates how little an integration API says about the behavior behind it. Adding a search tool may look like a single line of code, but complex queries can increase token usage, consume context, and raise costs. Hassan uses roughly ten pence per web-search-assisted chat as an illustrative cost that becomes difficult to sustain in a millions-of-users scenario. He does not identify a model, workload, or measured average for that example.

Control over quality is a separate problem. Hassan reports that Granola had used a model for some time when an overnight provider update degraded web search. The team could see the deterioration but did not know what had changed inside the provider’s system. Switching providers was the available response, even though the behavior directly affected Granola’s user experience.

![Slide listing token costs, quality drops after model updates, and lack of control over results, beside an image of Earth.](https://ai.engineer/api/talk-essay-media/eecb9468a32cc070e95503c95c24253ff29f417a32df49ad6838c5f5f203c45f)

Web search looks like one line of code.

The complete search slide groups these concerns together: token costs, quality changes after model updates, and limited control over results. Hassan points to the existence of billion-dollar search companies to emphasize how much complexity sits behind the apparently simple tool integration. **Making search callable does not make its cost or quality predictable.**

### One meeting, different useful outputs

[4:01](https://www.youtube.com/watch?v=ON5LIT0M4do&t=241s)

Even when the information is correct, the output can miss the user’s needs. The opening summary suited Hassan, but another role might expect a different selection and organization of the same meeting content.

| Role | Expected emphasis |
|---|---|
| Sales | Deals |
| Engineering | Action items, blockers, Linear tickets |
| HR | Different needs, left unspecified |

One prompt generally cannot serve all of those expectations. The problem is not simply generating a good-looking summary; it is understanding why the model produced this summary for this use case. That requires inspecting behavior that otherwise feels opaque and difficult to steer.

### Follow the agent from beginning to end

[4:39](https://www.youtube.com/watch?v=ON5LIT0M4do&t=279s)

Granola’s response was to build its own **tracing tools**. This is a place where Hassan finds one-shot generation useful: it reduces the effort required to build internal software around the product. The trace exposes individual tool calls from beginning to end, recorded reasons for those calls, search trails, reasoning trails, and cost. Instead of seeing only the final answer, an investigator can follow the sequence that produced it.

The team tailored both the data structure and the interface to its own investigations. Engineering, product, data, and customer experience staff all need to use it; diagnosing a failure should not require writing complex CloudWatch queries. Hassan contrasts this with the earlier economics of internal tooling, when a team might have depended on a SaaS product because building a suitable interface took too much time. The displayed trace makes that interface concrete, showing reasoning, meeting search, calendar search, and expanded calendar results in one vertical sequence.

![Slide pairing tracing guidance with a vertical trace showing reasoning, meeting search, calendar search, and expanded calendar results.](https://ai.engineer/api/talk-essay-media/810f1e0be6680a2c2d8cc72f0d5e910db721fd630556f20d2698ac778b32f14d)

Tracing the agentic loop end-to-end.

The implementation he describes is deliberately small: wrap AI SDK calls, save records to a database, and build the front end around how people investigate failures. He also names OpenTelemetry and other providers as options. The related [AI SDK telemetry documentation](https://ai-sdk.dev/docs/ai-sdk-core/telemetry) is an implementation reference, rather than a specification of Granola’s wrapper. The essential product decision is to make the collected data usable by the people who need to act on it.

Hassan describes Granola’s founder following an entire agent loop from front to back to find what went wrong. That changes the next iteration: a vague impression that an output feels wrong becomes an identifiable failure in the sequence. **Tracing is valuable when it turns dissatisfaction into a specific thing to change.**

### Make desktop experiments easy to open

[6:19](https://www.youtube.com/watch?v=ON5LIT0M4do&t=379s)

Inspecting model behavior addresses only part of the feedback loop. Users also experience the interface around the output. LLMs make it cheaper to build different features and multiple versions of the same feature, but those variants are useful only if people can try them. Granola’s desktop setup allowed one app instance at a time, which made testing variants in parallel awkward.

Previously, a tester had to install dependencies and run the [Electron](https://www.electronjs.org/docs/latest/tutorial/process-model) app locally. Sharing a change with a coworker meant asking that person to repeat the setup. Granola removed much of that friction by turning the Electron front end into an online **web shell**. Its CI now produces a preview link whenever a pull request opens, so colleagues can open the proposed interface directly. Hassan reports that this substantially sped up development.

The preview also gives an agent somewhere to exercise the change. In Hassan’s reported workflow, [Cursor](https://cursor.com/blog/agent-computer-use) tests the preview after a pull request opens and uploads a screenshot to the PR. This adds visible testing evidence to the review; it does not establish that every behavior has been verified. The same accessible environment serves both human reviewers and the agent.

### Separate the renderer from its desktop environment

[7:44](https://www.youtube.com/watch?v=ON5LIT0M4do&t=464s)

The web shell depends on a boundary Electron already provides. Its **main process** handles native capabilities, while its **renderer process** supplies the web-based front end. IPC—inter-process communication—lets the renderer request functionality across that boundary; IPC is not itself a system API. Granola abstracted those calls so that the web environment could use web-standard implementations.

A small TypeScript example shows the shape of that boundary using a clipboard operation. The renderer receives one interface; the desktop entry point can supply its IPC-backed implementation, while the browser entry point supplies the Clipboard API:

```typescript
interface ClipboardPort {
  writeText(text: string): Promise<void>;
}

function createCopyNotes(clipboard: ClipboardPort) {
  return async (notes: string): Promise<void> => {
    await clipboard.writeText(notes);
  };
}

// Browser entry point
const webClipboard: ClipboardPort = {
  writeText: (text) => navigator.clipboard.writeText(text),
};

const copyNotes = createCopyNotes(webClipboard);

const button = document.createElement('button');
button.textContent = 'Copy notes';
button.addEventListener('click', () => {
  void copyNotes('Follow up on the meeting action items.').catch(
    (error: unknown) => console.error('Could not copy notes', error),
  );
});
document.body.append(button);
```

Clipboard copying is an illustrative application of the abstraction, not a reported Granola feature from this demonstration. Browser clipboard writes require a secure context and are subject to browser permission and activation rules. The architectural point is that the consuming code does not need to know whether the implementation crosses Electron IPC or calls a browser API.

Granola applied the same separation to its React-facing infrastructure: routing, sessions, and the query layer moved to web standards. With those dependencies no longer tied to Electron, the renderer could run as a web app. The architecture slide brings the IPC and React abstractions together as the basis of the web shell.

![Slide describing Electron IPC and React API abstractions alongside code for a web shell and environment-independent routing.](https://ai.engineer/api/talk-essay-media/c29602dea8c11a9ada2e0b5fda7dcee390c033141198afff2845eabef93e0800)

How the web shell works.

The payoff is the ability to experience several versions of a feature in practice. Instead of judging only a Figma design, the team can use each version, feel how it behaves, and compare alternatives before deciding what ships. More generated variants become more useful experiments because the cost of opening and testing them has fallen.

### Build conviction through iteration

[8:45](https://www.youtube.com/watch?v=ON5LIT0M4do&t=525s)

**The useful improvement is a better feedback loop, not simply a better first generation.** Hassan compares working with an LLM to playing tennis: the result develops through repeated exchanges. Traces make the model’s failures inspectable; accessible previews let people experience proposed changes. Together, they replace hoping a feature will work for customers with a stronger basis for believing it will connect. The desired experience can feel like magic to the user without remaining an unexplained black box to the team building it.

The closing platform question tests whether that approach requires replacing Electron with [Tauri](https://v2.tauri.app/concept/architecture/). Granola has considered the move several times and tried Tauri, but Hassan says Electron currently serves the team well. Hassan reports that Granola’s Tauri trials did not produce major performance gains, the team’s main criterion for the move. He also mentions changing APIs without identifying which platform he means. The experiments have not shipped: the practical improvement described here came from making the existing renderer portable and easier to test.

## Resources

- [Granola](https://www.granola.ai/) — Meeting-notes application that captures computer audio and enhances notes without adding a meeting bot.
- [AI SDK telemetry documentation](https://ai-sdk.dev/docs/ai-sdk-core/telemetry) — Instrumentation guidance for generation and tool-call traces, including controls for recording inputs and outputs.
- [Electron's process model](https://www.electronjs.org/docs/latest/tutorial/process-model) — Explains main and renderer processes, preload scripts, and communication between desktop capabilities and web interfaces.
- [Cursor agents with computer use](https://cursor.com/blog/agent-computer-use) — Shows how cloud agents test applications in isolated environments and return screenshots, videos and logs for review.
- [Tauri architecture](https://v2.tauri.app/concept/architecture/) — Introduces the architecture of Tauri applications and their use of operating-system WebViews.

## Slides

### Product Engineer at Granola

Mehedi introduces himself as a Granola product engineer, tracing his experience from jQuery through React to the changes brought by LLMs.

[0:31](https://www.youtube.com/watch?v=ON5LIT0M4do&t=31s) [Introduction slide identifying the speaker as a product engineer at Granola, alongside a cropped app image.](https://assets.ai.engineer/sites/aiecode2025/sha256/b7ff9b4be825b8de0ae64ef639e7f187f1961767f3578db40af842b99333e1ba.webp)

Product Engineer
@ Granola

Making apps since jQuery was cool.

April 9, 2026
AI Engineer Europe

Interface text:
WE[illegible]
28
Google DeepMind

### Granola is an app for getting your work done

Granola sits in the dock and transcribes system and microphone audio in real time, then produces meeting notes without a bot joining the call.

[0:45](https://www.youtube.com/watch?v=ON5LIT0M4do&t=45s) [Pink slide describing Granola as a meeting notes app that sits on the dock, works without bots joining calls and stays out of the way.](https://assets.ai.engineer/sites/aiecode2025/sha256/30dc0ec942605449492c116e6af6d12d8c0d700eca48b25fb4c304a41a045234.webp)

Granola is an app for getting your work done.

No bots join your call. Granola sits on your dock and takes awesome notes.

Best in class meeting notes no matter what role you’re in.

Doesn’t get in your way.

April 9, 2026
AI Engineer Europe

Interface text:
Granola
AIE
Engineering the future of AI
AI Engineer EUROPE 2026

[1:02](https://www.youtube.com/watch?v=ON5LIT0M4do&t=62s) [Pink slide introducing Granola as a meeting notes app that sits on the dock and stays out of the way.](https://assets.ai.engineer/sites/aiecode2025/sha256/4048efd4042f2c9fa40d03029070cbdf1220e1435ed3fbfa98e67fb801564d1e.webp)

Granola is an app for getting your work done.

No bots join your call. Granola sits on your dock and takes awesome notes.

Best in class meeting notes no matter what role you’re in.

Doesn’t get in your way.

April 9, 2026
AI Engineer Europe

Interface text:
Granola
AIE
Braintrust
WorkOS
OpenAI
AI Engineer EUROPE 2026

### Granola meeting notes demo

The presenter shows notes from the previous voice AI talk and explains that he can add his own notes alongside Granola’s transcription to shape the final result.

[1:08](https://www.youtube.com/watch?v=ON5LIT0M4do&t=68s) [Granola displays notes from a talk about voice AI, including latency and conversational overlap.](https://assets.ai.engineer/sites/aiecode2025/sha256/7deba97880c7c752294d994f40a065125da8a6bbaec2b352aa9ea14f9705d3d8.webp)

AIE: Her moment for voice AI

- latency is the biggest bottleneck
- human convo has roughly 20% overlap and varies quite a lot by language
- takeways
  - interactive models that are fully-duplex will get us as close to a human convo
  - cost of TTS is way higher than LLM
- gradium phonon
  - on device tts
  - 1.48% WER, 56.3% on speaker run (?)

Interface text:
Granola  Edit  View  Window  Help
Thu 9 Apr 14:54
AIE  Me  Add to folder
Please select the GranolaMacWebcam camera in your meeting app
You may need to quit and restart the app or browser
Ask anything  Suggest topics
AIE
AI Engineer EUROPE 2026
Braintrust  WorkOS  OpenAI

### Generated voice AI summary

Granola generates a structured summary of a voice AI talk, expanding the presenter’s note on roughly 20% conversational overlap into sections on latency and technical approaches.

[1:28](https://www.youtube.com/watch?v=ON5LIT0M4do&t=88s) [Granola displays a structured summary of voice AI latency, conversation overlap, and technical approaches.](https://assets.ai.engineer/sites/aiecode2025/sha256/daadf8f59b7f91ff3d5fa785ab32b9069ceb269b6301affe314155673e5ceca9.webp)

AIE: Her moment for voice AI

Current State of Voice AI
• Latency is the biggest bottleneck for natural conversation
  ◦ Current TTS still >300ms vs required ~100ms for human-like flow
  ◦ Tool calls add 500ms-2s additional delay
  ◦ Fighting for every millisecond improvement
• Human conversation has ~20% overlap, varies by language
  ◦ Japanese: up to 20% overlap as sign of active listening
  ◦ Current models are duplex only (listening OR speaking)
  ◦ Cannot handle interruptions, overlapping speech, backchannel responses

Technical Approaches & Limitations
• Cascading systems (speech→text→speech) inherently high latency
• Speech-to-speech models reduce latency but still duplex
• Filler strategy: AI continues conversation while waiting for tool results
  ◦ Demonstrated with travel agent demo
  ◦ Keeps flow natural during data retrieval
• Full-duplex models needed for human-like interaction
  ◦ [illegible]

Interface text:
Granola  Edit  View  Window  Help
Thu 9 Apr 14:55
Suggested folder  Granola team  Add
Share
Ask anything
Write follow up email
AIE
AI Engineer EUROPE

### Granola introduction revisited

Granola takes meeting notes from the dock without adding a bot to the call. The speaker says staying out of the user’s way has been its product philosophy since day one.

[1:43](https://www.youtube.com/watch?v=ON5LIT0M4do&t=103s) [Pink slide describing Granola's meeting notes and unobtrusive design.](https://assets.ai.engineer/sites/aiecode2025/sha256/8d663fc55b957a06803e959e777404b2691d1f856e3d7311deb3251f6656f167.webp)

Granola is an app for getting your work done.

No bots join your call. Granola sits on your dock and takes awesome notes.

Best in class meeting notes no matter what role you’re in.

Doesn’t get in your way.

April 9, 2026
AI Engineer Europe

AIE
AI Engineer EUROPE
2026

### Moulding LLM behaviour to your use case is hard

Users encountered slow web search, mismatched follow-up emails and meeting coaching that brought up a football coach. These examples show the difficulty of shaping an LLM for a specific use case.

[2:48](https://www.youtube.com/watch?v=ON5LIT0M4do&t=168s) [A pink slide states that moulding LLM behaviour to a use case is hard, beside a shell image.](https://assets.ai.engineer/sites/aiecode2025/sha256/7311fcf6cd67f6ea168059fbcd0b423de58cf92a83c687647ab977ef5338efae.webp)

Moulding LLM behaviour to your use case is hard.

April 9, 2026
AI Engineer Europe

Interface text:
AIE
Engineering the future of AI
AI Engineer EUROPE 2026

### Web search looks like one line of code

Adding web search may take one line, but complex queries drive up token costs, a model update degraded search quality overnight, and developers have little control over what comes back.

[3:09](https://www.youtube.com/watch?v=ON5LIT0M4do&t=189s) [A slide lists web search complications beside an image of Earth.](https://assets.ai.engineer/sites/aiecode2025/sha256/22033f229db4298e1c1a2376b9e3953ce2f3a0c862cdca3f3fd4ba70538dde03.webp)

Web search looks like one line of code

• Token costs explode with complex queries
• [model] worked great, then [big corp] shipped an update and quality dropped overnight
• No control over what comes back

Billion-dollar companies exist just to solve this.

April 9, 2026
AI Engineer Europe

Interface text:
AIE
Engineering the future of AI

### The right output depends on who's reading it

Granola summaries serve different readers: sales wants deal focus, while engineering wants blockers and action items. One prompt cannot serve everyone, and consistent output quality can take months.

[4:00](https://www.youtube.com/watch?v=ON5LIT0M4do&t=240s) [Slide comparing the output needs of sales, engineering and HR, alongside an image of the moon.](https://assets.ai.engineer/sites/aiecode2025/sha256/1d999c846e82b43b5c1bb7374291789506ee646adf789161a7cc8cf4c5430dcf.webp)

The right output depends entirely on who's reading it.

• Sales wants deal focus, next steps, objections
• Engineering wants decisions, blockers, action items
• HR wants something else entirely

One prompt can't serve all of them. And LLMs are stubborn, getting consistent output quality takes months.

April 9, 2026
AI Engineer Europe
AIE
[illegible]

### LLM behaviour doesn't have to be a black box

Granola’s tracing tools show tool calls, search and reasoning trails, and cost, giving the team a closer look at agent behavior.

[4:29](https://www.youtube.com/watch?v=ON5LIT0M4do&t=269s) [Pink slide stating that LLM behaviour does not have to be a black box.](https://assets.ai.engineer/sites/aiecode2025/sha256/1027b811e29a57b41973cec8659134e7282bdd3f6d33c2377b6a2172a0cfb433.webp)

LLM behaviour doesn’t have to be a black box.

AI Engineer Europe

SPEAK IT INT[illegible]

Interface text:
AIE
Engineering the future

### Tracing the agentic loop end-to-end

Granola built an internal tracing tool to inspect tool calls, search and reasoning trails, and cost. The example follows meeting and calendar searches, with results shown for the calendar call.

[4:38](https://www.youtube.com/watch?v=ON5LIT0M4do&t=278s) [Slide describing tracing tool calls, search results, reasoning and cost, with an example trace interface.](https://assets.ai.engineer/sites/aiecode2025/sha256/e02ad63d531c496225b2d72ecc036b7e2ea41ede482f01de5e64c0c6118f17c2.webp)

Tracing the agentic loop end-to-end

Trace every step
Tool calls, search results, reasoning trails, cost - structured how we want it, without waiting on someone else’s roadmap.

LLMs make it much easier
Previously, building your own tracing tools would almost never make it into the roadmap, but with LLMs, you can actually one shot most of it.

Agent INPUT
Reasoning

Finding meetings about "AI Engineer Europe"
search_meetings_by_topic
limit: 10
10 meeting(s) found

Searching calendar for "AI Engineer Europe"
search_calendar_by_topic
keywords: ["AI Engineer Europe"]
10 event(s) found

success  Searching calendar for "AI Engineer Europe"
INPUT
keywords: ["AI Engineer Europe"]  maxResults: 10
10 CALENDAR EVENT(S)
AI Engineer Europe: Keynote speakers, Workshop sessions, Networking & Demo day
3/30/2026, 11:00 AM  13 attendees
AI Engineer Europe: Keynote speakers, Workshop sessions, Networking & Demo day
4/7/2026, 10:00 AM  13 attendees
AI Engineer Europe: Keynote speakers, Workshop sessions, Networking & Demo day
4/13/2026, 10:00 AM  13 attendees

April 9, 2026
AI Engineer Europe

Interface text:
AIE
Engineering the future of AI
AI Engineer EUROPE 2026

### User experience beyond LLM tracing

Tracing reveals LLM behaviour, but user interaction and product experience are equally important. The team uses LLMs to experiment with feature variants for different users.

[6:18](https://www.youtube.com/watch?v=ON5LIT0M4do&t=378s) [Slide explaining that user interaction and product experience are as important as LLM behaviour, beside two rabbits.](https://assets.ai.engineer/sites/aiecode2025/sha256/949374ad3a2ee12d78ad752da2f02a453c9c0a28c0bb4f56d320d942f0061a05.webp)

Tracing tools helped us go deep into LLM behaviour.

But LLM behaviour is only part of the picture.

How users interact and experience your product is equally important.

April 9, 2026
AI Engineer Europe

A future where the m[illegible]
your day was the new[illegible]
you covered, not the [illegible]
hoops you jumped.

Interface text:
AIE
AI Engineer EUROPE
2026

### The feedback loop

A fast feedback loop lets the team try product variants in practice and iterate with an LLM, building confidence that what they ship will connect with users.

[8:47](https://www.youtube.com/watch?v=ON5LIT0M4do&t=527s) [Pink slide says the answer is to make the feedback loop fast enough that the end product feels like magic.](https://assets.ai.engineer/sites/aiecode2025/sha256/adaf3a05fc42d9ecdd94babf89945afa3a37936cbf2fa8e1022b8d708adebb5b.webp)

The answer isn’t to one-shot better.

It's about making the feedback loop so fast that the end product feels like magic.

April 9, 2026
AI Engineer Europe

Interface text:
AIE
Braintrust
WorkOS
OpenAI
AI Engineer EUROPE 2026

## Full transcript

[0:00](https://www.youtube.com/watch?v=ON5LIT0M4do&t=0s) [upbeat music] Cool.

[0:15](https://www.youtube.com/watch?v=ON5LIT0M4do&t=15s) How's it going, guys? Uh, I'm Mehedi. Uh, we're gonna talk about some product engineering stuff that we've been doing at Granola. Uh, this is not gonna go deep into AI engineering stuff, so if you're here for me to, like, go into LLMs and stuff, it's not gonna happen.

[0:28](https://www.youtube.com/watch?v=ON5LIT0M4do&t=28s) I'm warning you right now, uh, so you know what's coming. Cool. So I'm a product engineer at Granola. I've been, you know, coding since jQuery was cool. I've seen React kind of change front-end engineering, and obviously now experiencing LLMs change, um, engineering and everything else, uh, just like many of you.

[0:45](https://www.youtube.com/watch?v=ON5LIT0M4do&t=45s) For those of you who don't know, Granola is an app for getting your work done. Essentially, we're a meeting notes app where we s- uh, s-sit on your dock, uh, like it is doing right now.

[0:56](https://www.youtube.com/watch?v=ON5LIT0M4do&t=56s) Um, and it has access to your system audio, as well as your microphone audio, which means we have real-time transcription, and then at the end of your meeting, we can give you really awesome notes.

[1:07](https://www.youtube.com/watch?v=ON5LIT0M4do&t=67s) So I'm just gonna give you a quick demo. So I was, uh, recording the previous, um, talk right here, and you can see it picked up literally everything the presenter said.

[1:16](https://www.youtube.com/watch?v=ON5LIT0M4do&t=76s) Uh, and the cool thing about Granola is that you can also write your own notes on top of what the transcription is saying. So the final result is more aligned to, like, what you'd normally actually write on a notepad.

[1:27](https://www.youtube.com/watch?v=ON5LIT0M4do&t=87s) So I'll go ahead and generate the, uh, notes here, and you'll see that this will go ahead and write a really good summary. Um, and as you can see, like, I wrote down this twenty percent overlap thing, and it focused more on the output, right?

[1:40](https://www.youtube.com/watch?v=ON5LIT0M4do&t=100s) So this is, uh, Granola. We have the best-in-class meeting notes, no matter what role you're in, and it doesn't get in your way, and that's been, like, our product philos-philosophy since day one.

[1:52](https://www.youtube.com/watch?v=ON5LIT0M4do&t=112s) So we ship a lot of, uh, AI features, uh, in Granola, and our product is known to be, again, not to get in your way. So let's see what happens when you put a simple AI feature into prod.

[2:03](https://www.youtube.com/watch?v=ON5LIT0M4do&t=123s) Uh, I'm gonna kind of give you an example with this chat feature that we have. This is a feature that already exists, uh, in Granola. You can ask questions about a meeting that you just had and across a bunch of different meetings, uh, or, like, shared context, uh, as well, and Granola will try to answer it to

[2:17](https://www.youtube.com/watch?v=ON5LIT0M4do&t=137s) the best of your, uh, its ability. So let's say I built, you know, like I one-shot this, uh, chat system. It's very easy to do. Uh, and I put it into production in my fake, uh, Granola app.

[2:28](https://www.youtube.com/watch?v=ON5LIT0M4do&t=148s) And then as soon as users hit it, you know, it's like it can't give me a list of to-dos. Uh, web search is too slow. Uh, it's not writing follow-up emails how I normally write my emails.

[2:36](https://www.youtube.com/watch?v=ON5LIT0M4do&t=156s) I ask it to coach me about my meetings, and it's telling me things about my football coach. Obviously, these are very, uh, pro-- very common problems that you're gonna run into when you make a generate, uh, chatbot.

[2:47](https://www.youtube.com/watch?v=ON5LIT0M4do&t=167s) So how do we get around this, right? So m-- what we've seen is, like, molding, uh, the LLM to work to a specific use case can be super hard.

[2:56](https://www.youtube.com/watch?v=ON5LIT0M4do&t=176s) Um, and one of the examples is, is web search. So web search for most, uh, LLM providers looks like a line of code. You simply add the web search tool, and you expect it to just work.

[3:06](https://www.youtube.com/watch?v=ON5LIT0M4do&t=186s) That's what the labs want you to be-believe, but once you get into it, there's lots of other complications. So for example, the token usage and token costs can bubble up quite a lot, especially for complex queries.

[3:16](https://www.youtube.com/watch?v=ON5LIT0M4do&t=196s) It's gonna blow up your context, uh, and each chat could be costing you, like, ten pence. Obviously, at scale, when you have millions of users, this is not really feasible.

[3:24](https://www.youtube.com/watch?v=ON5LIT0M4do&t=204s) Um, and then, you know, like, the web search providers are also, like, completely up to the labs as well. So for example, in, uh, in our, uh, development, what we see was, like, was we were using a model for a good amount of time, and then overnight they shipped an update, and for some reason web search degraded,

[3:40](https://www.youtube.com/watch?v=ON5LIT0M4do&t=220s) uh, and it was completely out of our control. And we generally had no idea, like, what was going on apart from just, like, switching, uh, providers. But we wanna have more control over that because it affects our user, user experience.

[3:51](https://www.youtube.com/watch?v=ON5LIT0M4do&t=231s) Um, and you know, like, there's literally billion-dollar companies who do web search. So that kind of tells you that it's what's much more than just adding a web search tool to your LLM pipeline.

[4:01](https://www.youtube.com/watch?v=ON5LIT0M4do&t=241s) The other thing that's super important for apps like Granola is the output. So the summary that you saw was pretty good for what I would expect. But someone in sales might expect more of a deal focus.

[4:12](https://www.youtube.com/watch?v=ON5LIT0M4do&t=252s) Someone in engineering might expect, like, action items, blockers, like Linear tickets. HR might want something completely different. And the thing is that one prompt can't generally serve everyone. And, you know, LLMs are stubborn, and we need to figure out how to get inside them and make them work how we want it to work. [clears throat]

[4:30](https://www.youtube.com/watch?v=ON5LIT0M4do&t=270s) And yeah, as you, as you know, like, LLM behavior is largely seen as like a black box, but we wanna kind of go very deep into the details and figure out exactly what's going on.

[4:39](https://www.youtube.com/watch?v=ON5LIT0M4do&t=279s) So what we did recently at Granola is we started building our own tracing tools, and obviously, thanks to LLMs, you can actually one-shot these things. And this is where one-shotting is kind of nice.

[4:49](https://www.youtube.com/watch?v=ON5LIT0M4do&t=289s) Um, and so we built our tooling, uh, tracing tools here, where we basically have complete visibility on the tool calls straight from the beginning to the end. So we have full visibility over the individual tool calls, why it's making those tool calls, the search trails, the reasoning trails, the cost.

[5:04](https://www.youtube.com/watch?v=ON5LIT0M4do&t=304s) It, it's structured exactly how we want it. And the most useful part of this is that we structured the data exactly how we want it, and the UI is built to, like, serve our, uh, our employees internally, not just, like, engineers, but also product, um, data, and, like, CX and everyone.

[5:20](https://www.youtube.com/watch?v=ON5LIT0M4do&t=320s) So you don't have to, like, you know, go into CloudWatch and do, like, very complex queries to figure out why something failed. And that's been, like, the key, uh, for us to, like, figuring out this black box.

[5:29](https://www.youtube.com/watch?v=ON5LIT0M4do&t=329s) Um, and previously, obviously, building this kind of tools would be, like, up to using a SaaS provider. Um, and it simply wouldn't-- You simply wouldn't have the time. But now you actually can spend time building this tracing tool that actually serves what you need.

[5:43](https://www.youtube.com/watch?v=ON5LIT0M4do&t=343s) And this is obviously a very, um, basic example, but obviously you can use OpenTelemetry or, like, other providers. But we essentially just, like, save things, uh, to a DB, wrap around, like, AI SDK, uh, and then the front-end is, like, kind of the most important part 'cause that's what people are gonna use to figure out what, um,

[6:00](https://www.youtube.com/watch?v=ON5LIT0M4do&t=360s) breaks and what doesn't. And we literally have, like, our founder literally goes into, like, the details, like following the agent, um, loop completely front to back to figure out exactly what went wrong.

[6:11](https://www.youtube.com/watch?v=ON5LIT0M4do&t=371s) So then at the end of this, you can actually figure out, like, you know, this output feels off to, like, exactly what failed. And then when you iterate, you can improve on those things.

[6:19](https://www.youtube.com/watch?v=ON5LIT0M4do&t=379s) But as I said earlier, this is gonna be more, more than just like basic LLM stuff. Um, and LLM behavior is obviously part of the picture. The how users interact and experience is, your product is also very important.

[6:33](https://www.youtube.com/watch?v=ON5LIT0M4do&t=393s) So with LLMs, you can one-shot more things, and you can have more variants, which we like, 'cause we can experiment with different features. We can experiment with, like, one feature looking very, uh, different and for different users.

[6:44](https://www.youtube.com/watch?v=ON5LIT0M4do&t=404s) But the problem for us specifically at Granola was that we are a desktop app, which means you can only run one instance of the app at a time. And there was a lot of friction when it came to, like, testing new features, different variants, uh, and actually testing those in parallel. [clears throat]

[6:59](https://www.youtube.com/watch?v=ON5LIT0M4do&t=419s) So before, you know, um, before you'd have to like run the Electron app locally, install the dependencies, uh, and test things. If you wanted a coworker te- to test those changes, you'd have to get them to do those things as well.

[7:11](https://www.youtube.com/watch?v=ON5LIT0M4do&t=431s) We have- we didn't have the same luxuries as like web apps do. So essentially what we did is we took our Electron app, and we turned the front end of the Electron app into a web shell, and this was deployed online.

[7:22](https://www.youtube.com/watch?v=ON5LIT0M4do&t=442s) So now our CI, whenever we open a PR, we get a preview link, and we can go and test those things. And this generally sp- sped up our development time so much more.

[7:31](https://www.youtube.com/watch?v=ON5LIT0M4do&t=451s) Uh, and like the cooler part of this is that because LLMs can now self-verify their work, these guys are now, like, once we open a PR, Cursor goes and tests it, uploads a screenshot into our PRs, which, uh, speeds up the testing so much more.

[7:44](https://www.youtube.com/watch?v=ON5LIT0M4do&t=464s) And again, this is like, you might think that this is a lot of work, but it's actually quite simple. So what we did is, um, for those of you who are not familiar with Electron, there's obviously a main process, um, and a renderer process.

[7:56](https://www.youtube.com/watch?v=ON5LIT0M4do&t=476s) The main process works with the system APIs, and the renderer process is basically your front end. Uh, and essentially, we abstracted our IPC APIs, which is the system APIs, uh, to fall back to web standards when we're in the web environment.

[8:08](https://www.youtube.com/watch?v=ON5LIT0M4do&t=488s) And similarly with React APIs as well, like routers, sessions, and query layer, we moved those to the, um, uh, web standards. Um, and essentially, this just made the renderer agnostic of re- of Electron, and we could just simply run it as a web, web app.

[8:22](https://www.youtube.com/watch?v=ON5LIT0M4do&t=502s) So essen- this has helped us, on top of the LLM improvements, uh, was like we were able to just like change, uh, like test like one feature in like multiple different variants.

[8:31](https://www.youtube.com/watch?v=ON5LIT0M4do&t=511s) So like whatever the end product is actually feels super good 'cause we know that we've tried so many different variants, um, and we actually felt those, um, products in, in like in practice rather than just like seeing it in Figma.

[8:45](https://www.youtube.com/watch?v=ON5LIT0M4do&t=525s) So essentially this, this is basically a long talk to tell you that the answer isn't to one-shot better. It's about figuring out how you can make that feedback loop where it kind of feels like playing a tennis game with LLM so the end product feels more like magic rather than just like a black box, and hoping that

[9:00](https://www.youtube.com/watch?v=ON5LIT0M4do&t=540s) the feature that you're gonna release works well with customers, and having that conviction that what you are shipping is actually gonna connect to the users. Yeah.

[9:09](https://www.youtube.com/watch?v=ON5LIT0M4do&t=549s) Thank you. Any questions? [clapping] Are you thinking about replatforming from Electron to Tauri? Um, we, we've thought about moving to Tauri, uh, a couple times.

[9:26](https://www.youtube.com/watch?v=ON5LIT0M4do&t=566s) Um, I think the way Electron serves us right now has been super nice. Uh, like the API is changing quite, quite a lot. We've tried Tauri before as well, and we didn't really see, uh, massive performance gains, uh, which we, which is what we care about the most.

[9:39](https://www.youtube.com/watch?v=ON5LIT0M4do&t=579s) Um, so yeah, it's been discussed before. We've played around with it but haven't shipped it.

[9:48](https://www.youtube.com/watch?v=ON5LIT0M4do&t=588s) Cool. Thank you, guys. [clapping] [outro music]
