Feedback Loops are All You Need

Talk transcript

1. 0:00

   [upbeat music] Cool.

2. 0:15

   How's it going, guys? Uh, I'm Mehedi. Uh, we're gonna talk about some product engineering stuff that we've been doing at Granola. Uh, this is not gonna go deep into AI engineering stuff, so if you're here for me to, like, go into LLMs and stuff, it's not gonna happen.

3. 0:28

   I'm warning you right now, uh, so you know what's coming. Cool. So I'm a product engineer at Granola. I've been, you know, coding since jQuery was cool. I've seen React kind of change front-end engineering, and obviously now experiencing LLMs change, um, engineering and everything else, uh, just like many of you.

4. 0:45

   For those of you who don't know, Granola is an app for getting your work done. Essentially, we're a meeting notes app where we s- uh, s-sit on your dock, uh, like it is doing right now.

5. 0:56

   Um, and it-- I have access to your system, uh, transcription, uh, system audio, as well as your microphone audio, which means we have real-time transcription, and then at the end of your meeting, we can give you really awesome notes.

6. 1:07

   So I'm just gonna give you a quick demo. So I was, uh, recording the previous, um, talk right here, and you can see it picked up literally everything the presenter said.

7. 1:16

   Uh, and the cool thing about Granola is that you can also write your own notes on top of what the transcription is saying. So the final result is more aligned to, like, what you'd normally actually write on a notepad.

8. 1:27

   So I'll go ahead and generate the, uh, notes here, and you'll see that this will go ahead and write a really good summary. Um, and as you can see, like, I wrote down this twenty percent overlap thing, and it focused more on the output, right?

9. 1:40

   So this is, uh, Granola. We have the best-in-class meeting notes, no matter what role you're in, and it doesn't get in your way, and that's been, like, our product philos-philosophy since day one.

10. 1:52

    So we ship a lot of, uh, AI features, uh, in Granola, and our product is known to be, again, not to get in your way. So let's see what happens when you put a simple AI feature into prod.

11. 2:03

    Uh, I'm gonna kind of give you an example with this chat feature that we have. This is a feature that already exists, uh, in Granola. You can ask questions about a meeting that you just had and across a bunch of different meetings, uh, or, like, shared context, uh, as well, and Granola will try to answer it to

12. 2:17

    the best of your, uh, its ability. So let's say I built, you know, like I one-shot this, uh, chat system. It's very easy to do. Uh, and I put it into production in my fake, uh, Granola app.

13. 2:28

    And then as soon as users hit it, you know, it's like it can't give me a list of to-dos. Uh, web search is too slow. Uh, it's not writing follow-up emails how I normally write my emails.

14. 2:36

    I ask it to coach me about my meetings, and it's telling me meetings about my football coach. Obviously, these are very, uh, pro-- very common problems that you're gonna run into when you make a generate, uh, chatbot.

15. 2:47

    So how do we get around this, right? So m-- what we've seen is, like, molding, uh, the LLM to work to a specific use case can be super hard.

16. 2:56

    Um, and one of the examples is, is web search. So web search for most, uh, LLM providers looks like a line of code. You simply add the web search tool, and you expect it to just work.

17. 3:06

    That's what the labs want you to be-believe, but once you get into it, there's lots of other complications. So for example, the token usage and token costs can bubble up quite a lot, especially for complex queries.

18. 3:16

    It's gonna blow up your context, uh, and each chat could be costing you, like, ten pence. Obviously, at scale, when you have millions of users, this is not really feasible.

19. 3:24

    Um, and then, you know, like, the web search providers are also, like, completely up to the labs as well. So for example, in, uh, in our, uh, development, what we see was, like, was we were using a model for a good amount of time, and then overnight they shipped an update, and for some reason web search degraded,

20. 3:40

    uh, and it was completely out of our control. And we generally had no idea, like, what was going on apart from just, like, switching, uh, providers. But we wanna have more control over that because it affects our user, user experience.

21. 3:51

    Um, and you know, like, there's literally billion-dollar companies who do web search. So that kind of tells you that it's what's much more than just adding a web search tool to your LLM pipeline.

22. 4:01

    The other thing that's super important for apps like Granola is the output. So the summary that you saw was pretty good for what I would expect. But someone in sales might expect more of a deal focus.

23. 4:12

    Someone in engineering might expect, like, action items, blockers, like Linear tickets. HR might want something completely different. And the thing is that one prompt can't generally serve everyone. And, you know, LLMs are stubborn, and we need to figure out how to get inside them and make them work how we want it to work. [clears throat]

24. 4:30

    And yeah, as you, as you know, like, LLM behavior is largely seen as like a black box, but we wanna kind of go very deep into the details and figure out exactly what's going on.

25. 4:39

    So what we did recently at Granola is we started building our own tracing tools, and obviously, thanks to LLMs, you can actually one-shot these things. And this is where one-shotting is kind of nice.

26. 4:49

    Um, and so we built our tooling, uh, tracing tools here, where we basically have complete visibility on the tool calls straight from the beginning to the end. So we have full visibility over the individual tool calls, why it's making those tool calls, the search trails, the reasoning trails, the cost.

27. 5:04

    It, it's structured exactly how we want it. And the most useful part of this is that we structured the data exactly how we want it, and the UI is built to, like, serve our, uh, our employees internally, not just, like, engineers, but also product, um, data, and, like, CX and everyone.

28. 5:20

    So you don't have to, like, you know, go into CloudWatch and do, like, very complex queries to figure out why something failed. And that's been, like, the key, uh, for us to, like, figuring out this black box.

29. 5:29

    Um, and previously, obviously, building this kind of tools would be, like, up to using a SaaS provider. Um, and it simply wouldn't-- You simply wouldn't have the time. But now you actually can spend time building this tracing tool that actually serves what you need.

30. 5:43

    And this is obviously a very, um, basic example, but obviously you can use OpenTelemetry or, like, other providers. But we essentially just, like, save things, uh, to a DB, wrap around, like, AI SDK, uh, and then the front-end is, like, kind of the most important part 'cause that's what people are gonna use to figure out what, um,

31. 6:00

    breaks and what doesn't. And we literally have, like, our founder literally goes into, like, the details, like following the agent, um, loop completely front to back to figure out exactly what went wrong.

32. 6:11

    So then at the end of this, you can actually figure out, like, you know, this output feels off to, like, exactly what failed. And then when you iterate, you can improve on those things.

33. 6:19

    But as I said earlier, this is gonna be more, more than just like basic LLM stuff. Um, and LLM behavior is obviously part of the picture. The how users interact and experience is, your product is also very important.

34. 6:33

    So with LLMs, you can one-shot more things, and you can have more variants, which we like, 'cause we can experiment with different features. We can experiment with, like, one feature looking very, uh, different and for different users.

35. 6:44

    But the problem for us specifically at Granola was that we are a desktop app, which means you can only run one instance of the app at a time. And there was a lot of friction when it came to, like, testing new features, different variants, uh, and actually testing those in parallel. [clears throat]

36. 6:59

    So before, you know, um, before you'd have to like run the Electron app locally, install the dependencies, uh, and test things. If you wanted a coworker te- to test those changes, you'd have to get them to do those things as well.

37. 7:11

    We have- we didn't have the same luxuries as like web apps do. So essentially what we did is we took our Electron app, and we turned the front end of the Electron app into a web shell, and this was deployed online.

38. 7:22

    So now our CI, whenever we open a PR, we get a preview link, and we can go and test those things. And this generally sp- sped up our development time so much more.

39. 7:31

    Uh, and like the cooler part of this is that because LLMs can now self-verify their work, these guys are now, like, once we open a PR, Cursor goes and tests it, uploads a screenshot into our PRs, which, uh, speeds up the testing so much more.

40. 7:44

    And again, this is like, you might think that this is a lot of work, but it's actually quite simple. So what we did is, um, for those of you who are not familiar with Electron, there's obviously a main process, um, and a renderer process.

41. 7:56

    The main process works with the system APIs, and the renderer process is basically your front end. Uh, and essentially, we abstracted our IPC APIs, which is the system APIs, uh, to fall back to web standards when we're in the web environment.

42. 8:08

    And similarly with React APIs as well, like routers, sessions, and query layer, we moved those to the, um, uh, web standards. Um, and essentially, this just made the renderer agnostic of re- of Electron, and we could just simply run it as a web, web app.

43. 8:22

    So essen- this has helped us, on top of the LLM improvements, uh, was like we were able to just like change, uh, like test like one feature in like multiple different variants.

44. 8:31

    So like whatever the end product is actually feels super good 'cause we know that we've tried so many different variants, um, and we actually felt those, um, products in, in like in practice rather than just like seeing it in Figma.

45. 8:45

    So essentially this, this is basically a long talk to tell you that the answer isn't to one-shot better. It's about figuring out how you can make that feedback loop where it kind of feels like playing a tennis game with LLM so the end product feels more like magic rather than just like a black box, and hoping that

46. 9:00

    the feature that you're gonna release works well with customers, and having that conviction that what you are shipping is actually gonna connect to the users. Yeah.

47. 9:09

    Thank you. Any questions? [clapping] Are you thinking about replatforming from Electron to Tauri? Um, we, we've thought about moving to Tauri, uh, a couple times.

48. 9:26

    Um, I think the way Electron serves us right now has been super nice. Uh, like the API is changing quite, quite a lot. We've tried Tauri before as well, and we didn't really see, uh, massive performance gains, uh, which we, which is what we care about the most.

49. 9:39

    Um, so yeah, it's been discussed before. We've played around with it but haven't shipped it.

50. 9:48

    Cool. Thank you, guys. [clapping] [outro music]