Scaffold Wisely

Talk transcript

1. 0:00

   [on-hold music] So a little bit about me.

2. 0:18

   Um, head of applied AI at Ramp. I've been working on LLMs for four years, which is--

3. 0:26

   Well, which is, uh, kind of a long time, I guess, uh, in LLM land. Everything started happening really when ChatGPT came out. Um, so I was trying to build what people would now call an AI agent company.

4. 0:39

   Back then, we were just doing customer support. We were trying to make our chatbot smarter, and we were trying to figure out what, what models to use to, or what tech to use to get them to respond to customers better.

5. 0:49

   And we were messing with GPT-2 and BERT, and models were so frustratingly stupid, and the context windows were small, and they were not very smart at reasoning, and it was just incredibly annoying.

6. 0:58

   And we just wrote lots of code around these models to get them to work at le-at least somewhat reliably.

7. 1:05

   And along the way, as models got smarter, just kinda had to delete more of that code and just ended up seeing a lot of patterns in what, what code needs to get deleted, how to build agents, in what ways that will scale with more intelligence.

8. 1:19

   And clearly, we're gonna continue to get a lot more intelligence. And I just wanted to, uh, maybe talk about a single idea throughout the talk, uh, through various examples.

9. 1:27

   Uh, we'll, we'll do some, uh, some, uh, setting, but I'll also have a bunch of demos to kind of, like, drive home the point, and maybe I can convince you guys that, uh, there's a certain way of building agents that's slightly better than other ways.

10. 1:41

    I also built a structured extraction library called jsonformer. Um, I think it was the first one. I don't-- I'm not fully sure, but timing-wise, it was before all the other major ones.

11. 1:51

    Um, and that was also scaffolding around a model. Models were too stupid to output JSON, and we were just really begging it, pleading, pleading it, and forcing it to, uh, act in ways that we want it to be.

12. 2:02

    So, as I said earlier, just have a one core agenda item here, which is wanna convey one idea. Uh, we'll start off. All, all of you have probably read the essay, The Bitter Lesson.

13. 2:11

    Just quickly go through what it is. Uh, we'll go through a production agent we have at Ramp and how it works, and three different ways of architecting it. And then I have a demo that to really push maybe how we all think about how software and backends and things will work in the future.

14. 2:27

    So very simply, the idea is just that systems that scale with compute beat systems that don't. So there's two systems, and, uh, without any effort, the sy- the sy- one of the systems can just think more or use more compute in some way.

15. 2:42

    That system tends to beat systems that are rigid and fixed and just deterministic.

16. 2:47

    So from that idea, it's pretty clear, like, if you're building systems, you might as well build systems that improve with more compute. And this, this seems pretty obvious, like, obvious conclusion from The Bitter Lesson.

17. 2:59

    Taking it a step further, why is this true? It's because exponentials are rare. Like, they just don't exist, and most things in the world aren't exponential. So when you find one, you just should hop on, strap on, and just take the free pass and go for the ride, and you probably shouldn't try too hard.

18. 3:13

    And there's a lot of examples from history that, um, kind of reflect this. So for, for chess and Go and computer vision, Atari games, like, people have tried to build lots of systems and written a lot of code.

19. 3:24

    And my way of thinking about rigid systems is just, like, spending a lot of time, grinding weekends, and writing very clever software, well-abstracted, um, maybe trying to synthesize human reasoning and thought process into features, and then using them in clever ways and trying to approximate how a human would think.

20. 3:42

    And if you actually fix the amount of compute, that approach will win. But if you just-- Turns out, if you end up scaling up how much search you're doing, the general method al-al-al- always ends up winning, even, uh, like, in all these cases, so Atari, Go, and computer vision.

21. 3:58

    A little bit about Ramp. So Ramp is a finance platform that helps businesses manage expenses, payments, procurement, travel, bookkeeping more efficiently. And we have a ton of AI across the product, so automate a lot of boring stuff the finance teams do and employees do with, uh, submitting expense reports and booking your flights and hotels and, uh, submitting

22. 4:17

    reimbursements, all that. And so a lot of the work behind the scenes is just we're interacting with other systems, um, and helping, like, legacy systems and helping employees get their work done faster.

23. 4:28

    So let's actually talk through one of the systems we have today at Ramp and, um, maybe some-- talk through, like, the different versions of the system and how it evolved over time.

24. 4:37

    So we're gonna talk about something called a switching report. It's a very simple agent. All it needs to do is take in a CSV, a CSV arbitrary format. So the schema could be re- seriously anything from the internet.

25. 4:50

    And we want these CSVs to come from third-party card providers. So when people onboard to Ramp, we wanna give them a nice checklist and say, "Hey, here are all the transactions you have on other platforms, and we wanna help you move them over.

26. 5:01

    And the more transactions come on Ramp, the more we can help you, and the more you'll use our software, and the more everyone benefits." And so the switching report is just really a checklist.

27. 5:09

    But to read people's CSV transactions, we need to understand those, and other platforms have all these kinds of crazy schemas. And so the, the description of the problem we have here is just for an arbitrary, arbitrary, like, CSV, how can we support, um, parsing it and then into some format that we, we understand?

28. 5:30

    So let's just start with the, the simple approach, right? Is like, let's just take the fifty most common third-party card vendors, um, and let's just manually write code for all of them.

29. 5:39

    Now, obviously, like, this, this will just work. It is some work, not a lot of work, but

30. 5:46

    you still have to maybe go to fifty different platforms and download their CSVs, see what schemas they have, and then write code. Maybe if they decide one day they change their format, your thing will break, but that's okay.

31. 5:56

    You'll get paged, and you can wake up and, and go fix it.

32. 6:01

    So let's maybe introduce some LLMs in here. So from the over-engineered code where you ended up writing a hundred thousand lines, maybe we don't, we don't-- we want a more general system.

33. 6:11

    So let's introduce a little bit of LLMs, a little bit of AI in here. And so in the deterministic flow, let's maybe add some, or just like scripting in classical scripting land, let's add some more, um, calls to OpenAI, or you have an embedding model, you wanna do s- uh, semantic similarity or something like that.

34. 6:28

    So then let's just take every column in the CSV that comes in. Let's try to classify what kind of column it is. Is it a date? Is it a transaction?

35. 6:35

    Uh, is it a transaction amount? Is it a merchant name, or is it the, uh, user's name?

36. 6:39

    And then we map it on, and then we probably could, uh, end up in a schema that we're happy with. Again, most of the com- compute is running in classical land.

37. 6:49

    Some of it is la- running in fuzzy, like, LLM land. But this is somewhat looking like a more general system.

38. 6:57

    Let's go maybe a different approach when, like, we just go all the way through. Let's just say, we're just gonna literally give the CSV to LLM and say, "You have a code interpreter, so you can write whatever code you want, pandas or all the faster Rust-based ones.

39. 7:11

    Um, you have all these Python packages. Um, you're allowed to look at the head of the CSV, the tail, whichever rows you want. Um, and then I just want you to give me a CSV, uh, with this specific format, and here's a unit test, here's a verifier that you can use to tell if it's working or not."

40. 7:28

    Turns out this approach actually doesn't work, like, we tried it, um, if you only run it once. But instead, if you run it fifty times in parallel, it's actually very likely that it works really well and generalizes across a ton of different formats.

41. 7:42

    The amount of compute here is actually probably, like, what is that number? Ten thousand times more than the, the first approach we came up with. But again, like, what is truly scarce in the world is engineer time.

42. 7:52

    Maybe not for-- not in a while, but at least today. And we'd rather have a system that works really well, and even with the ten thousand times more compute, it will probably cost less than a dollar.

43. 8:01

    And every transaction that's switched over, every failed CSV will cost Ramp way more money than whatever money we spend on this exact, this exact architecture.

44. 8:10

    So this is a very specific, uh, example. It's like, how does this apply to the agents that we all build and maybe the systems we're all working on? Turns out something like this actually generalizes.

45. 8:21

    So if you're looking at three approaches, and let's assume, like, the black arrow is just classical compute, and then the blue arrows are fuzzy land. So it goes into neural net, and all, all sort of weird matrix multiplication happens, and then we're in latent space, and it gets all alien intelligency, and then comes back to classical land.

46. 8:39

    First approach, there was no AI. We just wrote code, and it just worked, mostly. The constrained agent, so the second approach, we broke into fuzzy land from classical land when, when we decided we wanted similarity scores or something like that.

47. 8:53

    And then the third approach is actually flipped, where the LLM decides it needs to go into classical land. So it writes some code, writes some pandas or, uh, Python code, and it decides to break in into this classical land when it needs to, but most of the compute is fuzzy.

48. 9:09

    Actually, this is maybe not the most accurate graph, like, uh, because I proposed that we run it fifty times. It more so looks like this. But if you look at a back end in general, they're all request response.

49. 9:21

    So some sort of message is going in, it's like a POST request, or GET, or update, or read, any sort of credit operation. And we're a- really just asking the back end to take this piece of information, do whatever you must with it, run whatever mutations you want, and return me a response.

50. 9:36

    And almost all systems we've built so far is like humanity, I guess, like, look like the first one. But more people are using OpenAI. OpenAI makes billions of dollars, and probably a lot of the systems that use them look like number two, where just regular, uh, programming languages are calling into OpenAI servers, and we're running some fuzzy

51. 9:54

    compute. What we're seeing in, like, more and more parts of the Ramp code base, we're moving to the third approach because it just tends to work well. Because all the blue arrows, if you did nothing, absolutely nothing, we all went to vacation for the next year, the big labs are still working and spending billions of dollars making

52. 10:13

    those models better. And so the blue arrows will get better. And so how much blue arrow you're using in your code base actually will help directly your company without much effort from your end.

53. 10:23

    So this is what I was saying is, like, the bitter lesson is just so powerful, and exponential trends are so powerful that you can just hitch, hitch a ride.

54. 10:34

    Let's, um, let's take this idea, like, further.

55. 10:40

    Um, let's actually, like, go all the way, like, something, something crazy.

56. 10:45

    Um, on the left, you'll see a traditional web app. So usually, the way it works is you open, um, gmail.com, and some, uh, static file server and, and Google sending you a bunch of JavaScript and HTML and CSS.

57. 11:00

    Your browser renders that, um, and shows you some nice UI, nice HTML that's user-friendly. Maybe you see some emails, maybe you click on one of them. Um, the front end makes a request to the back end, the back asks the front end-- The front end asks the back end, "Give me the content for email," and whatever ID

58. 11:17

    it is, and then the back end hits the database and gives you the result. And maybe they use Codgen, maybe they use all the Codgen tools available to make Gmail.

59. 11:25

    So that, that was probably-- The LLM only worked when the software engineer was writing the code. But once the code is written, and it's, like, pushed to production, it's just classical compute.

60. 11:37

    And on the right, I'm actually proposing a different model, which is the back end is the LLM. It's not Codgen. It's this LLM is doing the execution. It is the back end.

61. 11:47

    So the LLM has access to tools like code interpreter and potentially has access to, um, through that, making requests, network requests, and also has an access to, uh, DB.

62. 12:00

    So I have a mail client actually that works with this principle, and this is my test email. So if y'all wanna see any emails you send to me- In a minute or so, you can send me an email.

63. 12:12

    But please be nice. All right, I think, um,

64. 12:38

    that's probably enough time. So I'm gonna go over...

65. 12:57

    So we have this email client. I mean, we still have some regular JavaScript to hook into the LLM, hook the LLM into the browser. But when I do log in, I'm gonna use my email as just showing you...

66. 13:16

    Hmm. Oh. [inaudible]

67. 13:40

    What? Oh, it's probably- Yeah. Okay, we're good, we're good. All right, we're saved.

68. 13:49

    I think. Thankfully, we have a room full of engineers. [laughing]

69. 13:59

    So there's a dot, but the reason it's so slow is because when I open this page and log into Gmail, the Gmail token is actually being sent to an LLM.

70. 14:07

    We're saying literally this is a L- LLM chat session. What we're, we're seeing on the screen is like, "Hey, LLM, you're, you're actually simulating a Gmail client. You have access to," oh, all the emails.

71. 14:19

    Uh, "you have access to, um, Rahul's, uh, Gmail token and a code interpreter. And so just render some UI based on, uh, what you think is reasonable for the homepage for a Gmail client."

72. 14:31

    And so looks like it decided to render as markdown. Uh, I think we actually tell it to render as markdown. And it's rendering all the emails that a bunch of people sent me from here.

73. 14:40

    So looks like it says, uh, "Hello from California." So I'm gonna click on that.

74. 14:45

    When I click on that, we're actually not running, um, any, like, backend calls or anything like that. We're just telling the LLM the user clicked on that piece of text.

75. 14:52

    In this case, it was, "Hello from California," and the ID number. So the LLM now has the information on what the user clicked on, and it has the chance to re-render the page, much like a web framework would.

76. 15:02

    So again, it goes back, it probably hits, uh, a GET request for that specific email and pulls the body.

77. 15:09

    "What is this agent gonna do? I'm watching you live." So the LLM just decided this is the appropriate, uh, UI for a Gmail client.

78. 15:18

    And also, I have, uh, other features the LLM thought was reasonable, so it looks like I could mark it as unread or, or delete the email if I want to.

79. 15:25

    Uh, maybe I'll delete it because it's not that good of an email. I'm sorry. [laughing]

80. 15:34

    It, it is very slow because we're doing a lot. But wanted to push you in this direction because this kind of software barely works. [laughing] [clapping]

81. 15:43

    Dang. I guess not. Um, also I clicked on it and now the LLM is trying to do something with me clicking on it. But anyway, um, this kinda software barely works today, and it doesn't mean it won't, won't work in the future, uh, but with exponential trends, like, things might just-- like this might just take off.

82. 16:03

    Um, so just wanted to push you all to think in this direction. Um, yeah. Will software, more software look like this? [upbeat music] I don't know. We'll see. Thank you. [clapping]