Building an Agentic Platform

Talk transcript

1. 0:00

   [on hold music] Hello.

2. 0:16

   Um, so I'm Ben Kus. I'm CTO of Box, and I'm going to talk today about our journey of, uh, through AI and in particular our AI agentic journey. Um, and, uh, if you don't know much about Box, uh, a little bit of background.

3. 0:31

   Um, so at Box, we are, um, a unstructured content platform. Uh, we've been around for a while, um, more than fifteen years and, um, our-- we very much concentrate on large enterprises.

4. 0:43

   So, uh, we've got, uh, over one hundred and fifteen thousand enterprise customers. We've got, uh, two-thirds of the Fortune five hundred and, um, our job really is to bring everything you'd want to do with your content to these customers and to provide them all the capabilities they might want.

5. 0:58

   In many cases, uh, for AI, many of these customers, their first AI deployment was actually with Box because, um, of course, many enterprises, uh, worry a lot about data concern-- security concerns and worry about data leakage with AI, make sure to do safe and secure AI, and this is one thing that we, uh, have specialized in over

6. 1:15

   time. Um, but the way that we think about, um, AI is at a platform level. So, um, we have sort of the historic version of Box, which, um, has the idea of the global infrastructure, sort of everything you need to manage and maintain content at scale.

7. 1:29

   We've got over an exabyte of data. We have an awful lot of, of, uh, hundreds of billions of, of files that our customers have trusted us with, um, and we have the natural way to protect them, in addition to the type of services that you provide when you're an unstructured data platform.

8. 1:41

   But then for the last few years, um, one of the key things we've been investing in has been in AI on top of the platform, and I'm here to tell you a bit about our journey here.

9. 1:51

   So, um, we started our journey in two thousand, uh, twenty-three, uh, w- shortly after, uh, AI became sort of production ready from a generative AI sense. And everything I'm talking about here today will be generative AI, of course.

10. 2:03

    So, um, we ended up with a set of features, things like QA across documents, things like being able to extract data, things like being able to do AI-powered workflows.

11. 2:11

    Happy to talk about these in general, but, um, today I'm gonna focus on one aspect of, uh, the features that we built, which is the idea of data extraction.

12. 2:20

    This is the idea of taking structured data from your unstructured data and, and using that in an enterprise setting. Um, and partly I'm going to, uh, focus on this one because this is interestingly like maybe the least agentic sort of, um, thing that you might think of when you're, uh, thinking about these other examples about how you

13. 2:37

    interact with AI. This is much less like a standard chatbot-style integration. But, uh, what we learned and what I'll tell you about is how you-- the concepts of agentic, uh, uh, capabilities applies well beyond just sort of, uh, e-end user interactions.

14. 2:54

    So, um, we'll be talking about data extraction for a moment. Just a quick background. When we talk about metadata or data, we talk about the things in unstructured data, be it documents, be it contracts, be it project proposals with anything, that then turns into structured data.

15. 3:09

    Uh, this is a very common challenge in, in enterprises, is that they have, like ninety percent of their data is unstructured, ten percent of their data is in databases, structured data.

16. 3:18

    Um, and, uh, and historically there has been this, this challenge that, like, it was kind of hard to, to utilize this. So many, many inter-- many customers have, for a very long time, wished they had better ways to automate their unstructured data.

17. 3:30

    And there's a lot of it, and it's really critical. In some cases, it's the most critical thing in an enterprise. So, um, uh, the things you do with it would be to like, uh, um, query your data, being able to kick off workflows, being able to do, um, just a, a better search and filtering across all of

18. 3:45

    your data. And so, so this, like, uh, the prototypical example, this is something like a contract where you have an authoritative unstructured piece of data, but then also, uh, the, the key fields in there are, are very important.

19. 3:57

    So, um, this is not a new thing. For many, many years, uh, the world, uh, for Box included, has been interested in pulling out unstructured-- uh, structured data from unstructured data.

20. 4:08

    And, um, there were a lot of techniques to do this, and there, there's a whole industry. If you ever heard of IDP, this is like a, a multibillion-dollar industry whose job in life was to do this kind of, of, of, uh, extraction.

21. 4:19

    But it, it was really hard. You had to build these specialized AI models. Uh, you had to, like, focus on specific types of content. You had to have this huge corpus of training data.

22. 4:28

    Oftentimes, you need to build-- get custom vendors or custom, uh, uh, ML models that you make, and it was quite brittle. And to, to the point, not a lot of companies ever thought about automating most of their most-- uh, their, their critical unstructured data.

23. 4:40

    So this was sort of the state of the industry for a very long time. Like, uh, just, um, don't bother trying too hard with unstructured data. Do everything you can to get it in, in some sort of structured format, but don't try to too, too hard to deal with unstructured data, until generative AI came along.

24. 4:56

    And so this is where our, our journey, uh, sort of begins with AI, uh, is for a long time, we've been using ML models at different, uh, in different ways.

25. 5:04

    And we, we-- it in-- and the first thing that we tried, um, when confronted with, uh, sort of a GPT-2, GPT-3 style of, of, of, uh, of AI models is that you just say, "Uh, I have a question for you, AI model.

26. 5:18

    We-- can you extract this kind of data?" And in the sa-- and, and as we mostly all know is, is, is, uh, AI is not only great at, um, generating, uh, uh, content, it's also great at understanding the nuances of content.

27. 5:30

    So this, uh, so what we did, we, we first started out with, uh, um, some, some, uh, a preprocessing, you know, doing sort of, um, OCR steps, classic ways to do this, um, and then being able to then say, "I want to extract these fields," standard AI calls, single shot or with some, some decoration of the, on

28. 5:49

    the prompts. Um, and this worked great. This was amazing. This was something where suddenly a standard generic off-the-shelf AI model from multiple vendors could outperform even the best sort of models that you had seen in the past.

29. 6:04

    Uh, and, uh, we supported multiple models just in case, and then it got better and better. This was wonderful. So this was flexible. You could do it across any kind of data.

30. 6:11

    You could-- It performed well. Um, it was, uh, uh, yes, you did have to do OCR and pre-process it, but that was straightforward. And so we were just thrilled.

31. 6:20

    This was like, uh, for us, it was like this is, this is, this is a new generation of, of, of AI. And, um, and interestingly, we would go to our customers and say, "We can do this across A data," and then they would give us some, and it would work.

32. 6:31

    And then we'd be like, "Great, AI models are awesome." Until they said, "Oh, now, uh, now that you do that well, and I, I get it, now what about this one?

33. 6:39

    What about this 300-page lease document with 300 fields? What about this really complex, uh, set of digital assets, and you want to give these really complex questions associated with it.

34. 6:48

    What about I wanna do not just extract data, I wanna do risk assessments and things that are these, like, more complex fields?" You start to realize, huh, like this-- As a human, when I-- if you ask me that question, I'm struggling to answer it, um, and then in the same way, the AI sort of struggled to, to

35. 7:01

    answer it. So, um, suddenly, uh, we ended up, uh, with, um, m-more complex documents.

36. 7:09

    Um, also, OCR is just a hard problem. Uh, like, like there's no seemingly, like, no end of, of, uh, heuristics and tricks that you do on, on OCR to get it right.

37. 7:18

    So I've got a scanned document. Somebody writes stuff in it. Somebody crosses stuff out. It's just hard. Um, and then, and then, um, for people who have dealt with, like, things like different file formats, PDFs, like, um, it, it, it's a challenge.

38. 7:30

    So whenever the OCR broke, it would just naturally give bad info to the AI. And then, um, languages were a big pain. Um, and, and so we started getting more and more challenges as we have an international set of customers across different use cases.

39. 7:42

    Um, also, there was a clear limit to the AI in terms of how much it could handle the, uh, attention to so many different fields. So if you say, "Here's 10 fields, here's a 10-page document.

40. 7:53

    Figure it out," they're great. They're-- Most of them are great. If you say, "Here's a 100-page document, and here's 100 fields that are each of them complex with separate instructions," then they-- it loses track.

41. 8:03

    And, and I have sympathy because people would lose track too. And so, um, this became very problematic because if you want high accuracy in an enterprise setting, like, you-- this just starts to not work.

42. 8:14

    Um, and then also just like, well, what is accuracy? What does it mean? In the old ML world, they give you confidence scores, that .865 is this one versus-- And then, of course, large language models don't really know their own accuracy.

43. 8:25

    So we would implement things like LLM-as-a-judge, and we'd come back and tell you, like, "Here's your extraction. Also, we're not quite sure this is right." And, and then our enterprise customers would kind of be like, "Well, that's helpful to know, but, like, I want it to work right, not just you tell me it doesn't work right."

44. 8:40

    And so this became this kind of set of challenges that, that, that, um, we, we, we focused on. And so customers were looking for speed. They were looking for affordability.

45. 8:47

    They were making this work. They were saying, "If AI is this future awesome thing, then like, you know, show it to me." And so-- And on these more complex documents.

46. 8:55

    So at this point, we kind of hit our, our despair moment. Um, our-- We thought LLMs were the solution to everything. We thought that, like, we could have these AI models that worked, but, um...

47. 9:05

    And we actually struggled, like, what do you do now? How do you fix this? And I know, let's just wait until, uh, the next, uh, Gemini model or, uh, you know, OpenAI seems to be on top of this, so, like, wait till the next one, which is part of it, right?

48. 9:16

    The models do get better. But, um, the fragility of the architecture was one that was, uh, we weren't really going to be able to solve on our own. So, um, naturally, uh, one of the answers, uh, that we were-- came up with was, um, bringing agentic approaches to everything that we do.

49. 9:35

    And this is really the, the-- one of the key things that, um, I want to sort of bring out in this session is that, um, it certainly was not obvious that the way to fix all these problems in something like data extraction was to do agentic style of interactions.

50. 9:48

    And when I say agentic, I mean an AI agent that does something like this, uh, amount of instructions, objectives, uh, uh, with the model background, uh, tools. We can make, have secure access.

51. 9:58

    Of course, it has memory from the purposes of, of advancing and being able to look up information inside of, of, of the system, but also with a, uh, full, uh, directed graph.

52. 10:09

    So the ability to orchestrate it to be able to do things like where you say, "Do this, then this, then this." Either it comes up with its own plan, or we actually can orchestrate it ourselves because we have knowledge of what we want it to do.

53. 10:18

    And this was, for us, um, I mean, it's controversial. Like, it was like our engineers were like, "What are you talking about? Like, uh, let's just make the OCR better.

54. 10:25

    Like, uh, like, let's just add another step somewhere. Like, let's just add a post-processing, uh, reg-regular expression checks." And then, and then, of course, everybody always like, "I have a way to do this, um, based on the old way of doing this.

55. 10:36

    Why don't we make, train an ML model? Like, why don't we fine-tune?" And then, and then, and, and so suddenly all of the genericness of it would be-- get lost in this process.

56. 10:44

    So, um, we came up with a mechanism which was a, uh... So this is, uh, think of like kind of LangGraph style of agentic, uh, capabilities. And, um, so we still-- We went, uh-- We still had the same inputs and outputs.

57. 10:59

    In document with fields, out answers. However, the approach was an agentic approach. And so, um, you know, we played with all the models, uh, reflecting, uh, back and forth and criticism, uh, being able to, uh, uh, separate it in multiple d- uh, tasks, uh, to be able to have different multi-agent systems work on this.

58. 11:18

    And we ended up with something like this, where you have a step where you prepare the fields, you go through, you group the fields. We learn quickly that, like, if you-- if there's, like, a set of fields that are like customers, uh, from a contract and then are like, uh, like parties, and then somewhere else, there's like

59. 11:30

    addresses of the parties, like you need the AI to handle those together. Otherwise, it's like you have three parties and two sets of addresses which don't ma-m- uh, match.

60. 11:39

    So we, so we had to break up intelligently the set of fields. We had to go through, and we had to, um, uh, like, uh, uh, do multiple queries on a document.

61. 11:47

    Then after we got that, we would then ha- use a set of tools to, to check and to double-check the results. In some cases, we use OCR. We would then double-check it by looking at pictures of the pages, um, and, and, and, and using multiple models.

62. 11:58

    Sometimes they vote, and they're like, "Wow," like, "this is a hard question." Three, three models from different vendors. Uh, you-- Two of them think this is the answer. That one's probably a good answer.

63. 12:06

    Um, and then on to the idea of the LLM-as-a-judge, not just a judge to tell you that this is a, um, this is the answer, but a judge to tell you, uh, "Hey, uh- Here's some feedback.

64. 12:17

    Keep trying. Now, of course, this takes a little bit longer, um, but, uh, this is something that then leads to the kind of accuracy that you'd want overall. And so for us, this was the, um, the, uh, uh, the architecture that then helped us solve a, a, a set of problems.

65. 12:32

    And it became, um, interesting because every time there was a new set of challenges, the answer was not rethink everything or let's then try like a whole new set of like, "Oh, you know, give us six months and, and we'll come with a new idea."

66. 12:44

    But, uh, I wonder if we change that prompt on that one node, or I wonder if we add another double-check at the end, then we can actually start to solve this problem.

67. 12:51

    So we bring the power of AI intelligence to help us then solve something that we used to think of as a standard function.

68. 12:58

    Um, and then not only that, it, it helped us in other ways. Like, so we-- we're naturally as an unstructured content store, like one of the first things you always see people, if I was like, "Can give you a demo right now," it's, "I have a bunch of, of documents.

69. 13:08

    I have a question." And, and we had the same thing, we had a judge and it would be like, it would tell us like, "Oh, that was a good answer or that wasn't."

70. 13:13

    And then why not just if it's not a good answer, we'll take another beat and, and tell the AI like, uh, try again. Before you tell the user this answer, like, I want you to, um, uh, like reflect on it for a second.

71. 13:25

    And this kind of thing just leads to higher accuracy and then it also leads to much more complexity. So we just announced our deep research capabilities on your content.

72. 13:33

    So in the same way that like OpenAI or Gemini does deep research on the internet, we let you do deep research on your data in Box. It would look something like this.

73. 13:40

    So this would be like roughly the, the directed graph that you'd have where you'd go through, you know, first we search for the data, kind of do that for a while, figure out what's relevant, double-check, then make an outline, kind of prepare a plan, go through, um, um, make, make a, a process.

74. 13:53

    And this is all agentic. Thank you. And it-- and, and this kind of thing wouldn't really be possible if we hadn't kind of laid the fra-- the framework of having an agentic foundation overall.

75. 14:03

    So, um, I will leave you with, uh, these, uh, a few lessons learned here. Um, so this is based on our time in the last two years. Um, the first is, uh, that, um, it, it wasn't obvious to us at first, but the agentic, uh, abstraction layer from an architecture perspective is actually quite clean.

76. 14:22

    It is, it is very, um-- Once you start to think this way, it is very natural to think, "I'm going to run an intelligent workflow, an intelligent directed graph powered by AI models in every step to be able to accomplish a task."

77. 14:33

    Not everything, but sometimes that's a great, that's a great approach. And this, and this is independent of some-- of a high scale set of, of, uh, sort of distributed system design.

78. 14:42

    And, and both are important. Like at some point you have to deal with, you know, a hundred million documents a day. At the sa-- other point you have to deal with that one.

79. 14:48

    And so be-being able to separate these two systems into like somebody who thinks about the agentic framework and somebody who thinks about the, the how to s-- uh, scale a generic process is this is, this is very helpful to keep these distinct.

80. 15:00

    Um, also it's just easy to evolve. Like, uh, in that deep research example, one of our biggest-- We, we, we did it and then it worked really well except for the output was kind of sloppy.

81. 15:08

    And so we were like, "Ah, I guess we gotta re-redesign the whole thing," or add another note at the end to say, "Summarize this and according to this," and it would just f-take that in and just redo the output.

82. 15:18

    Took not that long to fix. And this was something that was not obvious to me until later, which is that, um, if you're going to be using, um, a agentic, uh, uh, AI with a team who's been around for a while, like you start to need to get them to think about agentic first kind of thinking, AI

83. 15:34

    first thinking. And one way to do that is to, um, let them build something so that they can start to think, "Oh, like this is not only how we can build more things," but also because we're also a platform for our enterprise customers, they can think about how to make it better, make it better for them.

84. 15:47

    So things like, uh, really doubling down on the idea of, um, we, we publish MCP servers, what are the tools like for them? What can we do to make it easier?

85. 15:56

    How can we do our agent-to-agent communications, and so on.

86. 15:59

    So, um, this is, uh, all kind of summed up with is if you're confronted with a challenge, the lesson that we learned is that if it's plausible that an, a set of AI models, uh, could he-help you solve that problem, then you should build this AI agentic architecture early.

87. 16:16

    If I go back in time, I would wish to have done this sooner because then we'd kind of be, uh, have been able to continue to take advantage of that.

88. 16:22

    Um, and so that's my, uh, that's my journey and that's my, my, my lessons for you.

89. 16:27

    Uh, so thank you. [clapping] Uh, Ankur, are we, um...

90. 16:36

    Two minutes? Okay. So, um, if anybody-- What?

91. 16:39

    Yeah.

92. 16:39

    Two questions. Okay. If anybody has any questions, I'm happy to answer them.

93. 16:45

    Is this available as API?

94. 16:45

    Uh, question being is this available as API? Yes. Um, so we're very API-first oriented, so we have an agent API that you can call upon these agents to do things and give them the arguments.

95. 16:53

    So yes, uh, we, we, we provide, uh, uh, the, the agent, uh, just APIs across everything and tools, um, to, to call our, our APIs.

96. 17:06

    Um.

97. 17:07

    One question.

98. 17:09

    Okay.

99. 17:10

    Where you primarily just evaluating your agents with LLM as a judge or are there other ways makes when you start using a more manual approach as well?

100. 17:19

     Um, in terms of evaluating our agents, uh, and how do we do that? Um, so we, we not only use, uh, LLM as a judge, but we also create an eval set.

101. 17:25

     So we have our standard set of eval sets. Um, and then we've learned that, um, since the AI gets so, so good over time, we created a challenge set of, of eval sets too, so that we can better explore like things that not everybody asks, but if they did, it would be really hard.

102. 17:36

     And then that way you can better decide on whether or not you're not only prepared for now, but as people get more challenging things, we, we, we know that we can grow across that.

103. 17:44

     So a mixture of eval sets plus LLM as a judge, plus the idea of just having people give feedback. We, we have limited ability to look as an enterprise company what happening, but the, the, the idea of them telling us this is still useful in all cases.

104. 18:00

     Ben.

105. 18:01

     You can yell if you want. I'll hear you.

106. 18:03

     Uh, well, so, uh, AI I guess the first time you talk, so apologies if I-

107. 18:07

     Yeah.

108. 18:07

     -ask a question before you would have covered that up. Seems like you're mostly building agents like these put together across either certified tuning or any of those approaches.

109. 18:16

     Uh, so the question being why bother with agents if you can fine-tune a model? Um-

110. 18:20

     No, no, I'm saying- Have you tried, have you tried fine-tuning-

111. 18:23

     Yeah. We're-

112. 18:23

     -uh, agents?

113. 18:24

     We're, um, we're pretty anti-fine-tuning at this moment because, um, of the challenges of once you fine-tune something, you have to then fine-tune all of the evolutions of them going forward.

114. 18:35

     We support mult- every-- m-multiple models, Gemini, Llama, uh, OpenAI, Anthropic, and it's just hard to consistently fine-tune across the board in ways that like not on-- And usually just the next version of the model gets better.

115. 18:46

     So we've, we, we've gotten to the point where we use these prompts or cache prompts or agenticness as opposed to fine-tuning. That's the approach for our particular use cases.

116. 18:53

     It works quite well. Okay. Thank you everyone. [upbeat music]