Agents in Production: How OpenGov Built and Scaled OG Assist

Talk transcript

1. 0:01

   Hi, everyone. My name is Gabe De Mesa. I'm an engineer here at OpenGov, and today we're gonna be talking about agents in production, specifically how OpenGov built and scaled OG Assist.

2. 0:13

   Uh, so, um, this presentation is going to be jam-packed with just so much good stuff. Uh, we're gonna talk about, uh, AI agents. We're going to talk about our harness.

3. 0:25

   We're going to talk about, um, evals, observability, traces. We're going to talk about, um, tools and skills. Um, it's-- there's gonna be a lot of good stuff in here.

4. 0:38

   We're gonna talk to you guys about, uh, what we do at OpenGov and how we operate at the scale that, uh, we operate at, um, in production. So you'll be able to see a real use case and workload, uh, with AI agents.

5. 0:52

   Um, so without further ado, let's get started.

6. 0:56

   Okay, agenda. So just really quickly going to go through, uh, high level, what we're going to talk about today. Uh, I'm gonna tell you guys a little bit about OG Assist and what, uh, OpenGov is.

7. 1:07

   I'm gonna tell you guys the origin story of how this all kinda came to be. Uh, we're going to talk about OG Assist's, uh, big bet on Effect, uh, a little bit into our core agent loop.

8. 1:20

   Uh, we're gonna talk about the A2A protocol, evals, and sandboxing. We're gonna talk about how we manage long context. We're gonna talk about, um, monitoring, observability, how we collect feedback, uh, and how we iterate on that feedback.

9. 1:35

   We're gonna lastly, uh, also talk about tools and skills and how at OpenGov, uh, we use, um, AI not only externally, uh, that we, uh, serve to customers, but also internally to improve our development workflows.

10. 1:51

    Just a little bit about me before we go any further. My name is Gabe. I'm a software engineer here at OpenGov. I work on the AI agents team, and, uh, I'm one of the folks that helped build, uh, OG Assist and some of the systems that you guys will be seeing today.

11. 2:07

    So a little bit about OpenGov. OpenGov is a software company, uh, on a mission to power more effective and accountable government. Um, so OpenGov sells ERP software. That's things like budgeting, procurement, asset management, and permitting.

12. 2:22

    And, um, we were founded about fourteen years ago, and what's cool is, um, we have this thing called OG Assist, and OG Assist is this little button on the top of all of our products in the, in the navigation bar.

13. 2:38

    And what's cool is, um, all of our product suites and product teams, um, have built sc-- tools and skills in order to power this button. So for example, if I open up, uh, this, this, um...

14. 2:54

    If I click this button and I open up OG Assist, it says, "Hey, um, I'm gonna ask about rate codes," which is very specific to utility billing, the current product that I'm in.

15. 3:03

    And you can see that inside of this kinda chat interface, I'm able to speak to an agent, and the agent is able to make tool calls in order to, um, look up information against data inside of that suite.

16. 3:15

    So it's really cool, um, to be able to kinda first party create these experiences, uh, through the capability that we've built called OG Assist.

17. 3:27

    Okay, so just a quick story about how this all came to be. So, um, a little while back, we, we, we saw that AI was really starting to take off, and a principal, uh, spun up this new team called the AI agents team and asked me to join.

18. 3:42

    And, um, instantly I said yes, and OG Assist started to, to grow, and we started to integrate, uh, OG Assist into all our products, and, uh, not only our back-end capabilities, but also our front-end capabilities as well.

19. 3:56

    So you'll see that one of the capabilities that we give the agent is it's able to, um, see what's on the screen and, and see and, and, and take action, uh, on what's on the page.

20. 4:08

    So you could see that, um, I'm asking the agent here, "Hey, hey, what's on the screen? Can you maybe highlight, uh, some of the next steps that I could take?"

21. 4:16

    So you can see that the agent here is thinking. It's saying, "Okay, what tools do I have available to use?" And, "Hey, let me go and highlight something that you could actually click on and, and tell you more about it."

22. 4:27

    So just another capability of OG Assist and just a little sh-short story about how this all came to be.

23. 4:35

    So the big bet on Effect. Um, so I really wanted to include this slide because, um, here on the agents team, we made a huge bet to, um, to, to bet on Effect.

24. 4:48

    And suffice to say, it's paid off in dividends. Um, we write Effect. So Effect is this library for TypeScript. It's open source, and it helps you write better, um, TypeScript code.

25. 5:01

    Uh, you know, it's got a lot of, uh, stuff baked in it, like a sc- a, a schema similar to like Zod, if you've ever used that. It's also got, um, things for error handling, uh, for logging, for traces, for, uh...

26. 5:16

    It's just got so much in there. It really helps write better code and structure your code better and, uh, helps with architecture, spinning up new services for, uh, and, and for us on the agents team, really helping, uh, design and build the, the, the core agent loop.

27. 5:31

    So you'll see throughout this presentation sprinkled in, um, how Effect on our team, uh, has paid off in dividends. So we, we really love Effect here at OpenGov, and we encourage other folks to try it out and, um, yeah.

28. 5:47

    Let's keep going. The Effect native loop. So originally, we were on LangGraph, and that was fine until the team really started to scale, uh, and our use cases started to evolve.

29. 6:02

    So we decided to move over to our own kind of Effect native agent loop to have full regency over this, uh, agent loop, such that if we have complex use cases or features that we need to build, we could kind of get in-- We, we had full control of the, of the agent loop.

30. 6:23

    And not only that, but now we're fully on Effect. So all the cool things you get with Effect is now p-propagated throughout the entire agent loop, like the tracing, structured con-concurrency, the logging.

31. 6:34

    Everything is more fine-grained control, and it, it really allows us to really unlock the full potential, uh, having our own agent loop from the ground up. Um, so another thing I wanted to mention is on the left side, you'll see a code example.

32. 6:51

    This is really the basics of the Effect loop that we're using. Uh, we're using this thing called the Effect AI package, and in that package, there's this thing called, um, there's a chat and a language model.

33. 7:05

    So with the chat, you can instantiate like an, a chat, for example, and then you could stream text using, um, that, that kinda stream text function. You could pass in a prompt, and what's cool is, uh, with a language model under the hood of...

34. 7:22

    Since we're kinda doing dependency injection, we could pass in a different language model if we were to, uh, s- hot swap to another one, for example. So really just having full control of our own agent loop just kinda gives us all the levers, and it really just unlocks the full capabilities of the model and, uh, for the

35. 7:41

    team as well to have full agency over this, this loop. Another thing I wanted to mention is the Agent2Agent protocol. So here on the agents team, we've had a lot of success with this protocol.

36. 7:54

    So this protocol being the protocol that Google created, um, kind of an open protocol for agents to intercommunicate. But, um, we found this very useful for, uh, defining our agent routes, like for example, i-in the back end and our model and our schema to follow this kind of, uh, agent protocol.

37. 8:15

    So we modeled-- So for example, there's this thing called an agent card, which you see here, and it's got the name of the agent, a description, et cetera, right?

38. 8:24

    And having this kind of rigorous protocol, this rigorous spec, really helped drive our development and drive alignment because, you know, all we had to do was, um, align with this spec and follow this spec, and we knew that this was kind of the contract that our front end and back end would both consume and, and produce.

39. 8:48

    So, um, this, uh, I would say also has been, uh, very helpful for us. And, and what's really cool is A2A has a lot of extensions, right? So you could extend the protocol, uh, add in like metadata.

40. 9:02

    Uh, there's also A2UI. Um, so lots of fun stuff, uh, with A2A protocol, but, uh, this is kind of what's worked for us. So just sharing that with, with you folks. [lip smack]

41. 9:16

    Feedback and evals. So here the quote is, "Shipping is the start, not the finish." So what we do here, uh, on the agents team is we have kinda multiple ways we do evals and collect feedback.

42. 9:29

    Um, obviously, you know, we'll have folks, uh, call in or, or email us or, or just let us know and tell us. But the main way is we have this thumbs up and thumbs down mechanism, and here, uh, someone is able to tell us, "Hey, this, this worked really well.

43. 9:44

    This was a great response," or, "That wasn't a great response." And that signal we take and we're able to iterate on, uh, and we can take it back and help improve, uh, you know, the response in the future.

44. 9:55

    Um, we also have automated evals. So in, in the, in RCI, we, we have evals that run against real completions, so we could test a prompt against, "Hey, did it hit some tools?

45. 10:07

    Did it do what it's supposed to do?" And that also helps with our accuracy. So, uh, those automated evals in conjunction with collecting feedback really help us, um, improve our, uh, our, our tools, our skills, um, our harness, and, and that's really how, how we're able to iterate so fast and so quickly.

46. 10:31

    Humans in the loop. So this is a really cool feature we built where we deterministically interrupt the agent loop if there is a tool call approval required. So if an agent tries to make a tool call that it needs human approval for, it'll show this UI, and the human, uh, can click accept or reject, so explicitly rejecting

47. 10:51

    or explicitly accepting, uh, the action that the agent is trying to make. And this ensures that, uh, you know, we're building trust and also ensuring that, uh, you know, we're being safe, especially when the agent's trying to do a mutating operation and always, always, always making sure that, um, humans are in the driver's seat.

48. 11:13

    Sandboxing. So another thing that we, uh, worked on, um, kinda similar to the safety slide we just saw was, um, whenever an agent tries to execute code or tries to create files, it does so in a sandbox.

49. 11:29

    So we gave our agents sandboxes such that it could spin up these sandboxes on demand, and it could use those sandboxes to honestly write code, execute code, create files, and it's kinda this safe, ephemeral, isolated space such that the agent can, can, can take action in there and not-- and we don't have to worry about, um, any

50. 11:52

    risk to, you know, our, our production systems. Um, and it's, it's really cool 'cause they also get, uh, tied, uh, teared down at the end. So, um-

51. 12:03

    In this example, I said, "Hey, create a PDF, uh, for the folks of the AI Engineer Conference 2026, and, um, allow me to download it so I can share it with them."

52. 12:12

    And you can see that the agent created this really cool PDF inside of that sandbox. So just really wanted to cover this sandbox feature and just give you a brief kind of overview of, of sandboxing.

53. 12:27

    Long context. So, um, inside of OG Assist, we have hit many hurdles, like with, especially with legacy models now, like with, uh, token limits or just way too much, just completely overloaded with context, especially as conversations get longer.

54. 12:49

    So we found that having some sort of, um, rolling summarization was more effective than, you know, always stuffing in the latest and most recent m- uh, messages. Uh, rather just, you know, give like a running summary after N number of messages and, uh, maybe you only want the, like N minus five

55. 13:14

    most recent messages or N minus 10 most recent messages, right? Um, and it may be that you're only talking about a specific topic now, but you may want to refer to, uh, context earlier, like 100 messages above, then, um, it, uh, that's where kind of the, the memory component comes in, 'cause when you have this rolling summary

56. 13:35

    of a, a really long conversation, then you could do recall over that s- uh, summarization and, um, you know, if you ask the agent, "Hey, remember that thing that we talked about?"

57. 13:46

    Then the agent within the thread will, uh, be like, "Yeah, I do know what you were talking about. I have kind of this short little tidbit," and it can, you know, follow up and, and, and do more kind of with that, uh, uh, summary, rolling summary in mind.

58. 13:58

    So that's kinda how we handled long context and memory, and it's worked pretty well for us. So, uh, just wanted to share a little bit about that and how we've, uh, solved the long context problem.

59. 14:11

    UI on the fly. So, um, in this example, I said to the agent, "Hey, generate me a long essay, but give me some examples about what the essay could be about."

60. 14:20

    So what's really cool is the agent had this primitive registered of, uh, this form, and it was able to build out this form for me at runtime and give me some options of what I could choose from.

61. 14:32

    So it feels very personal, and it feels very kinda in the moment that it's able to, to give me these options just at runtime. So, um, this is kinda just a short little, um, thing I wanted to include here about generative UI and how we, uh, are able to render UIs on the fly.

62. 14:53

    You can't scale what you can't see. So, um, this, uh, this kind of section is about tracing, uh, and observability. Really, what's cool about Effect is you kinda get tracing out of the box.

63. 15:10

    Um, you know, when you use these Effect functions, they all get kinda tagged automatically with like these spans, and kinda the span gets picked up and feeds into these traces so that you can kinda get these kinda drill downs of these function calls.

64. 15:26

    So here is an, an example of a trace from the Effect team. Um, I have it linked. Uh, you could see that like, hey, when you hit this API, it goes to this endpoint, to this handler, and, uh, you know, et cetera, and it takes...

65. 15:41

    And what, what's really cool is you could profile all your traces, right? So this takes a total of this many seconds, and you can see where the bottleneck is here.

66. 15:49

    If there's a failure, you can cross-reference it across services. So really, really important, especially working in agentic systems where we're, we're integrating with other teams and other APIs and other, um, other platform capabilities.

67. 16:06

    So, uh, what's cool with, um, Effect is you get all this tracing out of the box, and it really makes building this agentic experience, debugging it, and maintaining it just a breeze.

68. 16:19

    Tools and skills. So, uh, not only did we make a big bet on Effect, but we also made a big bet on tools and skills. So we believe that tools and skills are really all you need.

69. 16:31

    And in this case, you could see on the left, we have this tool called Get Dad Joke, and this is kind of the Effect way and kind of the building blocks of how we do things here at OpenGov.

70. 16:43

    But, you know, this is pulled from the O-Effect website, but you could see, hey, this is how you make a tool, and then you add it to a toolkit, which is a collection of tools, and then you can register this, this toolkit with the, with the, uh, language model.

71. 16:58

    So for example, if you had a prompt that said, "Hey, generate some dad jokes about pirates." Well, guess what? The agent has a tool, uh, that can help get, uh, a dad joke.

72. 17:11

    So, um, really this is the, the building blocks of how we did tools, uh, and eventually skills, and really just is, has paid off wonderfully for, for our organization.

73. 17:25

    So, um, really recommend trying out this Effect AI package from, uh, Effect and, um, just trying out building out your own tools and skills.

74. 17:38

    Developer velocity. So not only do we build agents for our customers, but we also use agents, um, here internally, uh, in OpenGov. So we, uh, use a lot of Claude and Cursor, uh, and it's just really been a game changer for our team.

75. 17:55

    Um, it's funny 'cause we're building tools and skills for customer-facing, uh, agents and, and that has been great, but we're also building them internally as well to help accelerate our development workflows.

76. 18:05

    So things like Claude, Cursor, Claude Agents, they really help accelerate how we read, write, review code, and, and ship. Um, so it's, it's just been such an accelerant, so definitely just wanted to mention that, uh, before we wrap up.

77. 18:23

    That's it. Thanks so much for watching. You've made it to the end. Let's build agents that ship to production.