Your Finance Agent's Bottleneck Is You

Talk transcript

1. 0:00

   [on-hold jingle] Hello, everyone. Welcome to this session about your fin-finance agent's bottleneck is you.

2. 0:19

   So, sorry for the rude title. I don't mean to call the audience here the bottlenecks, but I'm here to talk about the harnesses that you guys are developing and using these internal harnesses to build your production agents.

3. 0:34

   So, my name is Siddanth Emani, and I'm a data scientist at Auditoria AI, and we build fi- production agents for finance. So if you're a CFO in the audience, I would love to speak to you after the session.

4. 0:50

   This talk is in between the harness engineering track and AI for finance. So this talk is mostly about identifying the bottlenecks within your developer harnesses. And if you're a developer yourself, how do you be ten-X productive with the agent harnesses that you're using?

5. 1:10

   So all of us have seen, you know, beautiful demos in this AI Engineer World's Fair. But once these demos are promoted to pilots and you start onboarding new customers, the agents has never seen these future data.

6. 1:27

   So all of us know production bugs are very high, and production guards bill by the hour, so that's a hard fact. And [chuckles] writing code is very easy, so shipping beautiful demos and showing it to a lot of people is very easy r- nowadays.

7. 1:49

   So what is the problem, and why do these demos fail in production? Is it the model? Do you need a better model? Fable Five perhaps? Or do you need faster GPUs?

8. 2:02

   Or do you need a better framework maybe? Or your RLHF loops are not working properly? So what is the answer? If we wait three and a half months, we are awarded with a new model in the market, so we can easily swap models.

9. 2:20

   If we wait perhaps one year, we have new chips, we have faster GPUs. And again, writing code is easy, so we have new fram- frameworks every day. So you can swap your framework every now and then.

10. 2:35

    So how do we, in real time, fix these production bugs? The answer is your develop velocity. The model capability increases very exponentially, and the developers have to spend a lot of time every day to automate your developer loop.

11. 2:55

    So I'm talking about four primitives here. All of you need to think about loops, and at the end of the session, I hope you can ten-X your production code.

12. 3:07

    So first we have subagents. Nowadays, whatever harness you're using, you can spawn new subagents. You can have a, you can have an army of them.

13. 3:17

    And Git worktrees are your best friend. So think of worktrees as isolated folders, and inside these folders, the agent writes whatever code it's generating. So you want these worktrees to be in parallel, so the subagents are doing independent tasks and are not fighting over the same thing.

14. 3:39

    Second, we have skills. These are your organization's secret recipes. So make sure you have a lot of skills because these skills, once you start say, uh, giving it to your agents, the agents will always make sure to use the correct and proper workflows to solve whatever production bug you're facing.

15. 4:00

    And of course, all of us have seen a lot of MCP tools being shipped into the market right now. Everybody says we can-- y- the agent can connect to whatever MCP tool and whatever third-party server there is, and your client data can live in any system you want.

16. 4:17

    And at the end of the day, if you have a lot of subagents, you have a lot of work to orchestrate, so minimal UX is the key here.

17. 4:29

    Let's look at the subagents. With you as the orchestrator, you can have, let's say with forty-eight GB of RAM on your MacBook, you can have fifty active worktrees. That is fifty active subagents working diff- independently on different tasks.

18. 4:43

    So where do these tasks come from? So let's say the production software you're going to ship has a lot of bugs that your QA is reporting. So all the Jira tickets can be thought of in a separate different worktree.

19. 4:57

    So different worktrees are h- handled by a separate agent, and these agents can spawn multiple subagents to solve that particular task. You don't want to queue up your tasks because the agent is-- will do that a lot better than you.

20. 5:15

    Let's look at, you know, um, an example harness. What if the QA reports a lot of bug tickets, and somehow, magically, there is an agent which parses the requirements, does a root cause analysis, pulls all the traces, pulls all the logs,

21. 5:35

    s- puts all this in a separate worktree,

22. 5:38

    does the TDD, r- does the-- implements the fix. Because it's in your local system, you have to do test scripts, local end-to-end testing. You create a PR. You submit the PR to your team for review.

23. 5:53

    And after review, you merge it into your master branch, let's say. After merging it, obviously you have to build a Docker image, deploy it into your development environment, test it.

24. 6:05

    Again, ship build an, an image to your stage environment, test it, deploy it to stage, and then you go back to the QA saying, "Here you go. You can test it now."

25. 6:17

    So I would like to ask a question in the audience, um,

26. 6:22

    at what points do you think the human contact is required in this, um, steps one to nine?

27. 6:30

    So I would say the human is only required at steps one and nine because the in between steps, the agent can do a lot better work.

28. 6:40

    There needs to be a human to see what work the agent is doing, and there needs to be a human at the end to validate after the work is being shipped to stage.

29. 6:52

    And obviously we need minimal UX because humans love minimal UX. So in the image if, um, you can s- if you squint your eyes and see, the image shows, um, the production agent software that you're building, the project dashboards which shows all your Kubernetes services, pods, examples, all the logs, system logs, all your Jira tickets, all your

30. 7:17

    GitHub PRs, and maybe a Claude Code session at the bottom. So this is basically a macOS widget, and you don't need to open multiple windows to do all of this work.

31. 7:30

    A developer does like variety of things in their software dev lifecycle, so you can use just this one widget to do a lot of things. So you can see from the graph also the number of neck rotations to ship one change like reduces a lot drastically, and I imagine all of you have like two to three monitors

32. 7:48

    on your table, and you just keep rotating your neck mo- orchestrating these agents.

33. 7:56

    So Auditoria works in finance, so there's a lot of regulation and policies happening in finance right now. So what does it look like for orchestrating a team of subagents in the finance sector?

34. 8:10

    If we take AI out of the picture, usually what happens is you have a human auditor which reviews the code, and you have a controller which signs off under your SOX compliance.

35. 8:21

    And reviewing agent to agent, it, it doesn't, uh-- where do you keep the accountability? If something goes wrong in production, you can't say Claude is doing this. Something is wrong.

36. 8:34

    So but let's say you have all these subagents, and you're using these harnesses to sh- fix bugs in real time. What is the bottleneck? It becomes a human attention because you yourself have to orchestrate all these different tasks.

37. 8:52

    And moving fast and breaking things in sector, in the finance sector is a lot different.

38. 9:00

    So let's look at part two, which is removing yourself from the loop. Till now I've been saying a human is required to see what the agent is doing, and at the end also to validate what the agent has done.

39. 9:14

    But with the self-improvement of the agent and model capabilities these days, we get Mi- Fable five and Mythos five and f- GPT 5.6 also. So what does it look like when you have this recursive sel- self-improvement in your internal developer, developer harnesses?

40. 9:35

    So all your production failures become input. So let's say you automate, keep automating these sel- developer harnesses every day, and you ask the agent to upgrade itself, uh, essentially.

41. 9:51

    So you do a task, you let the loop run, let's say one or two days. You solve five to six bug tickets, and you just tell the agent to analyze all the bottlenecks in this process, make a list of them, and somehow slowly keep removing these bottlenecks every day.

42. 10:10

    At the end of one month, let's say, you have a really nice self-automated loop where you just type in one sentence and just say, "Fix this bug for me."

43. 10:21

    And the agent goes off, connects to all your database systems, fetches all the logs, traces tickets, and ships it, and migrates it, migrates it to the Jira to QA pipeline.

44. 10:33

    And you can just book a vacation maybe or work from home.

45. 10:40

    And what does it look like internally, and what happens when you steer less and ship more? Nowadays, how many of you know you can give goals to your agents?

46. 10:52

    You can just set a goal and forget about it.

47. 10:56

    Anybody? Nice. Um, so what if you combine goals and loops? You can just set a goal saying there is some data discrepancy in this report, and in the production bug, like the source data is not matching with what the agent has generated.

48. 11:16

    So you can just set a goal to fix this, look into this, set a loop. You can even close li- close your laptop because you can do it from your phone nowadays.

49. 11:26

    And if you look at the last but one point, which is dreaming,

50. 11:30

    um, let's say a lot of customers are using your production software, and they're doing the same type of patterns and t- they're facing the same type of problems. So you let the agent dream like humans dream in the background so that it collects all the sessions that your customers are using and compacts it into a set of

51. 11:52

    data points which your system can use and basically upgrade yourself.

52. 11:57

    So with a combination of all these features, basically you can essentially remove yourself out of the loop. But as I said before, the developers do a lot of variety things in their software development life cycle, and sitting behind a desk from nine to five and just writing code is not valid anymore.

53. 12:22

    So just an overview of what I've covered till now in this session. You can have a team of subagents working in parallel work trees. You can have skills, your organizational secret recipes, your customers' recipes.

54. 12:36

    You can give all of these to an agent. Your agent can connect to whatever third-party server there is. It can be a logging system. It can be an authentication gateway.

55. 12:47

    And you just compress all of this into one pane of glass because minimal UX is the key.

56. 12:53

    And you can set goals and loops for autonomy. If you think this particular work can be done by the agent a lot better, you can just ship it to the agent.

57. 13:04

    Always have the human as a verifier, but not the throughput ceiling because human attention is very limited.

58. 13:13

    So thank you for your time, and thank you for your-- thank you for... I hope you, um, learned something from this session. Thank you. [audience applauding] [outro music]