AI Cut Our Research Time From 7 Days to 15 Minutes — Gareth Yoder, Vanguard

My name is Gareth Yoder. I'm the cash program technical lead here at Vanguard. I help oversee four to six teams in the cash program. This is No Cash Left Behind, a Vanguard modernization journey. This journey really starts after we delivered our Cash Plus offering to our Vanguard brokerage account clients, and we launched that in twenty twenty-two. And after we did the launch, we started to get the whispers that modernization

is now coming down the pipeline, and that will be our next major project. How many here have legacy code bases? Right? That's a lot of you. How many of you wish that legacy code base would just modernize? Right? Many of you would say the same. So looking here at the journey, we're gonna go through this timeline in today's presentation, but I wanna first talk to you what a cash ledger actually is. A cash ledger involves all money

movement transactions into our Vanguard brokerage accounts from the outside world. It also involves cash being moved between accounts. And so the payment rails, think ACH, wires, and more. We support the web channels, mobile channels, and batch channels. Our system runs in an active-active multi-region fault-tolerant ecosystem, and our services will receive upwards to thirty million

calls per day. Our clients will generate a half million ledgers every day as well. And so as we start to go down this modernization approach, we broke it down into three pillars: a data migration, a producer migration, and a consumer migration. But because this was an enterprise priority, we had to do all three in parallel. So now, as we look

towards the research analysis, remember we're at the beginning of our timeline, the AI tooling wasn't there. And so oftentimes we would work with mainframe engineers, and we would work with architects to try to determine what that data schema model would look like. And there was a turning point in this modernization journey, and that's when we had our legacy code base being available to our AI tooling via an MCP server. And now we saw faster

feedback loops. What was taking seven days for a mainframe engineer to go into the code, do their analysis, and then kinda, and come back to us for an answer now could all be done via an AI tool prompt. And what we saw was that the feedback loop was reduced from seven days down to fifteen minutes. It was a huge win for our product owners, our technical

leads, and developers alike. They all contribute this tool to helping our modernization succeed. So let's talk about data modernization. We utilized a standard CDC change data capture approach that's common in industry, and we had to then scan both databases. A team did daily audits. We first made sure we had strong consistency by verifying the row-level data counts,

and then we ensured that we had strong integrity by checking the individual columns, making sure that the amounts matched from the legacy to the modern, making sure that the account information would match, or the state withholding, the federal withholding. All of the attributes in the legacy database had to match to the, to the cloud. A major tip here is that you don't wanna over-engineer this solution because as modernization would complete, this code would then get deprecated.

Now, let's talk about producer migration. Our producers called our infrastructure in three ways from the legacy state. One was through a service request. The second was an API that wrapped that service request. And the last, and probably the most daunting, was direct table inserts because we would see data appear in our database, and we weren't sure how it got there or who produced that data. So we took that common data schema, and we

put it onto a gateway that wrapped our APIs that would do our cloud data inserts. Now, take your modernization journey as an opportunity to reimagine your data ecosystem. We took concepts from Fintech, such as idempotency. We also ensured that we had strong consistency across multiple writers. And so we used, utilized a similar paradigm to what a Snowflake ID would provide us. We reimagined, and we didn't do a lift and

shift. Now, we have a modernized producer offering available. How do we get our producers? We had thirty-five producers that had to then modernize to our ecosystem. Early on, we had to produce manual documentation to facilitate this transition. And as the AI tooling became available, we provided that documentation in our company's wiki but then exposed it via the MCP server.

And that was really great because what it did was we used to have many teams having chats with us, and we had to try to keep all of those organized. Now that communication and that back and forth could all be done via the prompt on the producer side, and we saw that the number of chats and, and producer support that our team was doing was greatly reduced when we had the MCP available. The AI tools also did a really good job of transitioning

state from the legacy ecosystem to the cloud. We made our cloud ecosystem, the ledger enums, to be human-readable versus just ledger status codes in our legacy ecosystem. And so our AI tooling did a really good job at bridging the gap of the intent from the legacy ecosystem to the cloud.

So now let's look at the consumer migration path. We had three ways to access consumer-level data. We had real-time use cases, batch use cases, and streaming use cases. And the commonality here is that we provided a schema registry no matter which channel that your, uh, consumer would access that data. And so for a funds available, that's our most, that's our most, uh,

heavy traffic, uh, real-time API, and that goes into determining what is the balance that's displayed on your account. They're gonna be calling us via GraphQL, and they're gonna subscribe to only the attributes that meet their use cases. Now, for batch use cases, think of reconciliation or data analytics. We have our data warehouse that will expose data share views to each individual consumer. Each of the consumers tells us

their use case, and we provide a, a view that's specific to their data needs. This, this was supported by our data governance team that strongly advocated that only the consumers that need the certain data attributes, they should only see, uh, those pieces of data. Our last piece was around streaming. We have a Kafka event topic that we would stream all of the inserts and updates to that database. Our fraud teams were able to take,

take advantage of this technology to provide real-time data ingestion into their fraud tooling. We provided at-least-once delivery, and so our concepts of idempotency that we did through the producer side, we carried forward through the event streaming. And so as they subscribe to at-least-once delivery, we made sure that they had idempotent safe, uh, operations so that if a batch job would,

would fail halfway through the process, those teams could retry, and we can ensure that we don't have, uh, duplicate ledgers being created. But the at-least-once delivery from the eventing side ensured that they had all of the ledger events that were created.

Now, when we onboarded consumers, we had fifty different consumers across our organization that needed our data. At Vanguard, when you onboard an application, you need to have a three-character application code to register to onboard. This application code describes what your application is and h- and what it performs. But we took it a step further. We requested that our consumers would provide us

a description and a bu- business case justification for their need for the data. And so think about if I am a trade order that needs a cash ledger to fill-fulfill the cash obligation of the trade, that could be a business use case. Or for a retirement account that needs to do a, a required mandatory distribution. A 529 account could do a contribution to their college tuition account.

By grouping together all of these business descriptions, now the AI tooling can compile that together and provide an accurate description of what use cases have been modernized in our ecosystem, as well as we now have what is the top business functionality that we support. We can also take it a step further by asking, "What is the impact if our SLAs are breached?" And this is critical because if you're in a

major incident, and if we see an app code that starts to go beno- go below the five nines of availability, we can go in and we can see, okay, a batch application, we may have hours to respond. A real-time API where there's a client on the other side that's receiving errors, we have minutes, and time is of the essence to self-heal. We're continuing to evaluate how we can get

faster response times by combining the business use case justification into our observability and, and, uh, log traces. So what we've learned. We've learned that although we started this journey before the AI tooling was even available, we adapted and continued to adopt the newer technologies. This greatly aided in our research, in our

development, and our testing. Today, in our cash program, ninety percent of our engineers utilize these AI tools on a daily basis. Our teams use multi-agent workflows, and we have shared knowledge bases. We're continuing to evolve this AI harness to meet our modernization demand. We have the concept of a team brain, that as engineers are, are building out product features,

that knowledge goes into a common storage where it's now accessible by all members on the team. And if there's an engineer that has just joined, they can tap into this common knowledge base And now they have the same context in their prompt as an engineer that has been here for the last five years.

Now, when you're in a modernization process, maintaining two code bases is challenging, and what you'll want to do is you'll want to continue to fix forward. Don't fix the past. As you encounter bugs that have-- may have listed-- have, uh, existed for decades, continue to fix the modernized state and allow that to be the incentive for your producers and

your consumers to modernize to the cloud. Now, this will throw off your audit logs, so you'll wanna have whitelisting available, and that's really critical. And the last piece is ensure that you have the right metrics, alarms, and traceability because you're gonna review those often as your product continues to grow. Now, because this is a leadership track, I wanna call out five main points. Number one was make your knowled-- or make your

legacy codebase AI queryable. That-- Exposing that via an MCP server or legacy codebase, it cut that research cycle time and gave quicker feedback loops. That's seven days down to fifteen minutes. That was a huge win. Treat your migration docs as part of your product. It needs to... Your migration docs are just as critical as your business functionality because that's what all of your producers are going to utilize to

onboard to your application. Eventually, yes, those will get deprecated out, but for that interim process, those migration docs are critical. Reimagine and don't lift and shift is, is the third point. The consumers and producers that we had the most trouble onboarding to our modernized state were the ones that wanted to do a direct lift and shift transfer of old to new versus those

that took a moment to step back and say, "How could we better evolve our product by taking advantage of the modern capabilities that the cash ledger ecosystem provides?"

The fourth component is adopt AI incrementally.

As the AI tools arrived, you continue to enhance your product. There's never going to be a perfect starting point in the world that we live in today. The important thing is that you get started and you continue to enhance and adapt as the technology changes. Last piece is observability is non-negotiable. You need to know who's producing your data and who's consuming your data, and this is ever more important as your product goes to

BAU, and once the modernization is complete, your teams will now have to support this as their, uh, daily operations.

Thank you very much, all. If we have time, we'll take questions. Otherwise, I'll be outside of the room. You can connect with me later, and thank you again for the opportunity.