Effective AI Agents Need Data Flywheels, Not The Next Biggest LLM – Sylendran Arunagiri, NVIDIA

Hi there. My name is Sylendran. I work with NVIDIA on the Generative AI Platforms team. In this video, we'll look at what it takes to build effective AI agents that always stay relevant and helpful over time.

And it's not about powering them up with the next biggest LLM in the market. All they need is simple data flywheels. In this video, we'll look at what data flywheels are and how we applied them to an internal agent at NVIDIA.

I'll also share with you the lessons that we learned along the way, and most importantly, I'll leave you with tools and techniques and a framework to think about how you could build your own data flywheels for your own agentic AI use case.

Let's get started. So AI agents are generating a lot of buzz recently, and it's rightfully so. They're already starting to make their way into your workforce as new digital employees.

They come in different forms and sizes depending on the use case that you apply it to. They exist as customer service agent, software security agents, research agents, and et cetera, et cetera.

But what really are agents? If you look at the definition of agents, you will see agents are systems that are able to perceive, reason, and act on the underlying task.

That means they're able to look at data, they're able to reason and come up with a reasonable plan to act on the specific user query, and they make use of tools and functions and external systems to achieve the underlying task.

But in my opinion, what completes that cycle is if AI agents are able to capture and learn from user feedback along the way, and thus learning from that user preferences and data and constantly refining themselves to be more accurate and useful for the underlying task.

Right. But we all know building agents can be very painful, and scaling them can be increasingly challenging for sure. For one, the data changes rapidly. If you are an enterprise customer and you're using an agentic system for a use case, you constantly see a new data business intelligence flowing into your system.

And augmenting that is changing user preferences and customer needs, right? And if you're looking to deploy large language models, bigger chunkier models to support the underlying use case, you also see that your inference cost is going to go high.

So increased, increased usage drives increased cost. That's exactly where data flywheels can help. At its core, it starts with the enterprise data. Data flywheel is a continuous loop or cycle of data processing and curation, model customization, evaluation, and guardrailing them for safer interactions, building

state-of-the-art RAG pipelines alongside enterprise data, uh, to provide relevant, accurate responses. So as AI agents run in production environments, this data flywheel cycle triggers a continuous cycle of data curation, of ground truth using inference data, business intelligence, user feedback

to continuously experiment and evaluate existing and newer models to surface efficient, smaller models that provide at par accuracy expectations as that of larger language models, but which offer lower latency, faster inference, and effectively lower cost to, uh, ownership, total cost of ownership or inference cost.

Before we go into how do you build your data flywheel, uh, here's something exciting that we announced a few weeks back at NVIDIA. We announced NeMo Microservices. It's an end-to-end platform to build powerful agentic and generative AI systems, um, and also powerful data flywheels around them.

It, it's a model microservice that has various components for each stage of this data flywheel loop. There's NeMo Curator that helps with curating high-quality training dataset, including multimodal data.

NeMo Customizer helps with fine-tuning and customizing underlying models using state-of-the-art, um, customization techniques like LoRA, P-tuning, full SFT, and we're continuously adding more fine-tuning capabilities.

NeMo Evaluator to evaluate, um, benchmark on academic benchmarks, institutional benchmarks, uh, as well as using LLM-as-a-Judge. NeMo Guardrails for providing guardrail interactions for privacy, security, and safety, and NeMo Retriever to build state-of-the-art RAG pipelines.

And the best part about these NeMo microservices is the ease of use. They're exposed as simple-to-use API endpoints, and with just a single or a few API calls, you're able to customize large language models, evaluate them, guardrail them for the underlying use case.

And you also have the flexibility to run these microservices anywhere: on-prem, on cloud, on data center, even on the edge. And with NVIDIA support, you get the enterprise-grade stability and support as well.

Now, here is a sample data flywheel architecture leveraging NeMo microservices. The-- I like to think of it as Lego, uh, pieces where I can just put together these NeMo microservices components to build this entire data flywheel setup.

So in this sample architecture, you can look at the end user who's interacting with the front end of an agent. Think of it maybe, say, a customer service agent.

It's guardrailed for safer interactions behind the scenes, and, uh, on the back end, there is a model that is served as an NVIDIA NIM for optimized inference. So to figure out what's the right model to power this use case without compromising on accuracy, you can set up a data flywheel loop to constantly curate the data, store it

in NeMo Data Store, uh, and use NeMo Customizer and Evaluator to trigger a continuous cycle of retraining and evaluating. And once you see a certain model meet the target accuracy, you can use your, your IT admin or your-- an AI engineer can promote this model, uh, to power, uh, the agentic use case as the NVIDIA NIM underlying

this. So that's a sample architecture. Let's take a look at a real-world case study on how we adopted and built this data flywheel for our NVinfo agent, for an internal employee support agent that helps NVIDIA employees with access to enterprise knowledge across multiple domains.

It's a customer service or an employee support chatbot agent. It can help answer queries across a variety of domains, from HR benefits, financial earnings, IT help, product documentation, or anything, everything that an internal employee needs access to.

So here's a quick look at the underlying data flywheel architecture that we built for this particular agent.

Just like we saw, a user, in this case an employee, when they submit a query to this particular agent, uh, it's guardrailed for safety and, uh, secure interaction. And underneath, uh, this most important router agent, which is run by an LLM, um, is orchestrated by multiple expert agents underneath.

So each of these agents do a great job in their individual specific domain, and they're augmented with, you know, a rec-- a RAG pipeline to fetch the relevant information to answer the user query effectively.

So in order to decide what models to use and power each of these green models that you see, we have a data flywheel loop set up, uh, which constantly builds on top of the user feedback and production data inference log that come in when this router is in action.

So using a subject matter expert and human-in-the-loop feedback, a ground truth is continuously curated. NeMo Customizer and Evaluator are used to constantly evaluate multiple models to promote the most effective model as a NIM to power this router agent.

And the... If you're curious to know what these red ones are, since this is a multi-model agentic system, the red ones are essentially everything that could go wrong, uh, with, with this multi-model agentic system.

But, um, for the sake of this exercise and understanding a concept in detail, let's look at one of these in, in, in, uh, further detail. Let's look at the router agent since we've been spending some time talking about it.

So it's, it's basically orchestrated by a mixture of agent architecture underneath. So the router agent, depending on the user query, understands the intent and context and guides or routes the query to one of these expert agents, which further uses the RAG pipeline underneath to fetch the relevant information.

So the problem statement is: Given a user query, which of the agents, the expert agents should the router route the query to? Uh, and the goal here is to ensure that our agent, the routing agents, uh, accurately routes it to the correct expert agent, uh, but also using faster and cost-effective LLM to achieve this task.

So to start off, we compared and contrasted a variety of models to power this router agent. When we used a 3.1 70B or 3.3 70B variants, we noticed a ninety-six percent baseline accuracy that is just deploying the foundational pre-trained model as is without any, uh, fine-tuning or any actions on that.

And we noticed that the baseline accuracy was ninety-six percent in terms of routing the query to the right expert agent. But if you see the smaller variants, the accuracy is subpar, with the 8B variant is at below fourteen percent, and that's not ideal for our use case.

And that's exactly where most of the enterprise evaluations get it wrong. So they see a bigger model, uh, deliver state-of-the-art amazing accuracy of ninety-six percent, and there's a smaller model that gives you low accuracy, and they figure that out, "Hey, we need to go with, uh, a 70B variant because we're seeing a ninety-six to a fourteen percent

comparison. I don't think even with fine-tuning we would be able to get there." But that's exactly where data flywheels kind of help you. So what we did was we ran the 70B, um, uh, Llama variant, and, uh, we used and circulated a feedback form among NVIDIA employees, and we asked them to submit few-- we asked the employees

to submit a few queries and capture, uh, the feedback in terms of whether the, the response was useful to them or not. Uh, based on that, we were able to curate around thousand two twenty-four data points.

Seven twenty-nine of them were incorrect... Sorry, satisfactory responses, and four ninety-five of them were unsatisfactory responses. There was something wrong that our employees were not happy with.

So here's what we did. We used NeMo Evaluator and used N-LLM as a judge to investigate these four ninety-five unsatisfactory responses. We figured out that one forty of them, uh, were due to incorrect routing.

And using a subject matter expert and further manual analysis, we found out thirty-two of them were truly due to this incorrect routing. So we figured out and set up a ground truth data point-- data set of eight...

six eighty-five data points. And we split them into 60/40 to train or fine-tune smaller models and 40 to test and evaluate, uh, the accuracy. So you will be surprised to know just with six AD five data points, the results that we achieved were outstanding, and this was possible only because of the data flywheel setup.

So the 70B variant, like we saw, provided us or gave us an accuracy of ninety-six percent, uh, for accurately routing it to the right, uh, export agent. But it came with an, uh, latency of point two six seconds,

uh, to, to generate the first token response. Whereas the 8B variant gave us an accuracy of fourteen percent, but the latency was almost seventy x-- uh, seventy percent, uh, lower.

So on further fine-tuning various variants like the 8B and 3B models, we observed that the 8B with a bit of fine-tuning was able to match the accuracy of a 70B variant.

And, uh, depending on how critical-- the mission-critical your use case is, uh, you can even look at the 1B variant. It's got a ninety-four percent accuracy. That's not bad.

Uh, it's just two percent below 70B, so like I said, you can take, uh, a trade-off between how accurate you want your agent to be versus the cost that you wanna manage and the resource that you want-- resource that you wanna manage under it.

So by deploying, let's say, a 1B model, you're looking at a ninety-eight percent savings in terms of lower inference cost, and also 10x-- 70x model size reduction with 70x lower latency.

So that's the magic of data flywheel. And imagine building an automated loop or a cycle of continuous and periodic evaluation and fine-tuning as newer models are published into market and, uh, effectively surfacing smaller models that can run your agentic workflow to replace larger models that you already had in production.

Continuously learning from, uh, the ongoing production log and knowledge, uh, and using that to train smaller models is, is the power of data flywheel. So as next steps, I hope you will explore NeMo microservices and NVIDIA NIM to build your next agentic use case and also the data flywheel.

But here's a good framework to think about how do you build effective data flywheels.

So the first step starts with monitoring user feedback. Start thinking about intuitive ways to collect user feedback signals, uh, intuitive user experience, privacy compliance, implicit signals, and even explicit signals to understand whether the models that are powering your agentic system is going through model drift or, uh, any inaccuracies.

And spend time analyzing and attributing the errors or model drift that you see as to why the agent is behaving a certain way. Classify those errors, attribute those failures, create that ground truth data set that can be further used for the third and fourth step of plan and execute.

The planning is where you identify different models, generating synthetic data sets, experimenting with them, fine-tuning them, and understanding and optimizing your resource and cost, and, uh, putting them into action by executing it.

Execution does not just mean triggering a data flywheel cycle,

but also setting up a regular cadence or a mechanism to track your accuracy, latency, um [clears throat], monitoring, uh, performance and production logs, and, and truly managing your end-to-end GenAI Ops pipeline.

So that's a good framework to think about building data flywheels, and hope, um, this was super helpful for you to get started with building your own data flywheels for your agentic use case.

So that's all I have for you in this video. Hope this was informative, and you start thinking about building not just agentic use cases, but also data flywheels around them, and hopefully with NVIDIA tools and frameworks.

Thank you. Have a wonderful day.