From Raw Documents to AI-Ready Data — Leo Platzer & Jeff Koss

All right. Hey, guys, let's get going. Can everybody hear me? All good? Perfect. All right. My name is Jeff Koss. I'm a staff customer engineer with Deasy Labs. Um, as you can see from the slide, it says, "Brought to you by Collibra." Collibra acquired Deasy Labs last summer. Uh, Collibra is more of a data governance tool for structured and AI, um, agents and models. So the gap that Collibra had was we didn't have unstructured, and that's really where,

uh, Deasy shines. So today, uh, with me is Leo Platzer.

Yeah. Hi, everyone. Uh, I'm Leo. I was previously the CTO at Deasy Labs, which, as Jeff has already mentioned, was acquired last year by Collibra.

And you can see we're both twins as well. We have our hats that we're giving away at our booth as well. So come by and get them. First come, first served. So today, we're talking about, um, how we can go from a stalled POC to production with Deasy data curation. So we're really focused on unstructured data. So our demo scenario I typically talk about is a manufacturing, uh, company, North River Manufacturing. You can see that, you know, there's four-- forty thousand employees. They operate in twenty

different countries. Uh, really what they're trying to do is they have millions of documents in SharePoint. They've accumulated them over years and years, and they need to figure out which ones are relevant for their chatbot. They're invested heavily in AI to improve operational efficiency. And their big objective this year is they wanna build a legal ops AI chatbot. Now, what they wanna do is they wanna help the legal and procurement teams answer questions around supplier contracts, partnership agreements,

and it's gotta be real time. So some of the questions they're trying to answer are, what are the termination clauses for a specific supplier? Um, which agreements expire this quarter? Which agreements expire next quarter? And what partnerships have e- uh, exclusivity terms?

So some of the folks in the teams that we've been working with, um, is Sarah. She's the head of data and AI. So she's responsible for the accuracy. Um, also, there's Jim. He's the AI engineer. He just needs to know which files do I need. I have tons and tons, thousands and thousands of files in SharePoint. I need to find just the relevant ones with no sensitivity. Priya is the head of knowledge management. She wants to make sure that no sensitivity is leaked, um, and make sure that if

there, if there is specific data that's, uh, classified, that it gets redacted as well. Then there's Michael. He's the director of legal ops. He needs a chatbot to deliver accurate responses to his team. So he's from the business. And then finally, there's Jackson. So Jackson's enterprise transformation, uh, lead, and this is his flagship use case, but also he has fifteen other use cases that he needs to, to deal with, right? So he

wants to have a framework, a repeatable, successful process. He doesn't just wanna do it for one time. He wants to do it for multiple different use cases.

So what they're able to do so far, they were able to take forty, around forty different files, pre-curated files, and build a small little POC, small little mini pilot, where they can do, uh, parsing, chunking, OCR, embedding, and then put it in a vector database and a knowledge graph. And they got pretty good responses, right? Seems pretty easy. You can do that with forty files.

But when they try to do it with many, many different files, with eighty thousand plus files, right, now all of a sudden, it's a little bit more challenging. And so some of the challenges they had were, "I can't discover the files that I need. I don't know where the sensitive data is in my-- on SharePoint. I can't trust the quality of the data." So my background, um, I used to work at both Informatica and Databricks. So for me,

data quality is typically the six different dimensions of data quality: completeness, conformity, accuracy. But when you deal with documents, it's a little bit different. It's I wanna see is there duplicate information, conflicting information, right? Leo's gonna talk about-- he has a really good slide about showing why it's so important and also the freshness of the data. And also everybody's talking about context, right? So what we wanna do is when we tag the metadata, we wanna use that in our

chatbot, in our vector database to make it more relevant, have better accuracy in the chatbot.

So you can see some of the challenges they had, right? They're getting incorrect answers. There's sensitive information that's being leaked out in the chatbots. It takes a ton of time to go to production because they can't find the files. They're spending too much time looking for the files instead of writing code and doing things that they, they really like to do, right? It's difficult to maintain. I'm working with another telecommunications company and they have six to seven different SMEs and they're trying to curate

twenty thousand HTML files and they can't keep up, right? So it's very brittle, very difficult. And what we do is we provide the ability to automate this entire process.

So over here, now you can see they wanna, they wanna try to access and leverage over eighty thousand different files. So they went from forty files to over eighty thousand files and really what they wanna do is they wanna find which files are relevant for their chatbot, which have high quality- Which ones are up to date with knowledge. They don't want to use a, a version where the contract is, is old and expired, right? And they wanna enrich their chatbot with the metadata that

they've tagged, and of course, they don't want any sensitive data. And so what we do over here is we provide the ability to do that, right? We can access the data, um, over here. We can understand what the data is. We can figure out which files are relevant, do the OCR, the chunking, and things like that.

And so let me show you a little-- a brief demo. So over here, you see in SharePoint, much like what North River has, they have a whole SharePoint site full of different, you know, there's call transcripts, there's PDFs, there's Excel, there's Word documents, and I-- they don't know which ones are relevant, which ones are focused around legal. Now, over here, when we look at Deasy, we've already opened up the application right here, and I can see, you know, I've already ingested three hundred and thirty-nine files. And now I can

see both the ch-- uh, page level or chunk level as well as the file level. I can see all the information right here.

And then over here, this is what we call a taxonomy. Taxonomy is a grouping of tags or a tagging tree. So I've ingested the three hundred and thirty-nine files, and now I can either manually create a user-defined tag, I can leverage tags that are reusable from my tag library, or I can leverage AI to auto-suggest what tags I should even have, right? I can leverage AI to say, "You-- these are the tags are relevant." So here I just type in my tag name, data, uh, document categorization, give it a prompt, and this prompt is what

we're gonna send to the LLM to determine, is this file-- how should we tag this file? Now, if I know what I'm looking for, I can put in the classification values. It's optional. And again, if I-- I can leverage AI to auto-suggest all these tags for me.

Now, also the other thing to point out, if I have my, uh, my taxonomy here, my valid values, um, already, I can upload them through a CSV. And so once, once we do this now, I can see I started to build this, this tagging tree. Now, the other thing I can do is I can have conditions on my tagging tree. So remember I said I only want legal and contracts? That's what they're looking for. Now what I can do is I can ha-have, um, sub-tagging. So only legal and contract files, I wanna

pull out more data around the contract type. Is it a supplier contract? Is it part of an NDA, right? Is it materials and, and time contract?

And now what we can do when we add this, you can see we have different branches in our tagging tree.

And I could have, um, unconditional tags as well. I can have any number of different tags. Now what we're doing is we're gonna generate the metadata. So I choose what files I want. I can choose what tags I want. I can say Generate metadata, and we'll run this, and you'll see

the results over here. So now I can see the co-- document category. I can hover my mouse. There's engineering documents. There's my contracts and legal, right? I have sixty-eight different files. And if I click on any of those, it automatically filters for me. Now, I can also see the evidence. Why did AI tag this? Where in the document did it find evidence that it's a legal contract, right? And I can also do thumbs up and thumbs

down, so there's few-shot learning as well. So I can give it an example if I want to. Now, you can s-- notice I ha-- also have filters. So I have two different filters. I quickly went from three hundred and thirty-nine files down to twenty files. So we also have the ability to scan for sensitivity as well. So whether it's PII, PHI, um, could be anything. I was in, uh, in London at a Gartner conference, uh, a couple months ago, and there were

some folks from Finland that came up, and they wanted to know, you know, "Can we find Finnish national ID?" I don't speak Finnish, so I put it in there. I did this demo. And not only can we do AI-based, we can do pattern-based with context. And it ended up figuring out the, the regex, um, syntax for me, but it also added some Finnish words which I thought were probably associated with Finnish national ID. So it's gonna be less false positives as well.

All right. Let me keep going here. So the other thing we can do is we can do quality. So again, for us, we have a dashboard in here for quality where we can go through and see what are the different groupings for, um, for duplication, for conflicting information, um, and then we'll, we'll leverage AI to auto-suggest which ones to keep. Now, the other thing we can do too is, again, for freshness, we can say, "I only want files within X number of days or within a specific

date range." And then the last thing we can do for data quality is, you know, here's the, the pieces of metadata that I absolutely have to have, and then what we'll do is we'll figure out, you know, of the three hundred and thirty-nine files, maybe you really need two hundred files. And then from there you can create a data slice. And then we can also give it context as well, uh, over here. So I'll go through one more video, and we also have a live demo for you that Leo's gonna be doing in a

second. So in this case now, what I can do is I can, I can leverage a, a data slice. So my data slice could be filtered based on the specific criteria that I want. So in this case, I have fourteen files. These are only the files that are relevant for legal. And now what I'm doing is I'm copying my use case description. What I can do is I can say, "Here's what I'm trying to build," and I can either do a flat, uh, taxonomy or multiple layer levels in my tagging tree.

And then over there, when I run it, it's gonna auto-suggest these tags for me. There's human in the loop, so I can choose which ones I like, which ones I want, which ones are relevant. But again, it's all done for me based on what I'm trying to do. We gave a context and based on the content of the files themselves. And then over here, it even builds out the classification values for me.

And then if we want, when we run-- when we generate the metadata, we're in the job, now we can see over here, here's all the sensitive information, right? Here's all the different information that we have over here.

And then again, I can see the evidence at the file level and the chunk level. I can see the confidence score. So coming back from, from AI, what the model returned, I don't have to guess. You know, it's a hundred percent confident, ninety-five percent confidence, here's why. So you can have ver-- a lot of confidence in the, the, um, AI-ready data product that you guys are gonna be building.

All right.

And now finally, what we can do, uh, for delivery and maintenance. So, you know, one other thing that, that's really important is with our data slice technology, we have workflows where we can put it on a set schedule, so it's auto-refreshed. So we wanna make sure that your chatbot never gets stale. So if there's new records or, sorry, new files that get added to SharePoint or deleted, we'll automatically pick those up and add those on a set schedule to my, um, uh, to my data slice or my AI-ready data

product. And then our SDK will pick that up in the chatbot, and then when we load the vector database, you'll always have the most freshest, uh, metadata.

So what's it mean for North River? So basically, they were able to go from four months of data prep work to down to days. They were able to increase their data utilization for AI, they increased their chatbot accuracy because they not only were able to figure out which files to use, but also the metadata from the files, they were able to inject that and leverage that through our SDK in their chatbot, right? And then the other thing they did was they reduced their compliance risk. They knew where all the sensitive

data is, and they made sure that they were not part of-- they were all filtered out in their, um, data slice. Okay, so I'm gonna turn it over to my brother from another mother- -Leo, and he's gonna talk about, um, context.

Yeah. Hi, everyone. Um, I just wanted to basically, like, give the so what, right? Technically, okay, you deduplicate your data, you reduce your search space, right? You clean it up, you have it fresh. What does it actually mean, right? So if we were looking at the vector space, right, when you ask a question, it comes in, and you have duplicated data, the same information, or potentially even a different version of the document with conflicting information now, right? Think of

your Slack messages. Today a person says this, tomorrow a person says that, right? The truth changes over time. If it cha-- if it stays, however, in your knowledge base, in your search base, your top K will always be full of pretty much irrelevant information, right? So in Deasy, for example, something which is considered duplicative is either another piece of context, like your database has changed and I can actually use it, or it is simply redundant,

and we will remove it from the knowledge base.

The second piece is, like, same as-- same reason as before. If your ma-- information is old, right? If you suddenly are working with data from before the two thousands, if you're working with information which was only relevant last week, is not relevant this week, in the future, some of the use cases when agents actually get into production for, let's call it, the most high-stakes use cases, that has detrimental effects. And we have done, um, a lot of

evaluation around what is the effect of making sure that your unstructured data quality is maintained, is consistent, and is always handled deterministically in the same way, right? And what we're essentially seeing is that if your raw corpus... Okay, let's imagine you have a thousand files, and thirty percent of your files are stale or duplicative. It can happen that up

to eighty percent of your actual context your agent uses to answer a question is filled up with stale information. Which means, like, it actually uses eighty percent of completely redundant, useless information and knowledge, and this has to be managed. Um, what we have seen is that by running on the exact same tasks, so this is a RAG chatbot. It's not, um, running an actual, like, let's say, Claude Code

task or Codex on a coding task. It was simply, like, multi-hop RAG evaluation. And what we saw is that if we look at recall one, like, I mean, we pretty much double, uh, the recall. Um, and overall, improve our accuracy, so, like, task completion by o-- between ten and fifteen percent. And why is this so significant? This is so significant because this has nothing to do with the use case you're working with. This is true

independent of what you're actually using the data for, right? 'Cause if you actually know what you wanna use your data for, you can apply relevancy and all nod other data quality dimensions, right? But duplication and freshness is true no matter what.

And yesterday, right, there is a lot of context topic, a lot of context flags around here. Um, I wanted to give a very, very quick demo on how you would use the DZI SDK to prepare a context repository, and I'm gonna run this live. So what this is doing is essentially it's gonna fetch all of the metadata we have about a given SharePoint site, and we will then contextualize each folder within that

SharePoint site, as well as create an index over all folders so that your Claude Code, for example, or your Codex, does not even have to look at the individual files to figure out what to look for, but simply looks at the context MD file. So what you're seeing here is essentially purely by using DZI, you can curate context files where you are directly aware what exactly is within each

folder, right? What is its purpose, right? When do you wanna use it? So for example, this is one-to-one the mirror of what I actually have in SharePoint. But now, instead of actually looking at these 23 files to figure out whether I even should look at this folder, I know exactly what document types I have, what it is about, what are the key topics, when to use this folder, and what are example questions.

So this is how-- this is all the kinds of use cases today, right? How you can use DZI for use cases like your RAG chatbot, or use DZI to actually improve the performance and reduce, for example, the token cost for your agents. Um, but yeah, basically I'll move over to Jeff again to summarize this whole presentation.

Awesome.

Thanks, Leo. All right, so guys, um, we're almost out of time. Our booth is way in the back by stage two. Please come by, bring any different use cases that you guys have. Um, we're happy to have discussions, do live demos, um, talk about what your challenges are around unstructured documents. Um, over here, um, before we leave, just wanna let you guys know we have one-week POCs, so we're happy to prove our technology.

Uh, again, you know, you guys bring your, your business problem, um, you're trying to solve with AI, bring some sample data, a bunch of different files, and then what we can do is we can get-- generate metadata for each different files. Um, we could get a full sensitivity report on your, on your data, data quality report as well. And then we'll show you how we can build a full taxonomy based on your specific use cases, your specific files. So with that, everybody, appreciate you guys-

Thank you so much

... spending time with us. And lastly, we only have one more case of these hats, so come on by and grab 'em. So we'll be out in the back. Thanks, guys. Appreciate you.