Parsing the Epstein Files for JMail — Palak Agarwal & Omar Alhait, Reducto
Read the talk
Parsing the Epstein Files for JMail
Palak Agarwal and Omar Alhait explain how Reducto turned difficult PDFs and handwritten flight logs into data for JMail—and why schemas, redactions, document-specific settings and representative evaluations matter before scaling.
From a talk by Palak Agarwal and Omar Alhait
At a glance
Ideas worth remembering
JFlights needed a flight-specific extraction configuration that could handle varied handwritten and redacted documents under one schema.
Regular schema generation uses the request; enhanced generation also uses parsed document content to tailor field descriptions.
A visible redaction can leave underlying PDF data accessible. The JMail team had to preserve intended redactions in the extracted product.
Start by inspecting a small sample, then evaluate systematically on a hundred or a thousand relevant documents using appropriate configurations.
Document-specific prompts can target figure or layout processing, while reliable model training still depends on careful human labeling.
The document problem comes before the interface
A clean PDF is the easy case. Scanned pages, rotated documents, charts and complicated layouts make it harder to recover information accurately enough for an application to use. Palak Agarwal, who leads developer relations at Reducto, opens with that distinction: document processing sits before the model and handles the messy material that a polished interface will eventually hide.
Reducto describes its approach as rasterization-based: it examines the rendered document visually. Its agentic OCR framework can flag uncertain readings, correct them and continue, rather than commit to a single pass. The platform exposes five capabilities—parse, split, extract, edit and classify—so recovering a page’s contents can support several different downstream tasks.
JMail gives that processing layer a concrete job. The product presents released Epstein emails through a familiar inbox interface, then extends the idea into applications such as JDrive, JFlights and JPhotos. Agarwal reports that Reducto parsed more than three million pages for the project, including redacted PDFs, scans, complex forms, tables and handwriting from many different people.
Omar Alhait, introduced as a Reducto founding engineer, takes over with the original obstacle: an appealing inbox concept still needed usable data. Even a cover sheet could have a difficult layout and a somewhat subjective reading order. Recognizing words was only part of the problem; the pipeline also had to decide how those words belonged together before other products could use them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
JFlights: one extraction schema across difficult flight logs
JFlights turns flight documentation into a dedicated view of flight history. Its inputs included poor-quality handwritten records, which added another interpretation problem on top of layout. Building the separate product meant taking the flight documents and applying an extraction configuration designed for that use case. Parsing came first; extraction then requested the information the flight application needed.
Studio was the place to develop that configuration before running it at scale. Alhait could upload a document, run the pipeline, adjust settings and inspect results. The useful comparison is between the difficult input and the information the pipeline returns: can one flight schema handle different documents, including those with redactions? He describes inspecting those variations to see whether extraction behaved as expected, before committing to a much larger run.
Throughput brought a separate constraint. During the initial document release, Alhait pushed parsing hard enough to stress the infrastructure and repeatedly page Alvin, the engineer on call. The late-night response was memorable: “Hey bro, can you slow down?” The team completed the processing in time, but the episode separates two requirements that a small demo can conceal: extracting useful information and sustaining the volume needed to process the corpus.
Once the corpus was available, it supported several products, including JFlights and JCal, which Alhait worked on himself. The configured pipeline could then be deployed and invoked through either a direct API call or a generated pipeline code snippet. Studio connected exploratory work to application integration: define the requested output, inspect it, then call the configured processing from code.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A generated schema still needs the documents
An audience question makes the schema’s role explicit: must a user define it in advance? Reducto can generate one from a request. Alhait creates an extraction pipeline for resumes, generates a resume schema, then uploads resumes and runs it. What does that first schema actually ask for? That is the first thing to inspect, because a plausible collection of fields may still request something different from what the application needs.
The schema is the “paintbrush” on the document-processing canvas: it directs what extraction should return. Uploading a real resume can expose a mismatch between the requested fields and the desired result. Generating a schema therefore provides a starting specification; running documents through it tests whether that specification asks the right questions.
A second question presses on validation across millions of documents. After an initial misunderstanding, Alhait explains two ways to generate the schema:
- Regular generation: a quick LLM call builds a JSON schema from the user’s request. This gets an initial result on screen quickly.
- Enhanced generation: the system parses supplied documents first, then uses their contents to tailor the schema descriptions. This gives generation information about the actual material being processed.
The distinction concerns the input to schema generation. A request alone describes the intended task; parsed documents add evidence about how that task appears in this collection. Enhanced generation can make the descriptions more specific, but it does not itself establish extraction accuracy across the corpus. The broader evaluation question returns later in the discussion.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A black box can hide text without removing it
The redacted flight documents raise a consequential question: were the redactions permanently applied? Reducto also parses PDF metadata, so the rendered page is not the only source it can read. A black box drawn over text can conceal that text visually while leaving the underlying data available to extraction.
Alhait distinguishes correctly redacted files from others where information behind the redaction remained accessible. That created additional work for the JMail team: material intended to remain redacted had to remain concealed in the resulting product too. He describes the responsibility, rather than a specific remediation algorithm. “Our tool was working a little too well” captures the failure: recovering more data can violate the document’s intended disclosure limits.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Requested fields, source citations and competing text sources
Schemas can request substantial output. Alhait gives an example he had recently seen with 156 fields, each requesting different information. He estimates a nesting limit of four or five layers; that is a recollection in the discussion, rather than a firm specification. Field count and nesting describe what the system can be asked to return, while correctness still depends on extracting the requested information from the document.
A question about entity resolution leads to a more specific account of post-processing. Alhait says the system does work to link entities, then describes a citation model: one model can extract a value, while another determines where that value came from in the document. This explains source attribution for extracted information; the discussion does not establish how the system resolves identities across records.
Another question separates two sources of text that can coexist inside a PDF: information recovered from the rendered image through OCR, and information available from the PDF itself. Alhait initially starts enabling figure output on the resume example, then addresses the clarified question through Reducto’s hybrid OCR configuration. It can prioritize OCR data in some cases and PDF metadata in others. The selection rules are not explained, so this establishes a configurable hybrid approach without exposing its decision procedure.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Parsing supplies the structure that later tasks consume
Does the extraction workflow first recover document structure, or simply hand everything to an LLM? Alhait’s answer is that extraction consumes the parse results. Parsing is an upstream operation whose output becomes input to a downstream extraction pipeline. That separation also makes other operations possible after the content has been recovered.
A math textbook illustrates the dependency. First parse its content. Then split it into regions by identifying where chapters begin and end. Extraction can operate on each chapter to produce something such as a chapter summary. The chapter boundaries determine the material each extraction receives; recovering those boundaries is therefore part of preparing the task, rather than merely formatting its final answer.
The team declines to name the models used for parsing and extraction, but describes how production failures guide training. If a particular kind of data performs poorly—for example, when recall over long context is weak—the team trains to address it. Comprehensive extraction is “all the needles in a haystack”: finding one relevant item is insufficient when the schema asks for many different facts spread through the material.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Inspect a few outputs, then build your own benchmark
The benchmarking discussion finally answers the scaling concern raised during the resume demo. Reducto’s team says it plans to release benchmarks, while acknowledging that customers usually need evaluations built around their own documents. The recording supplies no comparative results or evaluation protocol that would establish the team’s claimed performance advantage.
Configuration matters to those comparisons. A customer can evaluate a provider with unsuitable API settings and get results that do not represent the intended setup. The team describes working through configuration options and their tradeoffs with customers before letting them decide. A useful comparison therefore includes the settings used to process the documents, as well as the provider’s name.
The practical sequence starts small: use a simple dataset and inspect whether Studio’s outputs make sense. These “vibe evals” help reveal obvious mismatches before committing more effort. As the workflow scales, move to a more systematic evaluation with a hundred or a thousand documents. The shift is from checking a few understandable examples to testing whether the configured extraction holds up across the collection it must serve.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The interface earns attention; labeled data earns the model
Alhait returns to product design with an admission: his early way to expose the parsed corpus was a rough retrieval-augmented chatbot that answered questions about the emails. He sees a possible use for journalists, but credits JMail’s product experience with making the extracted material appealing to a much wider audience. The same underlying information can support very different ways of exploring it.
Another team member gives the observable change: an overwhelming collection of scanned documents became something they could log into and search for familiar names. Extraction made the material usable; the interface made starting easy. JMail is an unusual subject for a document application, but that movement from unstructured files to an approachable product is the pattern the team also sees in industry.
The model work underneath remains labor-intensive. Reducto has a labeling team that sifts through documents and checks the data used to train new models. Alhait emphasizes the repetitive classification work before discussing how to arrange the training data. Better training procedures still need trustworthy labels. His account leaves the labor in the story: “you still have to label that data.”
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tailor the processing to the document, then try a small application
The final technical question concerns documents that need specialized logic: particular metadata arrangements, model choices or processing instructions. Alhait describes configurations and system prompts applied to separate models for specific processing tasks. This gives the user a place to express document-specific requirements beyond the extraction schema.
Where should those instructions go? The examples distinguish two kinds of customization:
- Figure interpretation: for a collection of logos, add instructions that help distinguish Lincoln Financial from Google and produce the desired figure outputs.
- Layout interpretation: for difficult construction blueprints, spend time adjusting the layout prompt and checking how the extraction behaves across different blueprints.
The relationship is between the failure and the stage receiving the correction: instructions about figures belong with figure processing; instructions about hard-to-read layouts belong with layout processing.
The recording ends by turning the case study into an exercise. Participants are invited to sign up for Studio and use material supplied through a QR code to begin an extraction project with a coding assistant. The proposed session allows about 30 minutes, with a show-and-tell near the final five-minute mark. An idea and a single-shot prompt are enough to participate; the team will circulate to help with Reducto and answer model-training questions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Read the complete timestamped transcript
- 0:12
Hi, everyone. My name is Palak. I lead developer reduct- relations at Reducto. Um, and before we dive into JMAIL, I'd love to give you a little background on what we do. Uh, we're an agentic document platform, which basically means we help teams turn messy, unstructured documents into data that AI can actually use accurately. Um, to give you some context, in the last three years, we've raised 108 million from investors like ACC, Benchmark, and more, and processed over two billion documents
- 0:42
across customers like Harvey, Scale, and a top five global hedge fund. Um, and operating at that scale means that we've basically dealt with every document you can possibly think of, um, and we've iterated Reducto's product to handle anything. Um, now, getting to AI to work on a clean P- PDF with a frontier model is not that hard, but the truth is that most of enterprise documents are unstructured. That means, like, scanned documents, rotated
- 1:12
documents, charts, um, and that's when things get really complex. Reducto is the layer that sits right before your model. Um, we handle complex layouts, but also, um, as well as post-processing so you can focus on building your actual product. Um, the reason this works is that we take a rasterization-based approach. That means we look at your document like a human would look at the document instead of the underlying embedding. Um, everything is powered by our agentic OCR framework, where rather than
- 1:42
making a single pass and committing to an answer, the model flags things that it's uncertain about, corrects them, and then moves on to the next step. Uh, we'll give you access to our studio. Um, we've made it self-serve just for this session. Uh, but in total, we have five core capabilities: parse, split, extract, edit, and classify. Um, now, a quick show of hands. Who here has heard of JMAIL or looked through JMAIL before? Okay. Wow. That's a, that's a good chunk of people.
- 2:12
Um, but I'll, I'll give a quick rundown in case you haven't. So JMAIL was originally built by Riley Walls and Luke Igel in five hours last November. That's when the Epstein files were released. Uh, the concept is basically Gmail forked, and you can log into the inbox like you're Epstein, and then that grew into a full suite. So you can think of, like, JDrive, JFlights, Jamazon, JPhotos, and so much more. Um, by February, there were over four fifty million visits and eighteen million users that had accessed
- 2:42
JMAIL, and we're really proud to share that Reducto was behind parsing all of the documents. And that's not a small number. It's over three million pages of redacted PDFs, scanned documents, and messy government files as well. And to give you an idea, like, this ranged from anything, like, really complex forms and tables to, like, messy handwriting of, like, so many different people. Um, now I'm gonna hand it over to Omar, who's one of our founding engineers and worked very closely with the JMAIL team back in November. Um, and he'll give you a
- 3:12
technical rundown and more.
- 3:15
All righty. Who's ready to talk about some documents? Yippee. All right. So, um, yeah, so, uh, walking us back to, you know, a very fateful night in November, right? Um, I was talking with the founding team, and we were trying to figure out, you know, what would be a great way to use Reducto with this kind of transformational cultural event that was, you know, going to be unveiled to the public, right? Um, it just so
- 3:44
happened that, um, you know, we're very close to Luke, uh, who is a founder at Keno and is working downstairs, and we were all just kind of talking about, you know, what would be cool to build, what would be funny, ha ha ha. Um, and then, uh, it kind of snowballed into this idea of, like, what if you could, um, you know, sign into your email as if you were Jeffrey Epstein and view, um, you know, your email as if you were him? And so, um, you know, it was a really, really cool concept as a
- 4:14
product. But there was one really hard problem to solve that, you know, was basically a barrier between us and the productized solution. Um, and you're looking at part of it, right? So it's really, really... Like I, I would urge you, if you haven't done this before, to send this, like, uh, you know, a cover sheet to ChatGPT and try to get some results out of it that... and, and just see how accurate it is, right? It's a very, very complex problem where, you know,
- 4:44
layouts are, um, pretty hard to understand. The reading order of this is also somewhat subjective, right? Um, and so we needed something to basically solve this bottleneck so that we could work on all of these cool, uh, little, you know, products downstream. And so this is where Reducto came in. So I'm actually gonna show you guys, um, what we had used for a couple of our products-
- 5:09
Yeah
- 5:09
... right? So... Oh, yes. Let me do this real quick. Do, do, do, do.
- 5:22
All righty. Whoops. I'm gonna bring this over. All righty, so here you see a cool product, right? JFlights. Basically, all of Jeffrey Epstein and his affiliates' flight history, right? This guy was flying people out. He was flying diplomats out. He himself was flying many places to and from his island, right? Um, what was basically the barrier between us and finding out, uh, finding all these flights were these pretty shoddy
- 5:51
handwritten documents, right? Um, again, if you were... You know, if you thought that cover sheet was a little bit rough, think about handwriting, right? So- We can see here a pipeline that I had actually set up way back when, uh, when we were trying to build a good config to basically make our parsing really, really good, right? And so in this case, uh, we had... For the f- for the flights use case, you know, an idea had flown out that, like, what if
- 6:21
we, um, you know, spin this off as a completely separate product where you could just see all of his flights? Um, and so what that meant is taking all of his flight documentation and then applying a separate, you know, extraction config for it. And so what you'll see here is, um, kind of a little tour as to how Reducto works, right? So Reducto, uh, within the document extraction use case, first parses the document, right? And so this kind of shows you, you know, you could stick any document that you would like here and you could just click Run, right?
- 6:51
Configure any of our bells and whistles and see real-time results, right? So this was sort of the canvas for myself that I was playing around with to kind of understand before I start, you know, scaling this up to a million different documents and run up a quick check on Reducto's, you know, uh, on Reducto's, uh, um, you know, uh, receipts, let me make sure that our, um, extraction accuracy is what I would expect it to be. And so here is kind of what we ended up with. Um, you could
- 7:21
see here for a couple of these different documents, you know, some redactions that we had to deal with. Um, Woody Allen, I know that guy. Um, but yeah, just like a lot of different, um, you know, use cases that we had to handle through one extraction schema, um, for the flights use case, right? And so we parsed all the documents, uh, very quickly. A- actually a fun story about this. Um, is Alvin in the room? Alvin is in the room. So, um,
- 7:51
back when the fl- uh, the, uh, first, uh, drop came out, I was paging Alvin maybe like, you know, a couple times a night. Uh, he was... He just happened to be on call, uh, the night that I was parsing all of these documents and we were trying to parse them very, very quickly. And so we put a, you know, a decent amount of stress on our infra and so Alvin was sending me a lot of fun messages late at night like, "Hey bro, can you slow down?" Um, but yeah, we got through them. We got through them in time and then, um, once we had this kind of corpus of
- 8:21
documents that we were able to, um, you know, um, kind of take as a ground truth, we then spun off a, a billion different products, right? And so Paula kind of touched on this a little bit, but I'm gonna go back to JMail.
- 8:40
So, you know, we, we spun up any- anything from JAmazon to JFlights to Gemini, right? Um, yeah, so... And I... And also JCal. This was actually a product that I, um, worked on myself, so some fun stuff there. Um, cool. And so I guess, you know, if you could kind of take, uh, anything out of this, I do wanna say actually, um... Yeah. So if you could take anything out of
- 9:09
this, right? Like, the simplicity from this is really like, you know, getting a schema, right? Looking at these results, making sure that the results look like what I would expect, and then just deploying, right? So if I was to deploy this code snippet, I'll just redeploy here for our use case. And I can get either a direct API call or just a pipeline code snippet that I can invoke very quickly, right? Um, so the name of ga- name of the game here is like stick your documents in this platform and then
- 9:39
immediately start, um, you know, visualizing your data and you could immediately also productionize it, uh, afterwards. So I'm gonna stop here real quick. Um, do we want... Does anybody have any questions thus far about, um, JMail or Reducto in general before we can kind of start, uh, working on a little interactive exercise? Yeah.
- 10:03
Is schema... I'm just trying to understand-
- 10:05
Yeah
- 10:05
... if the schema pre-defined or is the schema something that one might have to-
- 10:10
Great question. So we can actually come up with this for you, right? Um, you know, my, uh, my fun analogy here is that, like, your extract schema is sort of like the paintbrush on your Reducto canvas, right? So it definitely drives everything that you, um, you know, you get out of the schema. So for example, right, I could create a new pipeline here, right? And create a new extract pipeline and just call this, I don't know. Uh, let me make a resume schema 'cause I
- 10:40
want to parse some resumes, right?
- 10:44
This is not what you're after.
- 10:45
Oh, whoops. Sorry about that. Actually, I'm gonna do this real quick.
- 10:51
Cool. So anyway, I'll, I'll do this again 'cause it's very simple. So I'll create a new extract pipeline here. By the way, we could do a lot of things with documents. Parse, split, extract, edit, even classify. Um, and so what I'm gonna do here is create a resume schema. Oops.
- 11:15
Uh, my Wi-Fi is not being helpful with me today. There we go.
- 11:20
Oh, work.
- 11:21
All righty. So I got a fun schema here and now it's up to me to basically decide whether or not this schema is, like, satisfactory or not, right? It, it's really giving... It's gonna give me exactly what I'm asking for and, you know, once I plug a document in here I might realize like, ah, this is actually not what I want, right? So I have a ha- bunch of handy dandy resumes. I'm gonna stick them in here.
- 11:46
Upload and then just run this.
- 11:54
So yeah, um, I hope that answered your question.
- 11:57
Yeah. Thank you.
- 11:58
Cool. Any other questions while we wait for this?
- 12:00
Um, just a-
- 12:04
Like you said, generating schema, looking at results and then just deploy
- 12:07
Mm-hmm
- 12:08
You have millions of documents. How do you really validate what you're parsing when you're say, looking at the results? What if... Can you explain a little bit more?
- 12:15
Uh, you're talking about the loading state? Uh, this is like a fun loading state. At the end of the day, right, like, our product is, like, a, a really... You know, it's a suite of different models, uh, and different VLMs that we deploy, right? Um, I, I'm, I'm-
- 12:27
Sorry, it's not the loading state. But, um, I was following what you were saying, is that generate, you know, instruct the schema for what the document looks like-
- 12:34
Yeah
- 12:34
... and you look at the result of that. Just do you-
- 12:38
Oh, that's a great question. So if I'm understanding correctly, right, what you're saying here-
- 12:42
To validate whether you parsed it correctly.
- 12:43
Yeah. Yeah. So this specific, um, this specific, like, schema generation that I had just used is really out of the box, and it's just kind of like a, a quick LLM call that descri- like, builds a JSON schema exactly for what I asked for. However, we do have an enhanced schema generation, right? And so what this will do is let's say I've parsed my documents. It will then take the output of, like, what's in the documents and tailor a schema description directly for them. So, like, if you're looking for example for, you know,
- 13:14
something just really quick to just get up on the screen and just see some results, I would recommend regular, you know, uh, uh, schema generation. But if you are a lot more interested in, like, nuance, uh, I definitely suggest parsing the documents first and then kind of passing them in. All, all you would have to do is just click Enhance Schema Generation, and what we will be able to do is, um, you know, run the documents first, try to get an understanding of what's in the, uh, in the documents so that we can tailor these schema descriptions directly for that. Make sense?
- 13:44
Yeah.
- 13:44
Cool. Cool, cool, cool.
- 13:48
You had some redaction on the document.
- 13:51
Yeah.
- 13:51
That is a permanent redaction burned into the document or-
- 13:54
Correct. Yes. So I mean, yeah, like, at, at the end of the... Uh, this is a great question, right? So Reducto actually parses PDF metadata as well, right? So some, uh, like, sometimes, uh, and this is honestly testament to, like, uh, who was doing the redaction, right? Sometimes the redaction was actually not that great, right? And so what that ended up being, or the responsibility ended up being on us as the JMail team to then have to do a bunch of redaction, right? Because you can't just stick a box on a PDF and then expect it to actually be
- 14:24
fully redacted, right? Um, yeah, seems like there was some, uh, a lack of understanding when whoever, who- whoever was cooking up that redaction.
- 14:32
This was, this was a layer on top of the data. A lot of permanent redaction.
- 14:37
Um, I would say it's like for the files that came out and myself looking through them, that's not the case, right? Some files were actually redacted correctly. Um, but there was some files where we were just able to see who was like, you know, who, who the redaction was behind. And, um, so it became a really, really serious, you know, situation, right? Because we're dealing with, you know, people's data that, like, um, you know, our... Data is redacted for a reason, right? And, uh, you want to really make sure that you're not the one
- 15:07
that, like, ends up getting sued at the end of the day because of this. Um, and so, yeah, there was a, you know, a, a heavy kind of level of responsibility on our shoulders to make sure that whatever was intended to be redacted was actually redacted. Our, our, our tool was working a little too well.
- 15:25
Cool. Yeah. At the front, and then I'll... After you.
- 15:27
Uh, two quick questions. One, what's the highest dimensionality schema that you've ever worked with when you're trying to pull up hundreds or thousands of different things from that pane on the right side? Any use cases for that?
- 15:38
Yeah, yeah, yeah. Uh, it's a great question. Um, I'll give you some, like, you know, an, an example, right? There was, there was a schema that I was looking at maybe on Friday, uh, that had like 156 fields, right? Um, and all of these different fields, you know, need, like, they were requesting something completely different, right? Uh, and so I urge you to kind of put our schema to the test. I think, like, our, um, the, like, nesting, uh, limits to our schema are maybe
- 16:08
like four or five layers. Uh, but in t- if you wanna, you know, ask for everything in the world, you can still technically be able to fit it in, in a schema.
- 16:16
Post-processing steps, and you might be getting to this later, do you have any capabilities for, like, entity resolution?
- 16:23
Entity resolution. Can you give me an example?
- 16:25
Like data mastering, like as you go through trying to, to pull different entities.
- 16:30
So this is a good question. Um, there is a lot of post-proc- post-processing that we end up doing, right? Let's say, like, on the extraction side. Um, I'm personally a little bit farther abstracted away from, like, the ML side of things. Uh, but I can tell you that we are doing a decent amount of post-processing to basically link, um, you know, entities together. For example, we have like a citation model, right? Where, you know, we have, you know, one model that could be responsible for extraction, right? But then there might be another model that's also responsible for
- 17:00
once this data is extracted, where is each piece kind of extracted from, right? Uh, NVIDIA actually did a cool, um, paper on this, uh, that they, that I just found out about like yesterday. I think it's called like Find Everything or something like that, um, where they try to address the bottleneck of like, you know, citation, uh, determination and like the speed at which you could find a citation for something that's in a document. If you're interested about these things, we are hiring and so, uh, come speak
- 17:30
to us after. Sorry.
- 17:33
Questions. So you mentioned there's context data in the PDF as well as image data. How does the model... How can you tune the way the model picks one versus the other in terms of source of truth?
- 17:41
Yeah, absolutely. So I'm gonna try to see if one of the resumes I uploaded has an image in it. So let's see here, right? So, um, we have some figure models, right? That are basically determining... Let me actually also rerun this real quick so that it return figures. Do, do, do, do, do, do, do, do.
- 18:12
Mm-hmm
- 18:13
So what, how, how does the model prioritize when it needs to text from the PDF document or OCR?
- 18:18
Ah, that's a great question. Um, Alvin, did you hear that?
- 18:22
Sorry.
- 18:22
No, no. So I'll, I'll explain. It's-- The reason why, um, I, I asked Alvin is because Alvin worked on our hybrid OCR model, right? Um, and so we have a specific OCR, uh, like, you know, configuration that you can enable here that will pri- figure out kind of intelligently what to prioritize, right? Um, so in some cases we will actually pick out like just, you know, strict OCR data, or we might pick out like, um, you know, metadata from the PDF. Um, yeah, we've done a lot of work here. Uh, and it's, it's not a simple
- 18:52
solu- Like, it's not a simple answer. Um, this is something we could do an entire different talk on.
- 18:59
Great.
- 18:59
Yeah.
- 18:59
I have one more. Um, do you do any kind of like, do you use any kind of document processing as your, like, initial step, um, frameworks or tools such as, um, like Docment, for example? Like, there are tools that let you kind of parse out PDF in terms of the structure content-
- 19:20
Mm-hmm
- 19:20
... and feed it to the LLM-
- 19:21
Mm-hmm
- 19:21
... or do you just use LLMs?
- 19:23
So I'll be frank with you, like this is actually exactly what this pipeline is doing, right? So like we first parse the, like the... Hold on, let me turn, turn figures. Cool. So, um, while I rerun this, what we're doing here-- Let me pick a different example. Ah. Actually, we're gonna have to wait now. But what we're actually doing in like extraction is taking the parse results and then feeding them into, you know, some extraction pipeline downstream. So
- 19:53
like parse is kind of like the, um, you know... It, it is, it is what made Reducto Reducto in the first place, right? And then we can do a lot of cool things, you know, downstream of Reducto so, uh, you know, once we have that parse content. For example, right, we could split, you know, the document into different regions and figure out, like if I'm parsing some, you know, a, a math textbook, I wanna understand like where exactly every chapter, you know, starts and ends. And then for each chapter I might wanna do like an extraction based on that, right,
- 20:24
and, uh, get, you know, the chapter summary or something like that, right? Uh, so parse is really the kind of, uh, foundation for all of these other agentic, um, document processing, uh, features.
- 20:38
What models do you use for parsing and what models do you use for extract?
- 20:42
Ha ha ha. Man, if I answer that question, I might lose my job tomorrow. So, um, I mean, it, it's a... Yeah, continue.
- 20:52
I think it depends on like what prod data we collect, um, as well. So we specifically, um, train on like certain types of data if we find that it's doing poorly on certain types of data. So we see that like long context recall is not great, then we gotta like try and fix that. Specifically for extract, um, it's, it's not really like needle in a haystack, it's like all the needles in a haystack if you think about it. So that's like a really, um, challenging like research problem
- 21:22
that our MLEs are like somehow able to figure out pretty well. But yeah, happy to chat more later. Yeah.
- 21:28
Yeah.
- 21:28
I think there were some questions there.
- 21:30
Yeah. Ooh, we've been neglecting you guys. Yeah. What's up?
- 21:33
You, you, you'll be... Yeah, you should, you should stay tapped into our, uh, like our, our socials and you'll find out about this very soon. I think like benchmarking is, you know, always a topic of discussion and something that we wanna really stay on top of, especially because, you know, somebody who is testing, who wants to do document extraction might not, you know, know where to go or who's better, right, without actually testing themselves. Uh, so that's why benchmarking is like something that's, you know, we're, we're prioritizing a lot, uh,
- 22:02
especially in the coming months to cr- to try to show people like how well we are doing. You know, we try... We, we're, we're very confident in our models and us being like, you know, the premier document extraction solution on the market. Uh, but there need to be, uh, more, you know, um, data points that we can rest on and that the, you know, entire industry can kind of see at face value, uh, without having to go test themselves or pass in the all, entire Arc AGI dataset to, you know, try to understand, uh, you know, how well Reducto is
- 22:32
doing against another solution. Yeah. So we're gonna release some benchmarks soon, but the real deeper answer here is you typically have to make your own benchmarks, unfortunately. So every single one of our customers typically have some benchmark. They're like, "Oh, we're evaluating document pro- providers. Uh, we're gonna run some head-to-heads, uh, of like Reducto versus like all these other like use cases." And, you know, sometimes they don't use like maybe our APIs really well because they use the wrong configs. So we work closely with them like, "Hey, turn this on.
- 23:03
Turn this on. You get this trade-off." And then, uh, we let them decide. Um, and then like we often like win these head-to-heads in like performance. So as with most products, you know, get a s- a simple dataset. You can do some like vibe evals. Just look, look in Studio, see if the outputs make sense and then like as you wanna scale it out, then get like a hundred documents, a thousand documents, do something more systematic. And you can also work with our team where we have a lot of people to, you know, work with customers to help you do your evals.
- 23:33
But yeah. Thank you.
- 23:34
Cool. All righty. Um, any more quest- Yeah, sure. Well, yeah, absolutely, right? Um, I think... So I'll answer this, uh, pretty honestly, right? I think when I was working on this solution, right? Or, or I was working on this problem, I was trying to figure out like what is a good productized way to, you know, surface our Reducto's parsing, right? Um, and to be frank with you, right, like I came up with something that was pretty shoddy in the beginning. Um, like basically
- 24:04
a, you know, rag chatbot application where you can ask anything about the emails, right? And it might be something that would be really, really useful for like a journalist, right? However, um- I think it's testament to the hundreds of millions of people that, you know, were, you know, using JMail and, uh, a- and the tens of million that still, you know, log on, uh, from week to week nowadays. Um, it's, it's testament to the fact that a nice product on top of a really, really nice novel, you know, uh, document extraction solution is what is going to kind of be a
- 24:33
one-two punch here.
- 24:35
Yeah. Like, would you read three million, like, scanned documents? Honestly, no. And honestly, when I saw the job, I was like, this is, like, way too much work for me. But once Omar built it, I just, like, logged on and, like, started searching, like, random, like, famous people for fun. And so that's the beauty of it, right? Like, I think, like, unstructured data is, like, everywhere, and sometimes all you need to take-- all you need to do is, like, to extract it and then, like, do whatever you want with it. It's basically what all our customers, like, do, um, conceptually.
- 25:05
So I think JMail is a pretty fun example, but it's actually very, um, it's exactly what happens in industry.
- 25:11
Yeah. You've seen it come in? The-- it's, it's just hard work, right? Like, you know, we, we have a data labeling team, right? We have to go, like, part, you know... Like, we have to sift through a lot of documents if we're going to train a new model on top of it, um, and be very, very certain in the outcomes of, you know, or, or the, the sanctity of that data, right? Um, and so, uh, the short answer for you is just, like, it's a lot of hard, menial, annoying work, right? Uh, but that's really what comes into, like, the beauty of a machine learning model
- 25:41
at the end of the day, right? There is nothing that, you know, really beats that, like, really, really hard, you know, just rote classification work, uh, that then compounds into, uh, model training and then, you know... Then you can kind of figure out, like, how to stagger your data and, like, how to train, you know, on, on top of your training set. But y- at the end of day, at the end of the day, you still have to label that data. Cool. I think we'll take... Yeah, we can take one more.
- 26:06
One more question.
- 26:07
But, um, yeah. Yeah. So to repeat the question, uh, and let me know, you know, how, how accurate it was, right? But, uh, it seems like, you know, s- like, there are cases in which, uh, you need to really customize your logic on top of different documents, uh, whether it be corresponding to specific models, right? Sometimes there might be meta- metadata set up in specific ways, um, in specific documents, and you want to really, like, uh, condition, um, for these specific use cases, right? Um, this is a fantastic question, right?
- 26:37
And so this is where our agentic, um, section really comes in, right? So you can actually apply all of these different, you know, uh, configs and apply, like, uh, system prompts on top of separate models that will do specific processing that's tailored directly to what you're asking for. So for example, right, if, uh, you want to do some complex figure extraction. Let's say I have a bunch of logos and I want to try to determine like, you know, what differentiates like, I don't know, uh, some,
- 27:07
you know, uh, Lincoln Financial from like Google and like how to separate those kinds of logos, right? I could add some additional color here within our, uh, you know, prompts, uh, in this config to custom kind of tailor, uh, the outputs that I'm looking for. So if you wanted some specific metadata here, let's say like, uh, you know, some layout might be like really, really hard to read in, uh, I don't know, like a construction blueprint, right? I would spend a lot of time in the, uh, layout
- 27:37
prompt kind of tailoring to try to understand, uh, how well, you know, uh, my current extraction solution is mapping across all of my different, um, you know, construction blueprints, I guess, right? Cool. Um, does that answer your question?
- 27:52
Yeah.
- 27:52
Fantastic. All right. So now we want to kind of, um, spin this off, um, into an interactive exercise, right? So, um, if you guys would like, we're going to basically give you guys, you know, um, however much time you would like. I think we have about 30 minutes left. Um, and you could sign up today at, uh, at Studio and essentially, uh, just get started, uh, on some document extraction solution, right? So what, what you'll see here in a QR
- 28:22
code is something that you could basically paste directly into a cloud code or a codex or whatever, and something you could just get started, uh, on, uh, document extraction with immediately. Uh, so we're gonna give you guys about 30 minutes and by, you know, maybe around like five minutes, uh, like at the five-minute remaining mark, we'd love to kind of see what you guys have cooki- uh, cooked up and we have a bunch of cool swag and a couple of cool prizes for you guys, uh, to just kind of show us what you've got, um, with respect to document extraction. Um, it could totally be just like an idea, um, and
- 28:52
like a single-shot prompt, but, uh, feel free to just kind of, uh, give it a whirl.
- 28:57
Yeah.
- 28:57
Sound good? Also, we'll be kind of like circling around if you guys have any questions as to how to use Reducto. Uh, I wanna come work for you guys. You know, I wanna hear about your models, how you guys train, whatever. Um, we are happy to answer. Sound good?