Parsing the Epstein Files for JMail — Palak Agarwal & Omar Alhait, Reducto

0:12 Hi everyone, my name is Ple. I lead developer redu relations at Reductto. Um, and before we dive into Jmail, I'd love to give you a little background on what we do. Uh we're an agent document

0:24 platform which basically means we help teams turn messy unstructured documents into data that AI can actually use accurately. Um to give you some context in the last three years we've raised 108

0:37 million from investors like ACCNC benchmark and more and processed over two billion documents across customers like Harvey Scale and a top five global hedge fund. Um, and operating at that

0:48 scale means that we've basically dealt with every document you can possibly think of. Um, and we've iterated reductive product to handle anything. Um, now getting to AI to work on a clean

1:01 PDF with a frontier model is not that hard, but the truth is that most of enterprise documents are unstructured. That means like scan documents, rotated documents, charts. Um, and that's when

1:14 things get really complex. Reductto is the layer that sits right before your model. Um we handle complex layouts but also um as well as post-processing so you can focus on building your actual

1:26 product. Um the reason this works is that we take a rasterizationbased approach that means we look at your document like a human would look at the document instead of the underlying

1:36 embedding. Um everything is powered by our agentic OCR framework where rather than making a single pass and committing to an answer the model flags things that it's uncertain about corrects them and

1:48 then moves on to the next step. Uh we'll give you access to our studio. Um we've made it self-s served just for this session. Uh but in total we have five core capabilities. Parse, split,

1:59 extract, edit, and classify. Um now a quick show of hands. Who here has heard of Jmail or looked through Jmail before? Okay. Wow, that's a that's a good chunk

2:11 of people. Um, but I'll I'll give a quick rundown in case you haven't. So, Jmail was originally built by Riley Walls and Luke in five hours last November. That's when the Epstein files

2:22 were released. Uh, the concept is basically Gmail forked and you can log into the inbox like your Epstein and then that grew into a full suite. So you can think of like J drive, JFlight, J,

2:33 Amazon, J photos and so much more. Um by February there were over 450 million visits and 18 million users that had access Jmail. And we're really proud to share that Reduct was behind parsing all

2:46 of the documents. And that's not a small number. It's over three million pages of redacted PDFs, scanned documents, and messy government files as well. And to give you an idea, like this ranged from

2:57 anything like really complex forms and tables to like messy handwriting of like so many different people. Um, now I'm going to hand it over to Omar, who's one of our founding engineers and worked

3:08 very closely with the Jail team back in November. Um, and he'll give you a technical rundown and more. >> Alrighty, who's ready to talk about some documents. Yippee.

3:20 All right. So, um, yeah. So, uh, walking us back to, you know, a very fateful night in November, right? Um, I was talking with the founding team and we were trying to

3:30 figure out, you know, what would be a great way to use Reduct with this kind of transformational cultural event that was, you know, going to be unveiled to the public, right? Um it just so

3:44 happened that um you know we're very close to Luke uh who is a founder at Kino and is working downstairs and we were all just kind of talking about you know what would be cool to build what

3:54 would be funny. Haha. Um and then uh it kind of snowballed into this idea of like what if you could um you know sign into your email as if you were Jeffrey Epstein and view um you know your email

4:09 as if you were him. And so um you know it was a really really cool concept as a product but there was one really hard problem to solve that you know was basically a barrier between us and the

4:21 productized solution. um and you're looking at part of it, right? So, it's really really like I I would urge you if you haven't done this before to send this like uh you know uh cover

4:34 sheet to Chad GPT and try to get some results out of it that and and just see how accurate it is, right? It's a very very complex problem where you know layouts are um pretty hard to

4:46 understand. The reading order of this is also somewhat subjective, right? Um, and so we needed something to basically solve this bottleneck so that we could work on all of these cool uh little, you

5:00 know, products downstream. And so this is where Redto came in. So I'm actually going to show you guys um what we had used for a couple of our products, right? So Oh yes, let me do this real

5:13 quick. Alrighty. Whoops. Gonna bring this over. All righty. So, here you see a cool product, right? JF flights. Basically,

5:33 all of Jeffrey Abstein and his affiliates flight history, right? This guy was flying people out. He was flying diplomats out. He himself was flying many places to and from his island,

5:43 right? Um what was basically the barrier between us and finding out finding all these flights were these pretty shoddy handwritten documents, right? Um again, if you were, you know, if you thought

5:55 that cover sheet was a little bit rough, think about handwriting, right? So we can see here a pipeline that I had actually set up way back when uh when we were trying to

6:09 build a good config to basically make our parsing really really good, right? And so in this case uh we had for the for the flights use case, you know, an idea had flown out that like what if we

6:22 um you know spin this off as a completely separate product where you could just see all of his flights. Um and so what that meant is taking all of his flight documentation and then

6:30 applying a separate you know extraction config for it. And so what you'll see here is um kind of a little tour as to how Reduct works, right? So Reduct uh within the document extraction use case

6:43 first parses the document, right? And so this kind of shows you, you know, you could stick any document that you would like here and you could just click run, right? Configure any of our bells and

6:52 whistles and see real-time results, right? So this was sort of the canvas for myself that I was playing around with to kind of understand before I start, you know, scaling this up to a

7:03 million different documents and run up a quick check on Reduct's, you know, on Reduct's uh um you know, uh receipts, let me make sure that our um extraction accuracy is what I would expect it to

7:16 be. And so here is kind of what we ended up with. Um you could see here for a couple of these different documents, you know, some redactions that we had to deal with. Um Woody Allen, I know that

7:28 guy. Um but yeah, just like a lot of different um you know, use cases that we had to handle through one extraction schema um for the flights use case, right? And so we parsed all the

7:41 documents uh very quickly. And actually a fun story about this um is Alvin in the room? Alvin is in the room. So, um, back when the, uh, the, uh, first, uh, drop came out, I was paging Alvin maybe

7:57 like, you know, a couple times a night, uh, he was, he just happened to be on call, uh, the night that I was parsing all these documents. And we were trying to parse them very, very quickly. And

8:06 so, we put a, you know, a decent amount of stress on our infra. And so, Alvin was sending me a lot of fun messages late at night like, "Hey, bro, can you slow down?" Um, but yeah, we got through

8:17 them. We got through them in time. And then um once we had this kind of corpus of documents that we were able to um you know um kind of take as a ground truth, we then spun off a a billion different

8:30 products, right? And so Paula kind of touched on this a little bit, but I'm going to go back to Gmail. So, you know, we we spun up any anything from Jazm to Jlights to Gemini, right?

8:46 Um yeah. So, and I and also JCAL. This was actually a product that I um worked on myself. So, some fun stuff there. Um cool. And so, I guess you know if you could kind of take uh anything out of

9:01 this, I do want to say actually um yeah, so if you could take anything out of this, right, like the simplicity from this is really like you know

9:13 getting a schema, right? looking at these results, making sure that the results look like what I would expect, and then just deploying. Right? So, if I was to deploy this code snippet, I'll

9:24 just redeploy here for our use case, [snorts] and I can get either a direct API call or just a pipeline code snippet that I can invoke very quickly, right? Um so

9:35 the name of name of the game here is like stick your documents in this platform and then immediately start um you know visualizing your data and you could immediately also productionize it

9:46 uh afterwards. So I'm going to stop here real quick. Um do we want does anybody have any questions thus far about um Jmail or Reduct in general before we can kind of start uh working on a little

10:00 interactive exercise? >> Yeah. Yeah, >> great question. So, we can actually come up with this for you, right? Um, you

10:14 know, my uh my fun analogy here is that like your extract schema is sort of like the paintbrush on your reductto canvas, right? So, it definitely drives everything that you um you know, you get

10:27 out of the schema. So, for example, right, I could create a new pipeline here, right? and create a new extract pipeline and just call this I don't know let me make a resume schema because I

10:40 want to parse some résumés right >> oh whoops sorry about that actually I'm going to do this real quick so anyway I I'll do this again because it's very simple

10:56 so I'll create a new extract pipeline here by the way we could do a lot of things with documents par split extract, edit, even classify. Um, and so what I'm going to do here is

11:09 create a resume schema. Oops. Uh, my Wi-Fi is not being helpful with me today. There we go. >> All righty.

11:23 So, I got a fun schema here. And now it's up to me to basically decide whether or not this schema is like satisfactory or not. Right. it it's really given it's going to give me

11:31 exactly what I'm asking for and you know once I plug a document in here I might realize like ah this is actually not what I want right so I have a bunch of handy dandy resumes I'm going to stick

11:42 them in here upload and then just run this so yeah um I hope that answered your

11:56 question any other questions while We wait for this. >> How do you validate? >> You're talking about the loading state.

12:17 Uh that's just like a fun loading state at the end of the day, right? Like our product is like a really, you know, it's a suite of different models uh and different VLMs that we deploy, right? Um

12:26 I'm I'm >> sorry, it's not the I was following what you were saying. >> Yeah. >> Oh, that's a great question. So, if I'm

12:40 understanding correctly right, what you're saying here? >> Yeah. Yeah. So, this specific um this specific like schema generation that I had just used is really out of the box

12:51 and it's just kind of like a a quick LLM call that describe like builds a JSON schema exactly for what I asked for. However, we do have an enhanced schema generation, right? And so what this will

13:02 do is let's say I've parsed my documents, it will then take the output of like what's in the documents and tailor a schema description directly for them. So like if you're looking for

13:12 example for, you know, something just really quick to just get up on the screen and just see some results, I would recommend regular, you know, uh schema generation. But if you are a lot

13:22 more interested in like nuance, uh I definitely suggest parsing the documents first and then kind of passing them in. All you would have to do is just click enhance schema generation. And what we

13:32 will be able to do is um you know run the documents first, try to get an understanding of what's in the in the documents so that we can tailor these schema descriptions directly for that.

13:43 Make sense? >> Cool. Cool. Cool. Cool. you had some reduction. >> Yeah. [clears throat]

13:54 >> Correct. Yes. So I mean yeah like at the end of the like this is a great question. Right. So reductto actually parses PDF metadata as well right. So some like sometimes uh and this is

14:06 honestly testament to like uh who was doing the redaction, right? Sometimes the redaction was actually not that great, right? And so what that ended up being or the responsibility ended up

14:15 being on us as the JML team to then have to do a bunch of redaction, right? Because you can't just stick a box on a PDF and then expect it to actually be fully redacted, right? Um yeah, seems

14:27 like there was some uh a lack of understanding when whoever who whoever was cooking up that redaction. um I will say it's like for the files that came out and myself looking through

14:42 them that's not the case right some files were actually redacted correctly um but there was some files where we were just able to see who was like you know who the redaction was behind and um

14:54 so it became a really really serious you know situation right because we're dealing with you know people's data that like um you our data is redacted for a reason, right? And uh you want to really

15:05 make sure that you're not the one that like ends up getting sued at the end of the day because of this. Um and so yeah, there was a you know a a heavy kind of level of responsibility on our shoulders

15:16 to make sure that whatever was intended to be redacted was actually redacted. Our our our tool was working a little too well. >> Cool. Yeah. at the front and then I'll

15:26 after you >> questions or thousands of different things. >> Yeah. Yeah. Yeah. Uh it's a great question. Um

15:42 I'll give you some like you know an example, right? There was there was a schema that I was looking at maybe on Friday uh that had like 156 fields, right? um and all of these different

15:52 fields, you know, need like they were requesting something completely different, right? Uh and so I urge you to kind of put our schema to the test. I think like our um the like nesting uh

16:07 limits to our schema or maybe like four or five layers. Uh but in if you want to, you know, ask for everything in the world, you can still technically be able to fit it in in a schema.

16:19 >> Do you have any capabil resolution >> entity resolution. Can you give me an example? >> So this is a good question. Um there is

16:33 a lot of post proc postprocessing that we end up doing, right? Let's say like on the extraction side. Um I'm personally a little bit farther abstracted away from like the ML side of

16:42 things. Uh but I can tell you that we are doing a decent amount of post-processing to basically link um you know entities together. For example, we have like a citation model, right? Where

16:52 we know we have, you know, one model that could be responsible for extraction, right? But then there might be another model that's also responsible for once this data is extracted, where

17:02 is each piece kind of extracted from, right? Uh Nvidia actually did a cool um paper on this uh that they that I just found out about like yesterday. I think it's called like find everything or

17:13 something like that. um where they try to address the bottleneck of like you know citation uh determination and like the speed at which you could find a citation for something that's in a

17:24 document. [snorts] If you're interested about these things, we are hiring and so uh come speak to us after. Sorry. >> Questions. >> Yeah, absolutely. So, let me try to see

17:45 if one of the résumés I uploaded has an image in it. >> So, let's see here. Right. So, um we have some figure models, right? that are basically determining. Let me actually

17:56 also rerun this real quick so that it can return figures. >> Ah, that's a great question. Um, Alvin, did you hear that? >> No, no. So, I I'll explain. The reason

18:24 why um I I asked Alvin is because Alvin worked on our hybrid OCR model, right? Um and so we have a specific OCR uh like you know configuration that you can enable here that will pri figure out

18:36 kind of intelligently what to prioritize, right? Um so in some cases we will actually pick out like just you know strict OCR data or we might pick out like um you know metadata from the

18:47 PDF. Um yeah we've done a lot of work here. Uh and it's it's not a simple sol like it's not a simple answer. Um this is something we could do an entire different talk on.

18:59 >> Yeah. So I'll be frank with you like this is actually exactly what this pipeline is doing, right? So like we first parse the like the Hold on, let me turn return

19:33 figures. Cool. So, um, while I rerun this, what we're doing here, let me pick a different example. Ah, actually, we're gonna have to wait now. But what we're

19:46 actually doing in like extraction is taking the parse results and then feeding them into, you know, some extraction pipeline downstream. So, like parse is kind of like the um, you know,

19:57 it it is it is what made reductto reductive in the first place, right? And then we can do a lot of cool things, you know, downstream of reductto. So uh you know once we have that parse content for

20:07 example right we could split you know the document into different regions and figure out like if I'm parsing some you know a math textbook I want to understand like where exactly every

20:18 chapter you know starts and ends and then for each chapter I might want to do like an extraction based on that right and uh get you know the chapter summary or something like that right uh so parse

20:30 is really the kind of foundation for all of these other agentic um document processing u features. >> Man, if I answer that question, I might lose my job tomorrow. [laughter]

20:48 >> So, um I mean it's a Yeah, continue. >> I think it depends on like what pro data we collect um as well. So we specifically um train on like certain types of data. If we find that it's

21:04 doing poorly on certain types of data. So we see that like long context recall is not great. Then we got to like try and fix that. Specifically for extract um it's it's not really like Neil in a

21:15 haststack. It's like all the needles in a haststack if you think about it. So that's like a really um challenging like research problem that our MLES are like somehow able to figure out pretty well.

21:26 But yeah, happy to chat more later. >> Yeah. >> Oh, we've been neglecting you guys. Yeah. What's up? You you you'll be Yeah, you should you should stay tapped into

21:36 our uh like our our socials and you'll find out about this very soon. I think like benchmarking is you know always a topic of discussion and something that we want to really stay on top of

21:47 especially because you know somebody who is testing who wants to do document extraction might not you know know where to go or who's better right without actually testing themselves. Uh so

21:58 that's why benchmarking is like something that's you know we're we're prioritizing a lot uh especially in the coming months to to try to show people like how well we are doing. you know, we

22:07 try we're we're very confident in our models and us being like, you know, the premier document extraction solution on the market. Uh but there need to be uh more, you know, um data points that we

22:19 can rest on and that the, you know, entire industry can kind of see at face value uh without having to go test themselves or pass in the all entire ARC AGI data set to you know try to

22:30 understand uh you know how well reducto is doing against another solution. >> Yeah. So we're going to release some benchmarks soon. But the real deeper answer here is you typically have to

22:41 make your own benchmarks unfortunately. So every single one of our customers typically have some benchmark. They're like, "Oh, we're evaluating document providers. Uh we're going to run some

22:50 head-to-heads uh of like reducto versus like all these other like use cases and you know sometimes they don't use like maybe our APIs really well because they use the wrong config. So we work closely

23:00 with them like hey turn this on turn this on you get this trade-off and then uh we let them decide um and then like we often like win these headto-heads in like performance. So as with most

23:14 projects you know gather a simple data set you can do some like vibe eval just look look in studio see if the outputs make sense and then like as you want to scale it out then get like 100 documents

23:24 a thousand documents do something more systematic and you can also work with our team where we have a lot of people to you know work with customers to help you do your emails but yeah

23:34 >> cooly um any more questions yeah sure well yeah absolutely right um I I think so I'll answer this uh pretty honestly right I think when I was working on this

23:47 solution right or I was working on this problem I was trying to figure out like what is a good productized way to you know surface our reducts parsing right um and to be frank with you right like I

23:59 came up with something that was pretty shoddy in the beginning um like basically a you know rag chatbot application where you can ask anything about the emails right and it might be

24:09 something that would be really really useful for like a journal journalist, right? However, um I think it's testament to the hundreds of millions of people that you know were, you know,

24:18 using Gmail and and the tens of million that still, you know, log on uh from week to week nowadays. Um it's it's testament to the fact that a nice product on top of a really really nice

24:29 novel, you know, uh document extraction solution is what is going to kind of be a onetwo punch here. >> Yeah. Like would you read three million like scan documents? Honestly, no. And

24:40 honestly, when I saw the drop, I was like, "This is like way too much work for me." But once Omar built it, I just like logged on and like started searching like random like famous people

24:49 for fun. And so that's the beauty of it, right? Like I think like unstructured data is like everywhere. And sometimes all you need to take all you need to do is like to extract it and then like do

24:59 whatever you want with it. It's basically what all our customers like do um conceptually. So I think Gmail is a pretty fun example but it's actually very u it's exactly what happens in the

25:11 industry. >> Yeah. You seen that? Come here. It's it's just hard work, right? Like you know we we have a data labeling team, right? We have to go like par you know

25:21 like we have to sift through a lot of documents if we're going to train a new model on top of it. Um and be very very certain in the outcomes of you know or or the the sanctity of that data, right?

25:31 Um and so the short answer for you is just [snorts] like it's a lot of hard menial annoying work, right? Uh but that's really what comes into like the beauty of a machine learning model at

25:41 the end of the day, right? There's nothing that you know really beats that like really really hard, you know, just wrote classification work uh that then compounds into uh model training and

25:53 then you know then you can kind of figure out like how to stagger your data and like how to train you know on on top of your training set. At the end of day, at the end of the

26:00 day, you still have to label that data. >> Cool. I think we'll take Yeah, we could take one more, but um Yeah. Yeah. So, to repeat the question uh and let me know, you know, how how accurate it was,

26:12 right? But, uh it seems like, you know, like there are cases in which uh you need to really customize your logic on top of different documents. Uh whether it be corresponding to specific models,

26:23 right? Sometimes there might be metadata set up in specific ways um in specific documents and you want to really like uh condition um for these specific use cases, right? Um this is a fantastic

26:36 question, right? And so this is where our agentic um section really comes in, right? So you can actually apply all of these different, you know, uh, configs and apply like, uh, system prompts on

26:48 top of separate models that will do specific processing that's tailored directly to what you're asking for. So for example, right, if uh you want to do some complex figure extraction, let's

27:00 say I have a bunch of logos and I want to try to determine like you know what differentiates like I don't know uh some you know uh Lincoln Financial from like Google and like how to separate those

27:12 kinds of logos, right? I could add some additional color here within our uh you know prompts uh in this config to custom kind of tailor u the outputs that I'm looking for. So if you wanted some

27:25 specific metadata here, let's say like uh you know some layout might be like really really hard to read in uh I don't know like a construction blueprint, right? I would spend a lot of time in

27:36 the uh layout prompt kind of tailoring to try to understand uh how well you know uh my current extraction solution is mapping across all of my different um you know construction blueprints I guess

27:48 right cool. Um does that answer your question? Fantastic. All right. So now we want to kind of um spin this off um into an interactive exercise. Right. So, um, if you guys would like, we're

28:05 going to basically give you guys, you know, um, however much time you would like. I think we have about 30 minutes left. Um, and you could sign up today at, uh, at studio and essentially, uh,

28:17 just get started, uh, on some document extraction solution, right? So, what what you'll see here in a QR code is something that you could basically paste directly into a cloud code or a codeex

28:26 or whatever, and something you could just get started uh, on uh, document extraction with immediately. Uh so we're going to give you guys about 30 minutes and by you know maybe around like five

28:36 minutes uh like at the five minute remaining mark we'd love to kind of see what you guys have cook uh cooked up and we have a bunch of cool swag and a couple of cool prizes for you guys uh to

28:45 just kind of show us what you got um with respect to document extraction. Um it could totally be just like an idea um and like a singleshot prompt but uh feel free to just kind of uh give it a whirl.

28:57 >> Sound good? Also, we'll be kind of like circling around if you guys have any questions as to how to use Redto. Uh, I want to come work for you guys, you know, I want to hear about your models,

29:06 how you guys train, whatever. Um, we are happy to answer. Sound good? >> [music]