From Raw Documents to AI-Ready Data — Leo Platzer & Jeff Koss

0:00 All >> [music] >> right. Hey guys, let's get going. Can everybody hear me? All good. Perfect. All right.

0:20 My name is Jeff CS. I'm a staff customer engineer with DZ Labs. Um, as you can see from the slide, it says brought to you by Calibbra. Calibbra acquired DZ Labs last summer. Uh Calibbra is more of

0:33 a data governance tool for structured and AI um agents and models. [snorts] So the gap that Calibbra had was we didn't have unstructured and that's really where uh DZ shines. So today uh with me

0:46 is Leo Platzer. >> Yeah. Hi everyone. Uh I'm Leo. I was previously the CTO at DZ Labs which as Jeff has already mentioned was acquired last year by Calibbra.

0:56 >> And you can see we're both twins as well. We have our hats that we're giving away at our booth as well. So come by and get them. First come, first serve. So today we're talking about um how we

1:07 can go from a stalled PC to production with DZ data curation. So really focus on unstructured data. So our demo scenario I typically talk about is a manufacturing uh company,

1:20 North River Manufacturing. You can see that you know there's 40 40,000 employees. They operate in 20 different countries. Uh really what they're trying to do is they have millions of documents

1:30 in SharePoint. They've accumulated them over years and years and they need to figure out which ones are relevant for their chatbot. They're invested heavily in AI to improve operational efficiency.

1:41 And their big objective this year is they want to build a legal ops AI chatbot. Now what they want to do is they want to help the legal and procurement teams

1:51 answer questions around supplier contracts, partnership agreements, and it's going to be real time. So some of the questions they're trying to answer or what are the termination clauses for

2:02 a specific supplier? Um which agreements expire this quarter, which agreements expire next quarter, and what partnerships have u exclusivity terms. So some of the folks in the teams that

2:16 we've been working with um is Sarah. She's a head of data and AI. So she's responsible for the accuracy. Um also there's Jim. He's the AI engineer. He just needs to know which files do I

2:28 need? I have tons and tons thousands and thousands of files in SharePoint. I need to find just the relevant ones with no sensitivity. Priya is the head of knowledge

2:38 management. She wants to make sure that no sensitivity is leaked. um and make sure that if there if there is specific data that's uh classified that it gets redacted as well. Then there's Michael.

2:52 He's a director of legal ops. He needs a chatbot to deliver accurate responses to his team. So he's from the business. And then finally there's Jackson. So Jackson's enterprise transformation uh

3:04 lead and this is his flagship use case but also he has 15 other use cases that he needs to to deal with right so he wants to have a framework a repeatable successful process he doesn't just want

3:19 to do it for one time he wants to do it for multiple different use cases so what they're able to do so far they were able to take 40 around 40 different

3:33 files pre-curated files and build a small little PC small little mini pilot where they can do uh parsing chunking OCR embedding and then put it in a vector database and a knowledge graph

3:47 and they got pretty good responses right seems pretty easy you can do that with 40 files but when they try to do it with many many different files with 80,000 plus

4:01 files. Right now a sudden it's a little bit more challenging. And so some of the challenges they had were I can't discover the files that I need. I don't know where the sensitive data is in my

4:15 on SharePoint. I can't trust the quality of the data. So my background um I used to work at both Informatica and data bricks. So for me data quality is typically the six different dimensions

4:28 of data quality. Completeness, conformity, accuracy. But when you deal with documents, it's a little bit different. It's I want to see is there duplicate information, conflicting

4:38 information, right? Leo's going to talk about he has a really good slide about showing why it's so important and also the freshness of the data and also everybody's talking about context,

4:50 right? So what we want to do is when we tag the metadata, we want to use that in our chatbot in our vector database to make it more relevant, have better accuracy in the chatbot.

5:04 So you can see some of the challenges that they had, right? They're getting incorrect answers. There's sensitive information that's being leaked out in the chat bots. It takes a ton of time to

5:16 go to production because they can't find the files. is spending too much time looking for the files instead of writing code and doing things that they they really like to do, right? It's difficult

5:25 to maintain. I'm working with another telecommunications company and they have six to seven different SME and they're trying to curate 20,000 HTML files and they can't keep up, right? So, it's very

5:38 brittle, very difficult. And what we do is we provide the ability to automate this entire process. So over here now you can see they want to they want to try to access and

5:52 leverage over 80,000 different files. So they went from 40 files to over 80,000 files. And really what they want to do is they want to find which files are relevant for their chatbot, which have

6:03 high quality, which ones are up to date with knowledge. They don't want to use a a version where the contract is is old and expired, right? and they want to enrich their chatbot with the metadata

6:16 that they've tagged and of course they don't want any sensitive data and so what we do over here is we provide the ability to do that right we can access the data um over here we can

6:30 understand what the data is we can figure out which files are relevant do the OCR the chunking and things like that so let me show you a little a brief demo

6:41 So over here you see in SharePoint much like what North River has they have a whole SharePoint site full of different you know there's call transcripts there's PDFs there's excel there's word

6:53 documents and I they don't know which ones are relevant which ones are focused around legal now over here when we look at DZ we've already opened up the application right here and I can see you

7:05 know I've already ingested 339 files and now I can see both the uh page page level or chunk level as well as a file level. I can see all the information right here.

7:18 And then over here, this is what we call a texonomy. Taxonomy is a grouping of tags or a tagging tree. So I've ingested the 339 files. And now I can either manually create a userdefined tag. I can

7:31 leverage tags that are reusable for my tag library or I can leverage AI to autosuggest what tags I should even have. Right? I can leverage AI to say you these are the tags are relevant. So

7:42 here I just type in my tag name data document categorization give it a prompt and this prompt is what we're going to send to the LLM to determine is this file how should we tag

7:54 this file now if I know what I'm looking for I can put in the classification values it's optional and again if I can leverage AI to autosuggest all these tags for me

8:08 now also the other thing to point out if I have my uh my taxonomy here, my valid values um already. I can upload them through a CSV. And so once once we do this now, I can see I started to build

8:20 this this tagging tree. Now the other thing I can do is I can have conditions on my tagging tree. So remember I said I only want legal and contracts. That's what they're looking for. Now what I can

8:31 do is I can have have um subtagging. So only legal and contract files. I want to pull out more data around the contract type. Is it a supplier contract? Is it part of an NDA? Right? Is it materials

8:46 and and time contract? And now what we can do when we add this, you can see we have different branches in our tagging tree. And I could have um unconditional tags

9:04 as well. I can have any number of different tags. Now, what we're doing is we're going to generate the metadata. So, I choose what files I want. I can choose what tags I want. I can say

9:14 generate metadata. And we'll run this. And you'll see the results over here. So now I can see the document category. I can hover my mouse. There's

9:26 engineering documents. There's my contracts and legal, right? I have 68 different files. And if I click on any of those, it automatically filters for me.

9:37 Now I can also see the evidence. Why did AI tag this? Where in the document did it find evidence that it's a legal contract? Right? And I can also do thumbs up and thumbs down. So there's

9:50 fshot learning as well. So I can give it an example if I want to. Now you can notice I also have filters. So I have two different filters. I quickly went from 339 files down to 20 files.

10:04 So, we also have the ability to scan for sensitivity as well. So, whether it's PII, PHI, um could be anything. I was in uh in London at a Gartner conference uh a

10:18 couple months ago and there was some folks from Finland that came up and they wanted to know, you know, can we find Finnish national ID? I don't speak Finnish. So, I put it in there. I did

10:27 this demo and not only can we do AI based, we can do patternbased with context and it ended up figuring out the the reax um syntax for me, but it also added some Finnish words which I thought

10:40 were probably associated with Finnish national ID. So, it's going to be less false positives as well. All right, let me keep going here. So the other thing we can do is we can do

10:57 quality. So again for us we have a dashboard in here for quality where we can go through and see what are the different groupings for um for duplication for conflicting information.

11:09 Um and then we'll we'll leverage AI to auto suggest which ones to keep. Now the other thing we can do too is again for freshness we can say I only want files within x number of days or within a

11:22 specific date range and then the last thing we can do for data quality is you know here's the the pieces of metadata that I absolutely have to have and then what we'll do is we'll figure out you

11:34 know of the 339 files maybe you really need 200 files and then from there you can create a data slice and then we can also give it context as uh over here. So I'll go through one

11:48 more video and we also have a live demo for you that Leo is going to be doing in a second. So in this case now what I can do is I can I can leverage a a data slice. So my data slice could be

12:00 filtered based on the specific criteria that I want. So in this case I have 14 files. These are only the files that are relevant for legal. And now what I'm doing is I'm copying my use case

12:12 description. What I can do is I can say here's what I'm trying to build and I can either do a flat uh tonomy or multiple layer levels in my tagging tree and then over there when I run it it's

12:24 going to autosuggest these tags for me. There's human in the loop so I can choose which ones I like, which ones I want, which ones are relevant. But again, it's all done for me based on

12:35 what I'm trying to do. We gave it context and based on the content of the files themselves. And then over here it even builds out the classification values for me.

12:50 And then if we want when we run when we generate the metadata we're the job now we can see over here's all the sensitive information right here's all the different information that we have over

13:02 here. And then again I can see the evidence at the file level and the chunk level. I can see the confidence score. So coming back from from AI what the model return

13:13 I don't have to guess you know it's 100% confident 95% confidence. Here's why. So you can have very a lot of confidence in the the um AI ready data product that you guys are going to be building.

13:30 All right. And now finally what we can do uh for delivery and maintenance. So you know one other thing that that's really important is with our data slice

13:41 technology we have workflows where you can put on a set schedule so it's auto refreshed. So we want to make sure that your chatbot never gets stale. So if there's new records or sorry new files

13:52 that get added to SharePoint or deleted, we'll automatically pick those up and add those on a set schedule to my um uh to my data slice or my AI ready data product and then our SDK will pick that

14:05 up in the chatbot and then when we load the Victor database, you'll always have the most freshest uh metadata. So what's it mean for North River? So basically they were able to go from four

14:18 months of data prep work to down a days. They were able to increase their data utilization for AI. They increase their chatbot accuracy because they not only were able to figure out which files to

14:30 use but also the metadata from the files. They were able to inject that and leverage that through our SDK in their chatbot. Right? And then the other thing they did was they reduced their

14:41 compliance risk. they knew where all the sensitive data is and they made sure that they were not part of they were all filtered out in their u data stores. Okay, so I'm going to turn it over to my

14:53 brother from another mother, Leo, and he's going to talk about um context. Yeah. Hi everyone. Um I just wanted to basically like give this so what right technically okay you dduplicate your

15:08 data, you reduce your search space, right? you clean it up, you have it fresh. What does it actually mean? Right? So, if we were looking at the vector space, right, when you ask a

15:19 question, it comes in and you have duplicated data with the same information or potentially even a different version of the document with conflicting information now, right?

15:29 Think of your Slack messages. Today, a person says this, tomorrow a person says that, right? The truth changes over time. If it ch if it stays however in your knowledge base in your search space

15:42 your top K will always be full of pretty much irrelevant information right so in DZ for example something which is considered duplicative is either another piece of context like

15:55 your database has changed and I can actually use it or it is simply redundant and we will remove it from the knowledge base. The second piece is like same as same

16:07 reason as before. If your information is old, right? If you suddenly are working with data from before the 2000s, if you're working with information which was only relevant last week is not

16:18 relevant this week, in the future some of the use case when agents actually get into production for let's call it the most high stakes use cases that has detrimental effects.

16:30 And we have done um a lot of evaluation around what is the effect of making sure that your unstructured data quality is maintained, is consistent and is always handled deterministically in the same

16:46 way, right? And what we're essentially seeing is that if your raw corpus, okay, let's imagine you have a thousand files and 30% of your files are stale or

17:01 duplicative, it can happen that up to 80% of your actual context your agent uses to answer a question is filled up with stale information. which means like it actually uses 80% of

17:15 completely redundant useless information and knowledge and this has to be managed. Um what we have seen is that by running on the exact same tasks. So this is a rack chatbot. It's not um running

17:31 an actual like let's say cloud code task or codeex on a coding task. It was simply like multihop rag evaluation. And what we saw is that if we look at recall one like I mean we pretty much

17:44 double uh the recall um and overall improve our accuracy. So like task completion by between 10 and 15%. And why is this so significant? This is so significant because this has nothing

18:00 to do with the use case you're working with. This is true independent of what you're actually using the data for. Right? Because if you actually know what you want to use your data for, you can

18:11 apply relevancy and all not other data quality dimensions, right? But duplication and freshness is true no matter what. And

18:21 yesterday, right, there is a lot of context topic, a lot of context flags around here. Um, I wanted to give a very very quick demo on how you would use the DZ SDK to prepare a context repository.

18:36 And I'm going to run this live. So what this is doing is essentially it's going to fetch all of the metadata we have about a given SharePoint site and we will then contextualize each folder

18:49 within that SharePoint site as well as create an index over all folders so that your claw code for example or your codeex does not even have to look at the individual files to figure out what to

19:01 look for but simply looks at the context MD file. So what you're seeing here is essentially really by using DZY you can curate

19:13 context files where you are directly aware what exactly is within each folder right what is its purpose right when do you want to use it so for example this is one to one the mirror of what I

19:30 actually have in SharePoint but now instead of actually looking at these 23 files figure out whether I even should look at this folder. I know exactly what

19:39 document types I have, what it is about, what are the key topics, when to use this folder, and what are example questions. So this is how this is all the kinds of

19:51 use cases today, right? How you can use the easy for use cases like your rack chatbot or use the easy to actually improve the performance and reduce, for example, the token cost for your agents.

20:03 Um, but yeah, basically I'll move over to Jeff again to summarize this whole presentation. >> Thanks, Leo. All right, so guys, um, we're almost out of time. Our booth is

20:18 way in the back by stage two. Please come by, bring any different use cases that you guys have. Um, we're happy to have discussions, do live demos, um, talk about what your challenges are

20:31 around unstructured documents. um over here. Um before we leave, just want to let you guys know we have oneweek PC's. So, we're happy to prove our technology. Uh again, you know, you guys bring your

20:44 your business problem. Um you're trying to solve with AI, bring some sample data, a bunch of different files, and then what we can do is we can get generate metadata for each different

20:54 files. Um we could get a full sensitivity report on your on your data, data quality report as well. and then we'll show you how we can build a full tonomy based on your specific use cases,

21:06 your specific files. So with that, everybody, appreciate you guys spending time with us. And lastly, we only have one more case of these hats, so come on by and grab them. So we'll be out in the

21:18 back. Thanks, guys. Appreciate you.