Image Decision Models for RPA: Forms, Scans and Screenshots

0:00 Okay, if you draw any business process as a flowchart, you get two main shapes. The rectangles are the steps. They're things like copy this, click that, file this, and the diamonds are the

0:13 decisions. Does this look right? Is this an invoice? Is this a payment? For years, RPA tools have been automating the rectangles. That's robotic process automation. software bots that click

0:25 through screens and copy data between systems the way that a person would, usually following fixed rules. But the diamonds still often go to a person because a rule can't look at a page and

0:37 tell you whether it's got a signature or not. Now, people have had a lot of success at using deep learning to do this. But generally, the challenge there has been that people have to use custom

0:46 models for each particular company and for each particular business use case. But it's turning out that this is another place that a very specific kind of Jev model can really have a huge

0:58 impact. So what I'm going to show you in this video is that I built an inspector that basically looks at forms, whether they're PDFs or images, and then runs them through two small open models that

1:10 make decisions straight from images. These are literally open jev models. And both the models are really high up on the image benchmark. In fact, the one that's at the top here was built by one

1:23 person in 15 days for just $1,200 of rentered GPUs. So, I know I've made quite a number of videos about the Jev decision model concepts and I've made videos of showing you how you can use it

1:34 for agents and how you can use it for text decisionm. In this video, I want to focus purely on images and purely on the image decision models. Remember, this is something at the time that I'm recording

1:47 this that even the Jev model itself can't handle images. So, in this case, we've really only got open models to try these things out. All right. So, what's the problem that this is actually

1:58 solving? If you've automated anything in business, you've hit some kind of version of this. You've got some kind of bot that can fill in a web form, can click and submit, but often you don't

2:09 know if it actually worked until someone checks the actual screenshot. You've also got examples of where a customer uploads a photo for a refund claim. Does that photo actually show what they're

2:19 claiming in it? Well, someone has to look at it. If you're looking at invoices or contracts or forms, you've often actually got to go and look at that to see was it actually signed. And

2:32 every one of these is one of those diamonds in that flowchart. It's a yes or no call that a person or a deep learning model will actually make after looking at something. Now, automating

2:42 all of these kinds of things has been a huge industry. It's got its own name, RPA, robotic process automation. If you look at some of the biggest companies in the space, you see ones like UiPath. And

2:55 so, I first came across UiPath in 2018. And at the time, what they were doing looked really impressive. They were certainly raising lots of money. And sure enough, by 2021, they had gone

3:06 public with a valuation around 3040 billion. Since then though, it's kind of collapsed. If you look at their stock price has dropped from the high7s right down to sort of $134 a share now. That

3:21 market cap nowadays is under $6.5 billion. So the biggest question here has got to be what went actually wrong here. And the thing that I found from talking to people who actually worked

3:32 with UiPath was that while the promise of full automation was really awesome, the models never actually worked often like they were promised. And to be fair, I don't think it was necessarily that

3:42 UiPath was making a really bad product. It's just that the type of deep learning that people were using for these kind of models really required a lot of fine-tuning for the specific use case.

3:55 they just didn't have the general universal classifiers to be able to do a lot of these tasks with a high level of accuracy. So, as I started to see people make these image models, I was really

4:07 curious to see is this something that could be applied to RPA. So, for the demo here, I've basically picked forms because pretty much every office has piles of forms that they're either

4:19 scanning in or they've got electronic versions of as PDFs, etc. And what I was curious to test out was could we make a demo where you drop a PDF of a form on there or even do it in batch mode. Could

4:33 you drop a PDF of a form and then have it run very quickly to generate answers back about that form? So for things like did the person fill out their date of birth properly? Is the form actually

4:46 signed? Does it have an address for the person on it? Is it a tax form? Is it an invoice? All of these things are sort of stock standard RPA tasks and they were all things that I was curious to see.

4:58 Can we take a model that hasn't been fine-tuned for this specific use case and see how it actually handles this? All right, so this is image bench, a new benchmark for systems that make

5:09 decisions from images. At the top there, you've got image 4B. I can see that that one's definitely scoring really well here. Below that, we've got Jev Omni. Below that, we've got Neo Horse and a

5:22 number of other models in here. So, interestingly, the person behind Image Jeb actually wrote a Reddit post all about not only how they made it, but why they made it. So, they're a process

5:32 consultant who worked on refunds, returns, and customer support. And his point was that in those process maps, almost every decision node ended up being made by a human because just the

5:44 ambiguity is really hard to deal with with code alone. So, I was curious to try out his model and some of the other models for my particular demo here. So, I decided to actually make a demo with

5:54 the first two top models here and see how it actually works out. Now, I know I've made a bunch of videos about Jev already, but those were mostly focused on text. The image open up a whole set

6:05 of new opportunities because a lot of the decisions in business aren't sitting in text. They're in screenshots. They're in photos. They're in scans. They're in PDF files. And that's what I really want

6:16 to dig into in this video. So, the whole idea of these models is that they're basically the diamond that gets the evidence, a screenshot, a photo, a page, and can really just go through and

6:28 answer a lot of different yes no questions based on that image as the context. So, in a normal Jev model, we basically pass in text as the state and then we pass in our typed questions.

6:39 Here we're passing in the image as part of that state, but we're still also passing in our type questions just like before. Now, it's important that I point out here, we're not doing any kind of

6:48 OCR here, right? It's not like I'm OCRing the form first and then passing that in. I'm just passing in a raw image to the model. So, the model itself is deciding what's actually represented on

6:59 that image and then using it to basically answer questions like, is there a handwritten signature? Is a particular field empty? is something filled out with a date there or not. And

7:10 the cool thing here is that it gives us back a probability for every option. And image gev has also been trained to give you back the probability of unknown. So the good thing about having that in

7:21 there is it just helps the model so that hopefully the model is not guessing, it's basically either sure that it's something or it's going to answer unknown. Now, with the responses that we

7:31 get back, we've still got the whole sort of same smart if statement that we had with the text models. It's just now the smart if statements are things that you can actually see. And one of the cool

7:43 things about having those if statements is that you can actually set a confidence score that if it's returning below that, then you can basically say, right, I don't trust the model. I'm

7:55 going to send it to either another model to either confirm the prediction or I'm going to actually send it to a human to look at. Okay. So, if we jump into the demo here, this is basically the form

8:05 decision inspector. So, I'll show you just quickly what it can do. We can basically drag in a PDF and just showing you what this is. This is basically a US tax form. And the text is pretty tiny in

8:19 there, but you can see at least that okay, it's got a name. It's got a bunch of details in there. And in this case, you can see that the signature is actually missing in here. The date that

8:30 person signing it is there. The date of birth is there, but signature in this case is missing. So now, if we come over to the actual questions, you can see we've got a bunch of questions in here.

8:41 We've got things like is the date next to the signature filled in? Right? The postal address is filled in, date of birth is filled in, and then we've got fields for missing. We've got fields for

8:52 what kind of form is this in here. And then we've got some other things in there about things like how legible is the handwriting on this page. So I'm going to run this. I can also change

9:04 things like the DPI. So basically the resolution of the images and what's going into the actual model. I'm running two models here. I'm running the image 4B and the Jev Omni in here. Now I'm

9:16 going to show you some of the issues you run into. So this goes through it basically runs and you can see sure enough we can see that this model has done a pretty good job right it's

9:27 basically said hey is it signed no it's false it's not signed in this case now up here we can basically set how confident does the model have to be before we actually accept the result or

9:39 send it back to a human there because it was basically lower than the 90% that I had set there it's going to say that it should go to a human because I had that set there. More importantly, this one is

9:51 wrong. So, let me come back to that in a second. Dated, they're both in agreement. They're both getting it right there. Address, that's looks right. We can see that this one, one said that the

10:03 date of birth is there. One said that it's not in this case. And we can see that this one now when we asked what is missing, the signature is missing here. Right? Yet up here when we asked the

10:15 signature is there true or false it said that it was there and this kind of highlights the problem that you are going to face with the different models at the moment is that they tend to be

10:27 better at different kinds of questions. So you can see here this model when we ask it what's missing it's saying that nothing is missing right? So that's a choice question that's a classification

10:39 question. the actual sort of true or false question in this case it gets it right. We can see that the response that it got back is going to be like that and we can see the image if we want to there

10:51 in this case it gets it wrong. Now this is where you have to be careful about what model you use and not necessarily that one model is great and one model sucks. They tend to be better at

11:02 different things. So I'll show you just quickly here with this exact same form. If we basically turn signature to be a choice, right? And I run it now, you can see that this one is now saying, okay,

11:17 that it's typed name only. There's no handwritten signature. And also this one now is basically saying typed name only, no signature. So it shows that especially in this model here, it's

11:30 probably better at the classifications than it is at the true or false in relation to this task. Right? I'm not going to say that it's better for every single task at this, but in the relation

11:41 to this kind of task, I need to be calibrating for that early on. And we can see that both of them got it right for a tax form. Both of them said that it was either clear or readable in their

11:52 in some ways probably clear and readable are almost the the same kind of thing going through this. The other thing we can do too is we can actually have a follow-up email. So this is not

12:02 something that's being sent to an LLM to write the email. This is literally just conditional logic. If the signature is not there, then we basically say, "Thanks for sending your" and we fill in

12:16 tax form before we can process it. Could you please sign the form in the signature field? So, that's not using a normal LLM in any way. But it's important to understand that there's no

12:28 OCR going on here. This is literally just passing in an image and then running it through for this kind of thing. And the cool thing here is that all of these we can basically test

12:39 different questions in here. So if I want to edit something is missing could be the date of birth. I can just come in here and you can see that each of these is basically just a string field that

12:50 goes in there. And now I've got a new class that it can predict as well. I'll run it again. Obviously the date of birth was here for this one. And you can see for this model it wasn't phased by

13:02 me adding that at all. For this model it was. We've now got a little bit of things where it's saying nothing missing to okay possibly the date of birth is missing. So you do want to test these

13:11 and work out okay one what are the best kinds of questions for what you're doing and then also which model is the best one for those types of questions. Okay so if I give it a new form now this is a

13:24 form where the date of birth is actually missing the signature is there that you can see in there. So let's try and run this one and see what we get. And you see they're running very quickly. So you

13:34 can see now we're getting handwritten signature is there. Both of the models concur. That's the key thing there. If we go through and we get to date of birth, we can see now both of them are

13:46 saying that date of birth is missing. This one's less sure about it than this one in here. And then again here when we ask what's missing, this one can clearly see now that the date of birth is

13:57 missing. and this model just doesn't seem to be doing great at this type of question. So, I'd probably think about, okay, am I going to rephrase this for this model? Am I going to only go for

14:09 the results with this model here? But the key point that I want to show you here is that you need to be aware of this, right? Don't just go and think, oh yeah, this model is the best model,

14:19 therefore it's going to work for all my tasks really well in here. So, this is just a really simple sort of thing. You can see again we can basically have it do a follow-up email which is

14:29 conditional logic here. It's basically saying fill in your date of birth. It's worked that out really nicely. And we can set this up to actually just use a sort of confidence score to decide if we

14:41 throw it back to a human or not. And that's something that we could set. Now with a little bit more work, we could actually set that confidence score to be different for each of the questions that

14:52 we've got in here. we can add in questions quite easily, right? You can build something where you could put in a choice question, you could put in a scoring question, you could put in a

15:01 null question in there. So hopefully you can see here that this is something that is pretty easy to do with these kinds of models, right? You're not having to train up a lot of models or do anything

15:13 like that. Now, that said, if people are interested, maybe I will make a video about actually fine-tuning some of the open jazz models, cuz that's the next stage that you take it to. If you're

15:23 starting to get some data where you've actually got some decent data, you could actually take some of these models, whether it's the image one or whether it's one of the text ones, and actually

15:34 fine-tune it with a little Laura adapter in there to basically make it work for what it is that you're trying to do. So anyway, hopefully that shows basically how these Jev image models can be really

15:47 powerful for things like RPA, for things like very quick image classification and dealing with large amounts of images in here. We can run a whole bunch of different questions through it and

16:00 basically customize the questions and the model for our specific use case in here. So anyway, let me know in the comments if you've got any questions about this. Hopefully again this shows

16:09 you a real world example of how this kind of thing can be used so that you've got a sense of what you can actually do with this. And as always, if you found the video useful, click like and

16:18 subscribe and I will talk to you in the next video. Bye for now.