Modality Misalignment and Originality Attribution in Short-Form Video — Aditya Gautam, Meta

0:01 [music] Hello everyone. Uh I am Adita and we will be talking about uh two main problems that happen on the short form surfaces. We are going to talk about how

0:21 do we deal with these things at a scale. Uh so to get started first thing is we will understand what are the characteristics of the data that we are trying to deal

0:30 with what are the two main problems that we are actually working on uh and what are the multi- aent systems to solve those problems at a scale what are the different specialized small scale WM

0:42 that we can uh build uh to solve those individual agentic problem and how do we build those agents what are the different ways to actually optimize it like to make it like scalable at a very

0:53 big scale and then we will be looking into the evaluation in a very holistic 360 manner not just like a precision recall what is out there how do we understand the entire multi-agenting

1:04 pipeline both from LLM level tools MCP and whatn not everything out there and then we will be getting into the optimization techniques which this which are very vision specific and metadata

1:15 specific which will allow us and give us some intelligence to tell that we really don't need to do intell like this intelligent like workflow So for all the videos we can figure out what are the

1:25 small set of videos which should be the candidate for doing these things and we will conclude with the takeaways. So the real life data is very messy. It is we are talking about at a scale of

1:37 100 million plus and a lot more viral content. There's a lot of adversal content people are trying to gain the system. There are a lot of uh multilingual text on the screens and in

1:46 their videos and images. And the uh data is very dynamic. it keeps on changing from like month to month different AI tools are coming everything out there so it's it's really really dynamic that

1:57 means there is a lot of drift issues and other thing that comes with a video data and then we know that there is no clear ground truth for the problem that we're trying to solve so these are like

2:06 existing problem which happens in a real world video data set we will be the first problem that we will be talking about is the modality misalignment where the first is intramodality which is a

2:18 very solved problem you can have a clip model you can have a modality on uh uh images, videos, uh audio, text and figure out what exactly is the cosign similarity on those embedding and figure

2:28 it out. So this is a much simpler problem. What we will be talking about how we are doing intraodality u issues. So this is consider that like a video we have a big video and then uh suddenly

2:41 you are seeing something some ad agenda some political things or something out there which shouldn't be there in the video as when you clicked on it. So how do we actually look into the small

2:51 segments of the uh video understand this problem and figure out where there is an anomaly or kind of like adversarial behavior out there which shouldn't be there in the first place. So this is the

3:01 first problem which requires a lot of granular understanding vision understanding and a video understanding to see what is happening at a very clip and a very frame level. The second

3:10 problem is uh to understand the unoriginal content. In today's economy with the AI tools available, it's really easy to actually duplicate the content. When someone uh upload a video or

3:22 something, we see that okay, it is being copied, it is being transformed and with tools, it is getting much easier to transform these videos. U so how do we actually detect this kind of unoriginal

3:32 content? How do we figure out the source of the video? And um this has actually caused a problem in attribution credit u ecosystem imbalance and lot of like user fatigue where we are seeing a lot of

3:44 repetitive videos which shouldn't be there in the first place. So uh coming to the multi- aent system uh the first thing is that uh why we are going to multi-agent not a single agent or single

3:55 lenm because the problem is really complex. It requires a really specialized uh nodes and uh kind of like an understanding at each part of retrieval content understanding and then

4:05 reasoning and if you can solve a problem with one agent one LLM you we don't need to actually do it on multi- aent systems. So this is where the centralized brain how we are actually

4:18 thinking kind of like how the uh decomposition happens of this problem. The first is basically you have a video and kind of like a clip video ID and everything given to a reviewer agent.

4:28 This is a centralized agent. This is basically the orchestrator. Consider that as an API gateway of uh doing a sing signal decomposition and finding what is happening around in the image.

4:38 So what this guy do is basically this guy takes ask the perceiver agent and perceiver agent is the one which is basically considered that a very sophisticated VLM expert with lot of

4:49 image and the video tools at uh disposal. Uh perceiver agent will take the ID from the retriever reviewer agent and it fetches the video from the database. It decomposes into smaller

5:03 parts using different tools, different like semantic embeddings and different kind of like temporal change that happens which is a little bit beyond our scope but of this talk but it's uh a

5:14 technique which is making sure that we are not doing at a fixed frame rate but we are finding where the temporal change happen and compressing those similar frames into one or two frames and then

5:25 the perceiver agent get all the data from a clip level. The video embedding level get all the information about a clip embedding tags OCR whatever is present out there what is the natural

5:35 language description and what are the time stamp from which frame to which frame what are those this metadata looks like it gives that data to the reviewer reviewer will look into all the temporal

5:45 this JSON object provided by the uh perceiver agent in a raw form along with embeddings along with semantic ids tags OCR and everything and it does a temporal analysis it looks into okay you

5:57 know for from first frame to frame or frame number 360 or like till 6 second it was a video talking about a sports and suddenly we are seeing from 6 second to 6.5 second or frame number this to

6:10 this we are seeing that like this is changing into some political thing. So just by looking into the metadata the reviewer agent is able to understand comprehend and figure out what exactly

6:20 is the anomaly coming in the this temporal space and once it do that it figures out whether this is actually a modality misalignment or there is some bigger issue out there on there. So

6:30 review agent will look and talk to the retriever agent and it tells that hey this is the video I'm looking into. These are the some of the metadata that I have already uh kind of like ddup and

6:40 kind of like post-processed. Now give me some understanding of this similar clips which are available in the corpus. So retriever agent will look into all the signals provided by perceiver agent. All

6:51 the metadata of a clip level and a whole video level and index it into different databases adaptively. For example the topics has to be maybe like a inverted index where we have topics and a lot of

7:02 videos. For the embedding it will be a vector databases which is available out there. And for different kind of entities which I've extracted we have a graph databases kind of like where you

7:12 review your agent will figure out okay these are the multiple databases these are the entities and these are the metadata let me index it so that in the online inference time I can figure out

7:21 for a particular clip what are the similar clips what are the similar entities and topics which are available to fetch and improve the recall. So going into individual agent we talked

7:31 about perceiver it looked into the entire video do a temporal decomposition into small clips based on the semantic embedding and couple of algorithms and it fine-tuned the VLM to actually emit

7:42 all the real uh concrete and a very um granular information about like each clips and the entire video. Uh because we are dealing at a very big scale that means we cannot just go with the

7:55 standard VLM which are out there. There has to be a compression. There has to be a cost effective way to actually serve this these models to do it at like billions of frames. That's where we are

8:06 getting into the pre-training fine-tuning and knowledge distillation quantization to actually deploy and solve the very very specialized VLM based on this particular problem and we

8:16 will be talking a little bit brief about that. U the retriever agent is the one which we talked at a very high level and then what it does in an offline processing is that all the signal which

8:26 is decomposed by the perceiver agent in an offline fashion consider that rather an agent you are actually taking that library and doing an offline analysis on a big scale let's say ray cluster or

8:36 something like that and then once you have all those metadata it is basically indexing it into different databases doing an uh periodic offline clustering and finding what are the similar content

8:47 what should be the embedding ID what should be cluster ID for each clips and entire video. So that is a data which is basically used by online inference to find similar videos and in online manner

8:57 it given a query a clip ID and every all the metadata details it's figured out what is out there in the corpus which this guy has is similar to what like what are the different things which has

9:08 a high similarity. So what it does it looks into databases is it find similar clips and like similar authors and different other metadata information rerank those candidates find uh and use

9:19 some of the tools about like for example the traditional model span score like a classifier score to see what are the different candidates which might be spam which might not be high quality and

9:28 those things out there and once it has those top end candidates is give back it to the reviewer and reviewer is where it actually process this is my main content signals and clip embeddings These are a

9:40 similar video which is given by retriever. Let me think over it and whether I need to retraate and get more data from retriever. So uh that's where reviewer have all the temporal signals

9:51 from a single clip. It has all the information of similar clips understanding and what is similar authors and other thing. It also has a realtime information about how the users

10:01 are actually interacting with this video. What are the different kind of like a reports or like likes, dislike, the comments? what is the u sentiment of those comments out there right so a lot

10:13 of these signals which are actually missed by the offline uh signals and the VLMs and other kind of agents are actually also incorporated to see if there is a change happen in the

10:22 sentiment what is the response I'm getting in the real time so there are different tools available for this to actually understand the video not just from the content and from the semantic

10:32 perspective but understanding from the user interaction perspective those signals are really really important So once we have this judge and everything we build this uh agentic

10:43 framework and each of these agentic frameworks uh all all these three agents are actually powered by specialized uh VLMs and small scale VLMs in a way. So we will be talking about uh pre-training

10:57 first. Now uh in most of the cases you see that okay you have a pre-training you fine-tune your vision transform a little bit here and there and basically tune it for your specific purpose.

11:07 Uh the thing is that these VLMs what we have outside and available the foundational model the front end models they are trained on a very clean very nice data set very web data which is

11:18 very well tuned cleaned and everything but uh data inhouse for a specific purpose is not having the same data characteristics. It's it is messy. It is generated by users. It is for your

11:29 specific workflow. That means you need to pre-train on those image tokens and language to fine-tune your vision transformer from scratch and see whether there is a delta in fine-tuning it and

11:40 tuning it from the scratch. That's where the pre-training is helpful. It's little expensive but um it if if it can get a delta that that actually works really well. The second is instruction

11:51 fine-tuning where we have the two problems. We have a certain policies and certain guidelines. We know the content out there, signal out there and it understand and tune it on that

12:03 particular data set with it with a particular output which is schema like a JSON schema to understand what is the modality uh misalignment or duplication of scores and other chain of thought

12:14 reasoning provided by reviewer agent. So here we have a video clip we have a vision encoder which we are already pre-trained a little bit like now we are doing a little bit more training on

12:23 that. There's a projector which is sitting between the vision transformer and the L language models and it is actually basically a bridge between them and then we have our output depending on

12:32 what is the instruction finetuning data we have on this side. So this is the critical part of actually making your uh model performance like go up for your domain specific. So uh the context is

12:47 basically you have a role, you have a policy, what are the tools available to ground that into some of the metadata or and what are the other things available out there give it entire things into uh

12:57 in in a very brief manner into the context and let it figure out like what the structured label you have for these labels are actually in-house label. These are the one where we have created

13:07 like what exactly modality things are, what are the different uh issues we are seeing, what are the different chain of thought reasoning there should be there in the model and how does the output

13:17 looks like for a human reviewer. So this is like a very high quality data data set that we are fine-tuning it on for different agents. Right? So once we have done with the pre-training just to

13:28 understand the vision aspect of or the other modality aspects of the videos then we go into the finetuning to make it understand and provide the context and the output in the manner that we

13:39 would want to actually process the data set on. So u the next phase is the DPO phase where uh which is basically this technique is used a lot in the uh post- training to actually fine-tune our

13:51 models into a specific uh like realm domain policy understanding or something. But this can also be used a lot in the prediction for actually understanding what are the samples which

14:02 are actually getting uh not so good by the by your multi-ent systems and by your u llms. So what uh can be done is that like uh you have this uh production data set which is coming out you have a

14:17 lot of inference happening you take a subsample of these uh data which is from the production you pass it through LLM as a judge which is trained inhouse on the human label data set and then you

14:29 have the human review Q to see what is the performance coming up on on the actual system. Once you have this thing if your performance is coming amazing and it is above like your whatever is

14:40 the prediction threshold you have 95% for each of the problem it's great but if it is not that means there is a way there has to be a way to loan on these samples where the model did not do well

14:52 that's where you have a human in the queue it understand those all the traces which is there from all the agents the LLM call MCP and it figured out where the problem happened is it like a chain

15:02 of thought reasoning or is it like some of the wrong tool tools are called the retrieval did not work. So that entire validations and basically understanding of each and every hook and node both at

15:13 a model intelligence level plus at a harness level is what you figure out and say that okay these are the improvement that we I need to do on these samples which was incorrectly um sampled by our

15:26 um system. So once you have this thing you have a positive sample negative sample what needs to be updated and that's where you retrain your model to see what uh how can we improve it better

15:38 and this is the continuous improvement where we are looking into these samples reiterating re um improving the models and having a new data set out there from the production u like samples and trying

15:50 to understand whether the drift happen or what exactly the model is doing. So this human in the loop is like always a continuous thing where you have a daily sampling from the production and trying

16:00 to understand like how the model and the L&M as a judge are doing. So uh because of the cost and ability to solve at this scale we cannot go with the standard like a like a VLM models

16:15 from frontier sizes because it's not scalable. It requires a lot of inference. It's a lot of complexity and we are solving a very specific problem like I really don't care if the model

16:25 can solve a coding problem. I only care about my domain specific problem. That is all I care. This is not exposed to the customer. This is an internal thing. So I would actually do a off policy and

16:35 on policy knowledge distillation and do a little bit of quantizations depending on like some experimentation quantization to understand whether a forbit work braining float what actually

16:45 work really well. Then I would have a table of different sizes of distillations and quantization and understand and see where exactly is my performance like up to the mark and

16:55 where I'm gaining a lot of compute and the cost resources um savings from on inference production by doing these optimization. So this is really critical because as a problem what we have solved

17:08 it's all good but in production it has to be scalable. It has to be cost effective. It cannot just be like something which is um out of the um which is just out in the market because

17:21 we are solving a very specific problem out here. Coming back to the evaluation basically first is the task of success. Of course these are the binary things the modality happen it does not happen

17:32 like there is alignment or disalignment. So precision, recall, F1 are obvious metrics to understand what the task is success is. But we want to look at a system in a very holistic manner. Not

17:43 just like the end goal but what is happening at each nodes. How well like the retrieval system is working. What is the latency and recall of the system and how well we are able to reason it? What

17:53 is the chain of thought reasoning coming from these models at each agentic level or wherever the its reasonings are applicable which is basically a reviewer agent in our case. And what is the

18:04 quality of that? Is it like overthinking? Can we reduce a budget somewhere like to make sure that like the models is actually doing a good job with less uh budget or can we increase

18:13 it the budget of the planning or the reasoning things to have it a higher budget and make sure that the complex problem that we are solving is maybe having a much higher performance and

18:24 accuracy. So having an adaptive adaptive reasoning budgeting is also uh quite important. And then we have the robustness. what are the edge cases we are seeing? Uh what are the error rates

18:34 we are having? What are the different nodes and hooks where they are happening? Is it like a tool call is not working really well? It's a retrieval part or LLM is not doing a good job

18:43 where or those kind of thing. And then we have a system efficiency where we are not looking uh just at the performance of the output and everything but we are looking at each and every aspects of the

18:54 token cost which LLM are calling can you optimize LLM little bit more to actually save the cost. What is the efficiency which is we are saying and the latency of each agents and the entire system

19:05 combined together. Then we have a LLM as a judge where we see that like in any of the prediction system there is always a data drift happening especially when you have the user generated content. So how

19:17 our LLM as a judge which is used for evaluation and everything is doing with respect to the human queue. Is there a drift happening? Do we need to retrain our LLM as a judge on new data set which

19:26 is coming from labeling team or how do we actually do on those parts in optimization? We have the spatial temporal reduction optimization which actually look into the frame which are

19:35 similar and just compress them into the one aspect. This reduces the total processing of the videos by a huge extent. The second one is basically you have a caching you have a viral content

19:48 which is basically coming up and you do not want to have the same content out there which is doing going through the entire pipeline and that's where you have a high similarity score and just

19:57 skip the multi- aent system and just make a call on that. The third one which is really important is the metadata pruning. This is where we actually shrink the space from lot of candidates

20:06 based on some of the metadata. For example, some topics and some creator which already have a really good record. That means we really don't need to process these all these images all all

20:17 the videos from the creator which has a really good high authenticity score which are really doing really well on this where the video quality is really high engagement is good those kind of

20:25 things. A metadata is something which can be used to filter out the video which shouldn't even go to the systems. So some of these flags u would be helpful. Um so the last takeaway that u

20:39 u we basically we can take out of this room is that like decomposition t is a key. Uh decompose uh a problem as and when necessary. Adaptive optimization is really important for uh ROI and cost

20:52 feasibility. Good evaluation is paramount. Everything in and out depends on this. This is the foundation of your entire system and entire VLMs and production monitoring and qualitative

21:03 improvement are really essential to make sure this is sustainable in long term. With that, I'll end it and thank you very much for listening. Thank [applause]