Multimodal Collaborative Agents for Next-Gen Commerce — Nidhi Kaushik Vyas, Google DeepMind

0:01 [music] >> Okay, good morning folks. Thank you for showing up. I'm Nidhi. I'm a product person at Google DeepMind. And today I'll be talking about multimodal

0:22 collaborative agents. Typically, these are agents that work with fuzzy intent intent when the intent when it's not clear the user hasn't gotten the right keywords to specify and when they come

0:33 in for a shopping intent, how can you have the agents still guide the user towards their goal with high agency execution, proactive elicitation, and a lot of hand-holding.

0:44 The framework that we'll be discussing today are grounded in shopping or commerce because I wanted to show you a few patterns that are much easier to see in commerce, but they're quite

0:53 applicable in other consumer verticals as well, finance, education, or whatever you guys work on. Um Yeah, so before we even start into the

1:02 frameworks, I wanted to discuss about why this is an existing problem. Currently, a lot of the agents that we have act more like a wrapper to the search bar.

1:12 They assume that the user has a well-defined intent, has the right keywords, already knows what they're looking for. And so when they come in, they just have the right vocabulary to

1:23 interact with the agent. However, there is quite a huge articulation gap. When the users come in, rarely they have their intent well formed. Rather, they kind of have a fuzzy feeling or a vibe

1:35 when they are kind of looking to shop. So, the agent needs to play quite a huge role in hand-holding them. They need to work with the user in first

1:45 understanding their preferences or even eliciting these preferences more proactively. And then helping the user kind of show them different

1:53 possibilities of what could be possible that they can shop and then eventually move towards kind of recommendations that work with these constraints that they have in

2:02 mind. So, what we're going to be discussing is this kind of a flywheel or a loop that goes from very, very fuzzy intent all the way towards kind of achieving the user's goal.

2:15 So, this is how the shopping loop looks like right now. The very first thing that the agent prepares for when the user comes in is what we call the discovery phase.

2:25 This is where the where the agent takes all the different contextual information that they have about the user. This could be past conversations, what the user has specified in their query. It

2:36 could be information present in their personal context, or even the references that the user have provided. And then comes up with a collaborative strategy, and we'll be going to the details of

2:44 this, but it comes up with a collaborative strategy on what more does the agent need to elicitate back from the user in order to help get the intent in a better shape, to get clarity on

2:55 what exactly the user is looking for. And then it moves to the second phase, which is what we call the research phase. Which is again a two-step process. First

3:03 is where it learns what is the best way to elicitate this preference. So, for example, a lot of times the text the text base elicitation may not be the

3:13 best way to get the preferences from the user because sometimes a user don't know what they're looking for. So, then how can the agent start being more creative in terms of elicitating these

3:22 preferences? Can they start using some kind of visual references or visual visual inspiration boards for grounding and speaking a common language with the user? And then the second phase to this

3:31 is also coming up going into the background and doing the heavy lifting for the user, so taking the burden away from the user to describing what they want, and rather going into the

3:40 background and doing all the comparisons, trade-offs, summarization of all the information that they're looking for, and coming back with the right set of potential

3:49 options for the user. And then the third phase, once um the user and the agent is ready to go into some kind of a is ready to go into the last phase,

3:59 which is where um the agent is ready to give out a response. This is where the agent needs to start adapting the response in a way that is most useful for the user. So,

4:09 typically a lot of systems fail here where they just give out a text-heavy response. What the agent should rather be doing is adapting to the query that the user had in mind. So, should it be

4:18 using um some kind of a bulleted list? Should it be using comparison tables? Should it be using um visual boards? So, this is where the there's adaptive response happening as well, where the uh

4:28 agent starts to develop uh more more of a uh smarter uh presentation skills uh for the user to really find the answer that they're looking for. So, we'll be diving into all these topics as I uh

4:40 kind of work through the presentation. So, first and foremost, we're looking at discovery. Like I was mentioning, this is where uh the the the agent needs to remember what exactly matters.

4:52 Um you the agent starts to look at the context across a bunch of different signals. It starts to look at past conversations. It looks at some of the reference images or reference links that

5:01 the user might have provided. It also starts to look at personal context and starts to build out a working state. So, if you look at the sample code that we have here, like there is there is a goal

5:11 uh one of the let's let's work with this query where the user is trying to redo their living room with a certain budget in mind. And some of the um some of the things that the agent develops as part

5:21 of the working state is the session history. It has a user context. It also kind of extracts out the hard constraint that the user might have provided in the query.

5:30 But, things start getting getting interesting when we come to the softer constraints. So, this is where the user may not be able to describe what they're looking

5:37 for and may might have provided like a reference image on, you know, an inspiration that they had in mind or some kind of a layout design that they really liked and is trying to get to the

5:46 agent in terms of uh this is what speaks more to me. So, this is where the agent starts to be more proactive and pulls out some of the salient signals from the from the reference images and starts to

5:57 develop a mental model of what the user might really be looking to get at. So, here the user is uh so the agent has specified uh has identified that maybe the style that is working out well for

6:07 them is of a certain kind. It also uh is also is also starting to work towards a confidence score on like how confident it is in terms of pulling out some of this information present in the

6:19 multimodal inputs provided. And then the last thing uh that happens as part of developing this working state is also figuring out what are the variables that the agent needs to pull out almost in

6:28 real time because these variables uh this variables do affect how the results will be displayed back to the user. This could be variables that need to be refreshed in real time like uh

6:38 inventory because if if what you're providing back to the user is stale information, then it's kind of a moot point. So, these are the uh these are the variables that you want to refresh

6:47 in real time and is and becomes a part of the agent's working state. Um few ways that you can evaluate this state uh the way we have developed our

6:55 auto raters, we we do make sure that all facts are retained meaning that whatever was mentioned in the part of uh whatever was mentioned in part of the context is properly represented in the working

7:08 state. We also capture if the confidence collaboration was within a certain error bound because uh the agent needs to be able to confidently pull out these signals from the input. We also focus on

7:20 getting out um the counterfactual sensitivity uh counterfactual sensitivity. So, we do this by flipping some parts of the queries and making sure that when the query changes, the

7:30 underlying constraints pulled out by the agent the those also change and the ones that are not relevant stay the same. So, we kind of want to measure the sensitivity both ways.

7:40 And then the second part that happens in the discovery phase as well is once the agent knows what they have what they already know about the user what more do they need to know about the user. So

7:49 what we call this what we call this is kind of discovering the intent gap. There's a lot of times that there are a lot of unknown variables before the agent can provide the best answer and

8:01 amongst all these unknown variables the agent doesn't need to go and find all the unknown variables up front. So what I mean by that is in this unknown in in our working example some of the unknown

8:10 variables that the agent said they need to find out more about is maybe knowing what the room width would be for the user or even kind of working on improving the confidence for the style

8:21 before they can recommend back the results. Now once these unknown variables are figured out the agent also needs to work on a collaborative strategy compare all the

8:30 different moves possible and then figure out what is that one unknown variable that it should prioritize such that it has the maximal information gain at that point. So in this case

8:39 I mean one could argue that maybe finding out the room width is the best next move for the agent because if the products that the the agent is recommending doesn't fit into the room

8:49 then again it's a moot point and that is one variable that is going to meaningfully change the the direction of the conversation and that's what the agent works on in this

8:57 step kind of figuring out what's the best next thing to ask to the user and what's why is that the best next thing to ask as well. Once it has okay so I'll go ahead into the auto data section on

9:08 like why how do we evaluate this collaborative strategy we first focus on making sure that the agent is able to identify all the different blockers that are needed to be answered before the

9:19 agent can come back with meaningful responses. We also work towards making sure that the agent is um optimal in trying to get some of these responses so we also don't want to have

9:30 the agent constantly going into the loop and continually continuously asking these questions so over asking is definitely something we flag. We also work on question utility. So, again, how

9:41 optimally is the question being asked? Is the question indeed useful to get the right or elicit the right preference from the user? And many more. Okay.

9:52 Moving on. The The second part that I was mentioning is the multimodal elicitation. This is where the research phase happens. So, play along, but like let's say the

10:03 the agent has asked about the room width, the user has provided the response, and then the agent goes to the next step, which is where the agent is asking about the next constraint that it

10:14 needs to know about is what was the style preference of the user. Now, the first first thing that the agent needs to do is kind of form this temporary bridge between the constraint

10:24 itself and how that maps back to the the product catalog and the ontology in your knowledge knowledge database. And this is going to be important because when you start retrieving these

10:33 products, you want to have a way to map these constraints back to your knowledge base. So, this almost happens in real time where we map the known constraint or the constraint that the agent is

10:45 exploring back to the knowledge database. We also work towards um figuring out We also work on like the agent knowing what is the best way to get the response for this constraint as

10:57 well. So, in this case, the agent has decided that maybe since this is this kind of a subjective constraint, the textual the actual elicitation is not the best way to do this. So, one of the

11:08 ways that the agent thinks this could this could be elicited back from the user is using some kind of a visual preference board.

11:17 So, the agent then goes back to determining what is the best form of options to show to the user. This could be a combination of figuring out from the existing constraints, the past

11:28 conversation, and then the temporary mapping that you've created from the constraints back to your product ontology. So, in this case, the the agent thinks that maybe coming up with a

11:37 few styles that are most similar to what the user had provided as reference image could be a good way to start thinking or guiding the user towards a common language on what could be uh something

11:47 that the user is interested in. And then the the then the agent also goes into kind of um observing the space of what kind of reactions the user is giving. So, it could start looking at

12:00 these micro signals of if there was a hover or a click in a certain direction and starts updating its its confidence model on what kind of signals uh what kind of signals can be used to im-

12:10 improve the confidence in like what could be the style preference for the user. Um some of the auto raters we use here, we do look at how efficient the how

12:20 efficient the agent is in discovering hidden preferences. So, typically, we would use a user simulator, feed it with some constraints, and then we'll see how how efficient the agent was in kind of

12:30 eliciting some of these constraints. We also focus on turn efficiency. So, how efficient was the user in how many turns did it take for the user to be able to to be able to uncover all these

12:40 hidden preferences. Ideally, we don't want the user to go into this loop and keep asking the same questions again and again and again. Or also, we don't want to go into the loop of asking some

12:49 um some questions which may not give you the best uh which may not give you the best information required to proceed the conversation. And then we also look at forma- format selection accuracy. So, we

12:59 typically also look for if the agent is asking the right right question in the right format. So, for example, if the if the question was something that was easily statable, the right format could

13:09 be a textual elicitation. But if if this was more of a fuzzy question where the user is clearly having an articulation gap and is not able to describe the preference, maybe

13:19 the best way to do this is speak a common language and come up with some visual anchor points for the user to say what speaks more to them. Um

13:31 Yeah, so then the last step in this process, once the So, to recap like basically the Now the agent knows exactly what they what the user is looking for. The agent has identified a

13:40 collaboration strategy, has figured out all the preferences for the user, and knows what is going to be most optimal in terms of um the different priorities for the user. The last step in this

13:50 process is to also use the model intelligence to figure out what is the best way to provide this response back to the user. So, in this working example, we found out that the agent

14:00 knows what the style preference is, has figured out that the user is looking to buy products under a certain budget, and knows what are the different dimensions along which it needs to find uh these

14:10 products for because through the conversation they figured out some of the different um different metadata information that is going to be relevant to surface when

14:18 they're giving out this product information back to the user. So, the one one important step that happens at this this stage is figuring out what's the best way to surface back

14:27 this response. So, for example, like if the user was looking for a particular policy or review information about a particular product, maybe the best way to do this is to go give out a summary

14:38 or a bulleted list. But if the if the user was looking more towards comparing two different products, maybe the best response is to give out a trade-off table or a comparison table across the

14:47 different axis that the user cares about. Um and in case that in our example where the user was looking more towards kind of style inspiration or ideas on how

14:56 they can kind of redo their room, then maybe the best way is to give out some visual references and product inspiration photos on like how What are the different options and possibilities

15:05 uh for the user to uncover. So, some other ways that we focus on evaluating uh this stage is focusing on the format accuracy. So, we want to make sure that the response format is kind of

15:18 optimal for the user query so that they they find out exactly what they're looking for. Uh ideally the information that they're looking for should not be buried in the

15:25 response, but should be easy for the user to spot so they can come into the next stage in the intent journey, which is to basically buy the product. We also look at data fidelity, which is to make

15:35 sure that the model is not hallucinating and it's really capturing the information in the correct format in the you know, the the information is just accurate and is captured across

15:44 the response. We also look at user actionability. So, this is ensuring that the response format is such that the user is very confident and and commits to the next action, which is like I

15:56 said, the action to purchase the product. So, just to recap so far, what we have is um you want to design the product. Um

16:07 you want to you want to design the product such that you're prepared to accept vibes, like I say. So, that is to say that users will come with fuzzy intent. Users will not have a

16:15 well-defined goal. So, you want to make sure that your system is able to work through queries that are not clean. Um the second takeaway is you want to focus on showing and asking rather than

16:25 always asking with textual textual with textual elicitations. Again, like visuals and comparisons do reveal preferences much much faster. It allows you the agent and the user to speak a

16:35 common language. Third one I would say is shape the answer. So, do focus on making sure that the presentation format is ideal for the user being able to find the right

16:46 information. Um the way you have the model response structure is also very much part of the intelligence. And then the last one is make sure that you grade the loop. You

16:56 have the right auto rater set up on every step of the process. Um and honestly, developing these auto raters is a is is almost like an evolving system. It

17:06 starts very simple, but as and when you start the the system starts evolving, you want the auto raters to kind of gradually grow with your system and start

17:13 um yeah, it which just gradually grow with your system. So, that's all I had. Um I can take a few questions, but

17:21 hopefully the learnings we shared are useful for whatever you folks are building. Yeah. Yeah.

18:10 Yeah. Yeah, great question. So, the question is about what should the how should the ontology be structured on the merchant side so that it's fair both for the

18:18 agent and the merchants. So, yes, we do take a lot of advantage on the domain expertise of the merchant on as to what they're trying to sell and like we do work towards creating So, you remember

18:29 how I was mentioning about the bridge between the constraints that the user might have specified and then what the agent kind of understands. That is where we do

18:37 expect a lot of intelligence to flow from the merchant side where the ontology on how that constraint could map to the different metadata that the agent has. Sorry, the that the merchant

18:45 has maps in. So, yes, we do partner a lot and then there's also like the UCP stuff that we launched recently which allows all the merchants to kind of start speaking the common language with

18:54 the agent as well. >> Yeah, so ideally, I mean, honestly right now we focus on making sure all of this flows back to the agent and the agent makes the decisions because you want to

19:27 build like a horizontal common layer across all the different merchants. So, and also like it should be a seamless experience for the for the user who's interacting with our apps. So, right now

19:37 the response format is very much part of the agent's intelligence. It is not something that the merchant gets to decide. Yeah.

19:46 Yes. >> I'm just curious your opinion what happens when your user is not a user anymore, it's an agent. >> That's a great question.

19:54 We're in the early stages of building this out, but I think I mean there could be a case where um Yeah. Yeah, yeah, exactly. So, yeah, I would I

20:03 would expect like an MCP to be the interface between the two for sure. Honestly, we haven't gotten to a point where we have agents interacting with our agents just yet. Also, like what

20:12 we've realized at least from our user studies is users really like to be more involved in the process of choosing or even exploring the different possibilities.

20:21 So, during the upper funnel journeys where users is looking more towards discovery, inspiration, that is where they would rather be interacting with the system than with their agent. I

20:29 think where the agent typically comes in or even where what we've heard is like the was the lower end of the journey where they're just looking to compare or negotiate or

20:37 compare prices across different merchants, but very much up there in the funnel, it's the users who kind of interact more with our systems. So, yeah.

20:46 Yeah, I think yeah, I can take questions outside, but thank you folks for coming. >> [music]