Agents Acting Safely on Our Behalf

0:00 I've been thinking a lot about do we need to change the web for agents or do we need to just make agents better so they can use the web for us? >> No, we should be changing the web for

0:11 agents. But you could imagine if it worked out of the box for React and Vue and all you had to do is do a package JSON flag change that that might make it a little easier. I do think that site

0:24 builders should eventually explore WebMTP. It's like it works, but it's not what you want to have working. Can you tell me the story of WebMCP? Like, how did that even become a thing?

0:38 >> It was a little wild, not going to lie. [music] >> What is the talk about at Agent Con North America? October 22nd and 23rd. >> Yeah, I'm really excited to talk about

0:59 how the web is changing due to AI. It's changing on a number of vectors. One is first of all just that agents are present and so we've got this fourth actor on the web. How do we secure it?

1:10 How do we expose good APIs so that it can navigate appropriately and well so that people can guide their businesses? And then also just making sure that we're putting the user first in the

1:21 whole um transaction. Yeah, so many different rabbit holes we can go down on each one of these. I mean, the secure part is probably what I think most folks think about when they

1:34 think about agents acting on our behalf, but there's also like just the web part and agents using the web. And so I've been thinking a lot about do we need to change the web for agents or do we need

1:49 to just make agents better so they can use the web for us? >> No, we should be changing the web for agents. And by saying that I don't mean that we need to rewrite everything. Um,

2:00 WebMCP offers really wonderful tooling and it's a pretty elegant simple API. Um, what we're working on now are ways and what I'm working on now are ways that we can actually use what people

2:13 have already written to expose WebMCP tooling. What WebMCP tooling allows you to do is to specify what exact actions you want the agent to be doing. And you can also add annotations. So as you're

2:27 adding the annotations, you can say that it's readon, that it's userenerated content. that can allow the agent to know, oh, okay, I'm not going to take action on anything in here. Like, if you

2:37 have a bunch of comments on your site that are userenerated, you can't necessarily trust that people aren't injecting prompt injection attacks in there. So, the agent actually has a

2:46 signal, hey, don't, you know, go forward with anything that's put embedded in there into the web ephemera. Um, so that can help um the web stay secure and safe. There's also a consequential hint

2:59 which allows you to say, "Hey, agent, you're about to do something that's potentially consequential for the user. Please go back to the user and confirm." Otherwise, agents are just scraping the

3:10 DOM, scraping the accessibility tree, taking screenshots. It's token inefficient. It's slower. The site developer is just kind of like, you know, at the whim of however

3:21 non-deterministically these LLMs decide to take action on the site. And so this gives us a really deterministic surface area in order to guide the agents exactly what we want them to be

3:33 doing and in a safer way. >> You're speaking my language. I love that. [laughter] There's so many pieces to that that I'm a huge fan of. How do you look through

3:45 and think about like computer use or browser use agents versus the API or MCP type agents? Yeah, I mean there's so many different ways that a browser can be used these

3:59 days, right? You have inapp browsers where people are, you know, if you're on a certain like super app, they might keep you within that instead of allowing you to go to whatever your browser of

4:10 choice is. You might have computer use uh apps or computer use um ways of working like 247 headless models where you kind of set it and forget it and let the agent do whatever. You also have

4:23 cobbrows where you are a human in the loop and you're actually in the experience. We're seeing more and more that people might want to switch between these modalities. Like maybe there's one

4:33 where you just want to do it yourself. Maybe there's one where you're you would like actually a little bit of cobrows because you want like hey Google maps like show me all of the restaurants in a

4:43 particular area that I can afford d and you want to actually be part of that experience instead of um letting an agent run wild. And then finally, there are tasks where maybe you're part of the

4:54 task. Let's say you're like getting ready for a holiday. This is the only time every year that you get to plan such a holiday. You want to be intrinsically part of that. But then

5:04 there's one part that's like go through all the trip advisor things and tell me like which ones are most valid to this kind of query. Like you don't have to sit there for for something like that,

5:14 but you might want to be involved in something like the airline per purchase or something like that because you want to pick your seats and stuff. So what we're seeing is like people more and

5:22 more it's not just one modality and this is also the dream of the web. The dream of the web was not just one experience ever or one person or one user or one type of commerce site content site

5:36 whatever it's supposed to be for everybody at every level. Some people have high appetite for agents. Some people have low appetite for agents. We have to keep the web safe and secure for

5:46 all of those different pathways. It's funny as you mentioned that I'm literally have my other computer over here off screen and it is downloading a bunch of uh logos of companies because

6:00 one of the things that I tasked my agent with doing is grabbing all of the photos from our recent uh MCPcon and Agent Con in China and Japan and then taking the speaker photos and uploading them to

6:14 Canva so I can go and I can make the thumbnails. But then I realized like, oh, I need the speaker photo, but I also need their the logo of the companies that they work at.

6:23 So, it got the speaker photos and it downloaded them from Flickr and then it uploaded them from it filtered out whoever wherever there was photos that weren't speakers and then it uploaded

6:34 the right ones to Canva and then it put their names and the title of their talk so I can easily search it. Then, and then I'm like, "Oh, but I need the logos." So now it's literally just

6:45 popping up with websites and grabbing logos and then uploading them to Canva too. And that's something where I look at it as exactly what you're saying where I am letting it do all of that

6:57 monotonous heavy lifting of downloading, uploading, searching for logos, but what I want to do is that creative part of taking the photo, adding the logo, and then making it is it should it be a

7:11 little bigger here? I think ah and that's where I actually find the joy in it. >> Yeah. And that I think that that's like that is the most powerful part is that

7:20 it should be tailored to what the user wants. It should not be oh there's agents so you're going to use an agent. It should not be oh I have to be in the loop. It should be exactly the

7:31 experience you feel comfortable with that you feel secure with and we should be also tightening the security and privacy of each one. So >> what we have seen is that it's not going

7:44 to be one thing that secures the web. We have to use multi-layer defensive strategies because agents can be non-deterministic and they can do all sorts of things. And so you have things

7:55 like special agent containment where you might say like, "Okay, me Sarah can look at all my passwords and how's my credit card info and all of this stuff, but agent Sarah, well, she can only go to

8:08 these five sites and she only has access to this information about me and she can only buy spend $20 this month or maybe none or whatever." And that way you can like kind of define what you what

8:21 experience you want your agent to have. Another one is um origin sets. So if you decide to go on like a trip or something and you're like, "Okay, plan this trip for me." It should only be going to

8:35 Expedia and United and those kind of sites. It should not be going to your bank [laughter] and to your like health site. Um another one is spotlighting. So like let's say you have a large context

8:45 window full of a lot of things. The things that you decide as a user are most important should be like in we use the term spotlighting like more important to in that context window or

8:57 have a greater weight than all of the other pieces of information. So it I'm just talking about a few. There's so many ways that we can protect the web, but we have to use them all to make sure

9:09 that we're creating good and safe journeys. I like how you got those agents on an allowance signal. [laughter] >> You can only get $20 if you make sure to

9:21 do all of your work. >> That's right. That's right. That's right. >> And if not, you're not getting that allowance. I'll tell you what. So the

9:29 other thing that I was going to ask you about which I always find fascinating is like when you're uh after you've reached a certain level of engineer and this is more like just

9:40 how you think less about your talk just um prefacing it. I think there's we always joke like uh friends and I tend to joke about how you

9:53 work with some engineers and they have like a really good nose or they can smell when there's a business problem that's worth solving. And so I wonder how you think through that like ah if

10:05 you're going to spend your time on something how do you make sure it's that really impactful thing versus something that may not have carry as much weight. >> Yeah. I mean as a distinguished engineer

10:18 that is one of the like parts of the job is being a strategic lead like deciding you know I'm what we call an area tech lead meaning that I go over a large surface area of Google and so I look

10:32 over the AI and the web ecosystem for Chrome um which includes a lot of different things is one of them ondevice AI all sorts of things um so when I think about things I do have to

10:46 constantly because I get deep as well. So if you get deep, >> what happens is you naturally think about the problems in a very tactical tactical way too but I always have to

10:58 bump myself up a level to also see things from a higher purview. It helps that I can be across many many different projects. Um, and I actually pull different groups together so that

11:09 they're aware of each other's work and kind of highlight and have the they do them do information sharing across groups too because people tend to silo real deep and then you you forget to

11:21 bump up at that level. >> But I also encou I make myself write papers with that thousand foot view that could look like a premortem. Premortems can look like, okay, the web failed in

11:36 five years. What what happened? >> And that forces me to look back and say, okay, well, this couldn't happen. This can happen. This can happen. And then revisit like, okay, how do we make that

11:48 not occur and trace backwards some steps? So that's one like way of thinking that helps. Another way of thinking is to have deep empathy for the users and the site builders and think

12:03 through from their perspective. What do I need? If I am an, you know, if if I'm smart and not technical, if I'm a doctor or a lawyer and I have to I have to I'm under HIPPA or I have attorney client

12:15 privilege, what do I need in order to feel like I can even use an agent at all? like what kind of privacy and security would I need to be assured that my things aren't leaking and getting um

12:29 used improperly or that I'm being s suspect to a prompt injection attack or something like that and from there developing strategies of like okay well I would probably need this I would need

12:41 this and you can extract from that what can make it easier to do those things like I mentioned um finding these pathways to make WebMCP work more like out of the box with existing systems.

12:56 That doesn't need to happen, but I think it should happen because that also increases like the ability for developers to opt in without a lot of work for businesses to be able to use it

13:09 without having to like rewrite in a whole new mental model because they can reuse existing infrastructure they've already built. So, anything that helps me kind of think through things and pose

13:21 questions to myself at a higher level and walk backwards into the more tactical work because you can start to extract the tactical work from those larger questions. That's something that

13:33 I force myself to do. >> That's so cool. And yeah, the like thought experiments and thinking about how to come up with thought experiments I imagine has got to be uh a lot of the

13:46 way that you spend your time just because there's >> I can only think about how many different folks that are pulling you and trying to get you like, hey, this is

13:57 important right now. We need to we need to think about it. And then you're having to balance between all of those different like demands. >> That's right. That's right. And it does

14:09 require a lot of reading, a lot of so you you also requires I mean honestly time management because I'm in so many like different things that I have to always set aside a big amount of focus

14:24 block and Google uh our love language and communication is docs and slides. Um, so I have to have time every week to process all of that material to understand what people are trying to

14:39 build and make go through CL's like play with things myself, make sure that I understand in order to make some of those calls and decisions. Like we were, you know, going through something the

14:51 other day where I noticed that one of the decks and docs was in congruent with how we were actually doing the implementation. And you have to dig in deep enough to

15:04 trust but validate the things that people are mentioning to you across teams. It doesn't mean that you don't it doesn't mean that you don't trust the people. It just means that like if we're

15:14 strategically saying this, are we really standing behind that in technical implementation and like making sure that >> it's the same? >> Can you tell me the story of WebMCP?

15:26 Like how did that even become a thing? >> Yeah. So just to be clear, I moved into this role in April. So I haven't been here from like the beginning of the entire spec. Um okay,

15:39 >> the um the spec started as I understand it at uh Microsoft and Edge. Um I do actually think Alex Na Nahas who does um WebMCPB originally started thinking about the idea and then Edge started

15:55 working on like the spec proposal and then Google got involved um and now Google does a lot of the work um but it's always been like a cross team cross system collaboration which is a good

16:10 thing not a bad thing >> [snorts] >> um so I joined in April before that I was running core infrastru structure for Google. So I ran web at Google, Android,

16:19 iOS, multiplat experimentation, observability. I was also al also the reliability for all of core, which means that if anything went down across Google, I was on point to for our

16:32 internal stacks. It was a little wild, [laughter] not going to lie. >> I worked with Chrome before I had this job, but they were one of many, many people I worked with, right? So I I

16:43 worked with Workspace and YouTube and search and um those were all my customers and then I moved over in April to this role. So I'm no longer doing the large scale um technical organizational

16:57 leadership of core infrastructure at Google. Now I'm a distinguished engineer. So I switched um tracks but not levels. So now I'm on the ice what we call the IC track.

17:08 >> Yeah. Nice. So, a little bit of a random one on WebMCP that I've been really enjoying is can I build a WebMCP proxy or gateway to many different websites that would if I

17:26 know how the your website works because I'm just worried that half the internet is never going to build anything WebMCP related because they either can't or they don't want to.

17:37 >> No, you don't. you you are the owner of your site's webmc webmc implementation. You're not making it for other sites. That's not a a thing that you can do. Just like when you build your site,

17:50 you're building your site. You're not building other people's sites. It's the same kind of concept. It doesn't go outside of the like site development things. Um, but you could imagine

18:01 if it worked out of the box for React and Vue and all you had to do is do a package JSON flag change that that might make it a little easier to adopt. We're also exploring things to make it work

18:16 out of the box without like the site builder having to do like you could just use this other thing that would like analyze your DOM. The problem with the analyzing like that that's a good it's a

18:28 good stop gap if you don't have time to do a web MP implementation. Yeah. >> But the issue with it is it tracks based on DOM and DOM isn't the state of your application. So um I think that approach

18:42 can be helpful in a transitionary period but I I do think that site builders should eventually explore WebMCP. Yeah. >> Yeah. Yeah. So, it's it's like it works, but it's not what you want to have

18:58 working. >> Yeah. And it's also like I think like some people get confused because they think that if you have WebMCP, you're trampling on MCP or you're trampling on

19:10 REST or GraphQL. This is not the case. Anything that is exposed within your UX or UI that is a state that you'd already be exposing with React or what have you can just be the execute piece of WebMCP

19:26 and that includes allowing you to use MCP and GraphQL and REST and things like that. So I I think that they come together if you actually work with it. They actually play really nicely

19:40 together. It's not like a either or. >> Yeah. At Agent Con, I heard someone talking about how at Agent Con EU, I mean, um, someone was mentioning how they built a web MCP and the only thing

19:54 it did was it linked to the MCP server, but it was it's like ah, it's not ideal. Again, it's but it's something that will get you what you need to get to uh, in a little bit of a hacky way. Yeah, I mean

20:11 you don't have to expose it that way either. I mean you can what I would think about it more like okay if you let's just do a cart checkout flow for a second

20:20 >> on the client you probably normally expose to the user your cart total like things like um whether or not you're you know navigating to the page the like full thing and then you do the cart

20:35 checkout and then you have on the client like success error submission things like that and especially if you're in a co- browse environ environment, you want the page

20:45 >> to actually be activated to show all of these things. >> Yeah. >> Um, and so you would use both and you could actually just do webmc, webmc,

20:54 mcp, webmc, webc. [laughter] So like there's, you know, you don't, you can also do just chained webmc and then one of them calls MCP. You can do it either way. Um, but I I do think that

21:07 there's also things like off and things like that that you want >> to be exposed via the client to the per to the user. So, a lot of it has to do with like how your implementation is

21:19 already working in your UX. The other thing I'll say is I've had a lot of fun thinking outside of, you know, the first step is exposing what you already have on the site, like what whatever you're

21:31 already doing on the site. But the next step is I've been making tools that allow you to do batch processing. So I have a to-do list. I can just say, and this is good for accessibility, too. I

21:44 can now say to Gemini or whatever, like, "Hey, close out my day." And it will automatically take all the to-do items and carry them over for tomorrow and also start my journal entry. or it will

21:58 allow me to in a shopping cart thing that I've made and open sourced, it allows me to dump usuals into my cart. Like anything that I've ordered more than 10 times, it automatically puts

22:10 them all into my cart. Like those are things that first of all manually you'd have to do differently. Um, there are also things that you're kind of like exposing on the client to the user and

22:21 you probably want them to be able to see that that's all working appropriately and what's in the card and things like that. And so there's opportunities to go beyond

22:32 just the like exposure of the simple actions into something that is potentially really good for people with accessibility issues, but also just people who are busy. I don't I don't us

22:44 I don't want to go through and like and it's doing that mental math for me of like wait I always get eggs. >> Um so that's good. >> It's an opportunity to delight the

22:54 customer in ways that they weren't expecting. >> That's right. That's right. >> Brilliant. Sarah, this has been awesome. Thank you for doing it. Y