MCP Goes Stateless | ​John Dellenbaugh & Pankaj Kumar | MCP Release Party - Seattle

0:00 So, um, a little bit of, uh, backstory here. It was probably six months ago. I sat in a system design interview and it was a pretty standard question. I was like, "Oh, yes, I can easily horizontal

0:15 scale behind a firewall or behind a load balancer." And it was, you know, the question was specific to scaling MCPs. So I drew it out, you know, very very standard load balancer, multiple

0:27 systems, easily scalable and I got the question wrong and I got the question wrong because I I at the time MCP I just didn't know that MCP had this special thing you had to do in order to in order

0:40 to scale it horizontally and um for this particular role they needed someone who knew about a sticky store and and um keeping track. So it was a hard lesson to learn. So when this came out, I was

0:53 actually very motivated. I was like, "Oh, I know the answer to that problem. I'm actually going to build a demo and and and do better at that system design interview, but do it in real life." So

1:03 um that's what we're going to show here today. We're going to talk about stateless and we're going to give you a nice little demo on it. So thank you. >> You go.

1:10 >> Yeah. >> Thanks, John. So yeah, I think uh thanks John for the nice uh introduction. Yeah. So I think uh we all know like one of the most important changes like MCP

1:24 has gone stateless and as shown in the den slide it was there was one line actually that actually covered a lot that uh it has gone stateless uh there is no handshake is happening in the

1:36 protocol level ch uh the protocol levels sessions are gone. So yeah uh now in this demo uh we kind of tried to uh make sure that it is uh it is motivated by some real use case. So what we're trying

1:53 to do is uh we imagine a scenario where suppose if you have a AI shopping uh assistant and uh it is trying to actually uh it is trying to actually talk uh to

2:06 uh talk to a store suppose it's uh it's actually creating a cart and it adding and it's adding items and then also it is trying to uh get the information about a cart like what I have added and

2:18 these kind of informations And at the back end it is the agent is actually talking to MCP server and this MCP server is actually providing you like uh some basic tool calls like create card

2:30 uh add item and uh getting the card. So and when we were trying to do this demo uh this was we were actually searching and Shopify uh which is one of the like one of the companies which provides you

2:45 stores to sell items. They als they also have uh their own MC server which is known as shop uh what do you call it storefront MCP server and it provides actually the same functionalities. it as

2:58 it can actually ask you to get the metadata for your card especially related to a store and this uh example was related to that. So now um usually in our scenario what happens like um

3:12 suppose a customer is trying to uh talk to the agent to add cards and this and do all the operations and suddenly it may happen that while talking to the agent it may okay what which card are

3:24 you talking about? So what's the catch here like why did that error happen? So yeah let me actually go through the this demo first. So what is happening here? So in this

3:39 demo this is the agent that is talking to the MC MCP servers in the back end. We have like uh two MCP that is two instances of the MCP servers. one is instance A is instance B and we have a

3:52 postcare database that contains your cart uh data. Now uh if I go back to the example when an agent tries to talk to tries to certain actions each uh each of that conversation is actually multiple

4:07 conversation is happening between the agent and the uh the server instances and it is that's where actually the session comes into picture. Uh okay. So if I move next now what happens like

4:20 okay uh uh for suppose your store actually initially it had like very limited sessions so you had like less customer but now it kind of blew up you want to scale out things so you started

4:33 adding instances MCP servers in the back end now what happens actually so suddenly you'll start getting errors under the old protocols okay session not found this card is not found so what

4:44 happens actually in that So okay after that yeah so this is a running demo um let me just run the demo actually so that because it takes time a bit 10 seconds

4:56 once it shows the result I can tell like what is happening in the back end second okay great okay so what happened actually when I clicked scale it so when

5:11 I clicked scale it what happened like it the MCP agent first did a create card call and it was served by legacy a instance server. Okay. And under the old protocol when you it talk to that

5:22 instant that instance created the session ID it also created the uh session related to that agent in its own local memory. Now uh you can see that uh we again because to simulate the scale

5:37 out feature like to handle more customer we created another instance. Uh now you see that when I do another item call okay uh it uh the request landed to legacy A and we added three bananas uh

5:51 but the next item called that add item it happened now it landed on legacy B which does not have any information about uh your session because it was initially served by legacy A so that's

6:01 why you got session not phones and that's the root cause of the errors are like okay where is my cart and those kind of similarly when did when we did get cart this time it landed on legacy

6:12 So and we got the information around related to our card. So yeah it works on one instance when everything is there but once you scale out these kind of issues happens. Now what do you do? So

6:25 there are two problems here uh that needs to be done. One is the smart routing like uh we have to remember which instant created actually the session and then we have to route it uh

6:38 as per uh the previous session ID. Second thing is uh what if my the session which had the information restarts it can happen actually anything can happen in the machines it can

6:50 restart it can lose the memory. So when it restarts it kind of lost all the conversation all the session data that you actually had earlier. Uh so how do you actually solve these things? So to

7:03 do this uh second yeah the first problem that the routing actually was solved by sticky routing. We implemented a sticky gateway that

7:17 kind of you can say maintains a map of session ID with the instance ID and then after that we also had something like a session store which actually contains the session data.

7:29 So you can see that to uh get all this uh working we actually had to uh add some other infrastructure which kind of does not actually add much to our product. it is more like it is kind of

7:42 pushed on on us to maintain all the uh things working. Uh let's uh do the demo now. This time actually when I do click on add add the text it does the same thing. It actually creates a sticky uh

7:57 sticky routing inside the load balancer and also also has a session store. So now you can see that okay uh respect to our previous result. So create actually happened on legacy A and

8:11 all the other items we because of sticky gateway we now know like okay for this uh conversation it should always get redirected legacy A. So all the uh calls tool calls got redirected legacy and we

8:25 got a proper working system. Uh I also mentioned about recycling. So what happens actually when uh MCB server actually goes down it can happen. So for this this is so when I click on this

8:40 button what it does is it kills you it kills your actually the legacy server where your server where your session was stored. So now you can see that uh after we try to do two calls like add item and

8:53 get cut but uh we actually because the session is not there itself that instant is not there so it's not going to serve actually your uh tool calls. Yeah. uh and that's where

9:06 actually the session store comes. Suppose it comes back up then we can actually retrieve the data from session store and then uh we can uh do the proper processing. I think uh that's all

9:19 from me. I think I will then move on to John for showing the stateless part. So, I'll go ahead and take us through the rest of the demo. Thanks, M. Great. So, we showed the model working

9:53 in a, you know, we're basically paying the tax. We're paying the tax with infrastructure. That's another system to manage the state and another another another um data store to store the

10:03 state. So that's the sticky gateway and and the session store. Um so you know how do we defeat that? Well, we go stateless. And um you can see in the demo where we're using legacy A and

10:20 legacy B. Um the infrastructure here, this is just running in Google Cloud. um and um both the gateway and and the legacy system. So when we go to go stateless, you'll see we have a standard

10:34 load balancer and we have the ability to scale without issue. Takes about 10 seconds here. So here we've switched to the modern systems. the modern systems are running the um

10:49 the latest MCP release um or the the pre the pre-release software and uh it's behind a regular um roundroin load balancer and you can see that the cart was created on a modern B and we

11:03 introduced one item that's the sourdough bread and we added another item avocados and that's on modern A and from a backend perspective if I take you to I use Koopa base because it was free. So,

11:19 um could get in relatively cheap and that was also the deciding factor there on on why I went with GCP. But as Den mentioned, you could, you know, these are just using cloud runners in

11:33 Google. You could do the same with Lambda. You could do the same with Azure Functions. Very, it all it all operates the same. So you can see that um this is the cart ID that that

11:50 essentially replaces what the the the the the state had to manage in additional infrastructure. So, and then in case you're curious, um what happens when we when we when we

12:03 prove it at scale, uh this button here is going to initiate 60 concurrent um parts with one item. So, we'll see what it does here. That that is what I showed here. It's

12:22 all tracked. It it it's this ID here. So you create the cart, it creates the ID and then every every additional M MCP system is going to um make the request with that ID.

12:40 >> When the item is created, you're you're providing you're you're providing the handback is the is it creates the item the ID and and that goes with it. So,

12:55 so we'll go back here. Here we scaled 60 events and we had 46 of them go to the first system. If I run it again, you know, it might just because of the

13:07 additional traffic, it may go, you know, 12 or so. So, now we've got nine systems. 19 went to one system. And if I go back to my to my backend, what you'll see is you'll

13:25 just see individual carts. So you can see it updated. We got individual carts with with one item. So So that's stateless. Um the other piece

13:38 of information here I wanted to give you guys, a lot of folks ask about security, right? They they think, "Oh, well does stateless get me security as well?" The answer is no. You have to handle

13:49 security. Um, you know, if anything, it may make the security aspect of things easier. Um, I recommend if you're curious about this demo and you want to learn more, I built it for for learning.

14:02 Um, you can go into my GitHub repo, John Dell914, and you can download it within there. You will you this this demo is there as well. all of this information, instructions on how to um um install the

14:18 demo locally or build it again in in in GCP. Um it's about two bucks a day. Um and you know, and you can play around with it. Um so highly re my next phase on this is actually to go and and do

14:34 something to make it secure. So what's that? Uh John Dell-914. I'll put it up here. And I'm by all means not trying to promote my GitHub,

14:46 but it's there for you to get um because I want people to learn and um this has been a good opportunity for for myself even to learn. So um anyways, wanted to make sure I did touch base on

15:02 the security aspect of things and um feel free to go get the demo, read through the security. the next iteration of of this will be kind of pouring in the security aspect specifically around

15:15 you know ways to protect against um session stealing prompt engineering and things like that. So anyways we've run long and that's it. So any questions? Yes.

15:44 >> Mhm. >> Yeah. Set the number of apples to five and then remove one apple. timestamp that arrives to individual

15:59 servers that are sickless and then >> you know how that actually arrives if there's any kind of delay that mismatches with the original request then the entire thing got messed up and

16:09 we'll get different results >> right so by switching this to a stateless um how does how do you ensure that the final state is correct >> yeah so the the question is merely

16:20 around ACID principles in terms of correcting or or um you know, it's the resiliency and the durability >> mostly and and I would say that as an engineer, you still need to go and

16:33 follow best practices of durability and resiliency. I don't know that there's anything inherently built into MCP that handles that for you. So, if anything, it makes it on par with how you would do

16:46 that with other applications that are follow that classic modern, you know, behind a load balancer horizontal scaling thing. So if anything, you can take some of those common engineering

16:57 best practices that you use with other applications and apply those to your MCP systems and you know um and or in this case it's a it's a it's a shopping cart where you're adding items, you're

17:08 removing items. So there's there's standard patterns there. The MCP um spec by nature won't solve all of those things for you. So question in the back

17:34 So in this particular demo, the the token was was created per cart and for every item that was added in the cart, it was the same it was the same token. But how you design your application may

17:48 be different. But to keep it stateless, I think that the the goal here is that it creates that ID, it stores it in the database and that is effectively what goes with the transport versus that

18:00 information is is stored and required and lives on the on the actually on the actual MCP system itself. So we've removed that dependency. It's no longer on the MCP system. Now it's in it's in

18:13 the database with the cart in this case. So I would say to answer your you know I don't have a specific answer for your question other than every application is different.

18:27 Yes. Last one. >> Yep. Mhm. >> You need to define the utility and then dig in from there.

18:49 >> Um I'm working on a couple of different things. One is a a presic voice inference um CAD modeling agent essentially um API into Grock and then Grock will kick back an answer. And

19:03 then I was looking at like containerizing or using docker to kind of filter gro's feedback to utilize prootic voice inference and then to have voice inference uh do seale you know

19:15 highle command function in specific applications and I guess that's a a patent that I'm writing with with my attorney that we've gotten about a month until it goes to full patent. So I was

19:28 actually looking the I API in whisper flask specific set of PI audio wheels. It works. Got gro to spawn a Imperial Star Destroyer over Colossus data center in Tennessee with my voice. It was

19:42 pretty fun. Um but to actually get it to drive Unity or Blender PTC on shape or Solid Works is is a whole another ball wax. And I was looking for a way to containerize like high level command

19:53 level function in those programs to have the LLM kind of like take inference and then push a solution right and then I guess the other thing I saw Stripe Shopify and all that I've got

20:08 another brand and uh we're looking at doing carbon fiber espresso machines and I've got a local carbon fiber manufacturer that does all the carbon fiber for the SSC Touitar which is like

20:20 $2 million hypercar and um we are prototyping carbon fiber espresso machines. So we would definitely like to build an agent that can kind of answer people's questions about like what is a

20:30 P boiler and why should I go with the $1,500 espresso machine versus the $7,500 espresso machine yada yada yada. Uh but we're looking you know if I could question more around in your specific

20:43 case how MCP can benefit you because it would be the mechanism to um call tools and those tools are specific to what you're doing >> 100% for the for the preotic voice

20:57 inference thing and then I guess the their thing would be like had it have it function as like a a chat agent for people shopping would be the >> y two tiers I I would say that that

21:08 that's in my opinion that's been the industry standard of why people are why MCP is growing so fast or at least one of the reasons and um I I would say that it's a

21:22 it's a safe place you know my opinion and and it's it's not just mine you can see as Den showed you the growth trajectory a lot of people are pouring into it a lot of people are using it and

21:34 they have um significant investments from people, you know, owners of the code that are that are consistently pouring into it and getting into a true delivery life cycle. Um, I

21:45 think the last release was November and now we're talking 6 months later there's another release. Um, not to put any pressure on you, Den, but um, you know, are we going every 3 months here after?

22:00 >> Yeah. So, you know, they're getting on a cadence and and I'm I'm guessing after this event, they'll probably have more and more people involved in in the codebase as well. So, um

22:13 that help? >> Uh forget your email afterward. >> Yeah. Yeah. Great. Perfect. Well, thank you very much. Thanks. All right.