The Human Is an Async API — Melanie Warrick, Temporal

0:13 All right. Well, I'm here to talk to you today about this demo and I've already stepped outside the box they told me not to step outside of. Anyways, I'm here to talk to you today about an ice cream

0:22 delivery demo that I've got for you and ultimately I'm here to talk to you about Temporal. So, you're going to hear me pitching about the company that I work for and specifically in a way that will

0:32 be about this ice cream delivery that's happening as you can see online right now. So, what's happening? Well, this is Ziggy. Ziggy's Ziggy's is our mascot. I love Ziggy. And Ziggy has an ice cream

0:43 store, not really, but let's say he does down in the Ferry Building. Currently working hard delivering all kinds of ice cream. Now, what's happening on this screen is that um

0:54 we have multiple agents that are currently coordinating the delivery that's happening for deliver for Ziggy's ice cream service. We have a fleet agent and a customer agent and a

1:05 dispatch agent all working in coordination to make sure that the ice cream goes where it needs to go. In this specific demo, what I've got up here is showing you we have 80K integrated with

1:16 Temporal. So, 80K is a um solution that Google's providing to help you orchestrate agents or multiple agents um and in this case it's specifically helping to coordinate the activities in

1:28 relation to these agents. And Temporal is baked into it. We are integrated into it so that we can help track the state of what's happening with this kind of delivery. So,

1:39 Temporal actually helps you track your state and manage that state and we provide a UI that allows you to see what's going on on that screen. So, this is what

1:49 you're seeing over here. And in that UI it's showing you the event history. There's this this is the parent workflow for all the activities that are happening for Ziggy right now in its

1:58 delivery suite. And in this world, sometimes what we'll have in multiple agents is we need to have human in the loop. So, let's say a customer made an order and the customer said, "Hey, I

2:10 think I changed my mind about that order. So, I'm going to submit a change." And of course, when you do submit that change, it will not it if you depending on how

2:21 you build it out, it could actually break your system. But, heaven forbid you have it break your system when somebody has a change to submit. In this case,

2:30 we have an order and this order is currently pausing driver A. And so, driver A is waiting for that update to come through. The system is running while this one

2:42 specific change is only pausing one part of your system. That one change coming from a human. And another human needs to make a call

2:54 and whether or not they want to accept that. So, they say approve, and you'll watch that as I mentioned, the rest of the system kept running, orders were coming in, they're being

3:03 filled, but in this specific question that came up about changing that order, the update is getting processed and should go through in a second.

3:16 I suspect internet is part of it. You can see it does get processed, the update is read, and that order will now move on to its next location, which is to the Oracle Park. And there goes

3:26 little A delivering the order for you. So, there you go. That is durable execution with Temporal and 80K integrated. All right. I'm here to talk to you about the human is an async API.

3:41 Let's go into First off, we've been hearing about agentic loops, right? You can reason, act, and observe. That is our whole thing that everybody's been talking

3:52 about when they're talking about building out these agents and these agents running as autonomously as we possibly can. Um there's this kind of loop that we're

4:00 looking at, right? We've heard other words for this where we're thinking, we're acting, we're perceiving. Um and that's that's the essence of it. It can be a for loop, it can be a while

4:11 loop. Most of this year we've been hearing people tell us about the harness and what the harness is supposed to be to help us get those agents into

4:19 production. So, the harness I'm not going to sit here and tell you I I am the expert on the harness. Um I'm going to tell you from everything I've researched, I'm trying to give you at

4:29 least a map to a degree of what I understand so far with the understanding that there's a lot of debate about this. I've heard that the harness is possibly the model is is part of all of this or

4:40 it's not. I think there's there's debates around whether the model's in it, whether tools are in it, memory's in it, uh guardrails, security, but ultimately it's all about getting these

4:49 agents with structure so that they can work reasonably well when they're put out into production. And part of that picture should be durability. The system that I showed you, the demo

5:02 that I just showed you, this is what it is. It's got these loops. We've got a loop for the fleet agent, we got a loop for the customer agent, we got a loop for the dispatch agent.

5:11 The fleet and the customer are doing research. One's doing it on the actual drivers using thing tools like Google Maps. Um the other one is doing research on the orders like Google with tools

5:21 like Google Search. While the dispatch agent is taking that input and making decisions with it. All right. So, that's great that these activities are

5:32 happening with these three loops and we've got the human potentially providing input in part of this structure. But what's important is that we need to have some durability in that

5:41 system to make sure that it stays up and running and doesn't fall over. And that's where Temporal comes in. So, Temporal is helping you to standardize your fai- failure handling and your

5:51 state management in your system. Any system, any distributed system, especially any systems with agents, we are going to help you standardize that. We're software and we're also a service.

6:03 The software itself is creating a lot of primitives that you can just use out of the box and basically apply to your code base. We come in all kinds of flavors of code, uh so you can use this in Python

6:14 or TypeScript or Rust, um but ultimately we're giving you some standard primitives to apply to your code to be able to make sure your your state your

6:24 your system is stateful. So, the three main primitives we tend to talk about are the worker, the workflow, and the activity. The worker is really running that code.

6:34 The workflow is where you're going to be able to uh keep track of the steps in your system. And so, this is where like an 80K framework would be running inside a workflow. Uh your agent could work run

6:46 inside of a workflow cuz it's a loop, right? Whereas the activity is all the stuff that's integrating or interacting with stuff externally. So, if you've got a

6:56 loop, uh any kind of model call or any kind of tool call, those would be set up as activities. So, these are the primitives. So, you structure your code with our primitives to be able to track

7:07 the state of your system. And we're free and open source at that GitHub URL. Um you pay us if you you have us manage your state with our service. Otherwise, you can host it self-host it

7:20 on your own system. Let's get back to human in the loop. Why uh what I wanted to really get across to you about this whole talk. So,

7:30 the human, granted, you know, we're not tools, but we are a tool to the agent. Um and you might think, "Okay, well, we're going to call the human or we're going to have the human come in and give

7:41 us uh some kind of information to pass into this loop. If you write it just straight out as a function to say call the human or let the human have a an input,

7:51 it can become a blocking call. And it can get lost if your service went down. It might lose track of what was going on with that human's engagement.

8:04 What you want is these two core primitives to be applied to your your code and your agents. You want a wait condition and a signal. You're allowing yourself to isolate an

8:14 event, apply a wait condition, store that in that workflow, in that durability layer,

8:23 so that if anything fell over in the system, when that system came back up, it would know where it left off. And you could send a signal at any

8:33 point, but the whole system could keep running without you necessarily being blocked on that one call to the human. The code pretty much is is is fairly simple in terms of the primitives. And

8:46 granted, I know this is Python, but it's the same situation. You're going to apply a wait condition and you're going to apply a signal. You basically you're not holding a

8:55 thread, you're not it's going to survive any kind of crashes. It's got all kinds of built-in primitives like timeouts, for example. If you're waiting on a human to respond

9:06 and you want to make sure that it only waits for so long, you can apply timeouts and other types of primitives. Um and this can scale. You can have millions of workflows out there parked,

9:17 and it can keep running. So, what I want to show you actually I'm going to show you some more code, but I'm going to kick off the next demo so that it's up and running. And I'll share

9:26 with all of you, if you haven't already figured this out, uh my eyes are injured. >> [laughter] >> I am squinting at times and also

9:34 realizing that I need additional in support here that I did not prepare for, but that's okay cuz I can adapt. All right, so let's get this other example up and running and then I'm going to

9:44 tell you a little bit more about the code that's up here. Let's make sure that's running. I'm resetting my example and of course it's not going to be as easy peasy as

9:56 I'd like. Also, is everybody hearing me okay? Think so. Great. It's an interesting condition to be working in this space with the noises

10:07 around us. All right, I think it's up and running. All right, so let's talk a little bit more about the code. Now, I already told you about our primitives of activity

10:18 workflow and worker. And so with the activity, you are basically for ADK in particular, we have we have the temporal model class that

10:29 you will wrap around the model and then you'll pass it in for ADK to be able to set up that actual agent. It's pretty simple. You apply this temporal model, you apply our activity tool

10:41 to wrap around your tools and you you run your agent. You run your agent like you would with any kind of ADK setup that you're going to do. And you also set up a

10:52 worker and you pass in the Google ADK plugin. So that's setting up your worker workflow and activity. And then similarly, you have your wait

11:01 condition and the wait condition is using this async io when it uses a wait to say like for that specific workflow, I want you to pause it, don't run any other code, but you can keep running the

11:12 co-routines on that workflow or any other workflows in the system. Meanwhile, that workflow.signal, it will be able to take any kind of input and inject it into a running

11:23 workflow. And the workflow will be taken off of your memory or taken off of your system and put in in parked if it's not running anything at that

11:33 point. So, I showed you human to agent. What I want to show you is what does it look like when the agent comes in and needs the human to be involved?

11:44 It's pretty similar. You're using I'm just going to go ahead and tell you right now, you're going to use a workflow you're going to use a wait condition and a signal.

11:52 I'm going to use LangChain in this example and also 80K because the reality is with Temporal we can work with multiple frameworks. We are framework agnostic. We work across a variety of

12:03 different tools. Um so, in terms of LangChain they build out graphs and to build out the multiple agent setup. And in this integration the

12:12 way we work, you're passing us into each node for your graph. So, you're setting up with the metadata looks like. And if you're making any kind of model call like an invoke the model, that would be

12:24 set up as an activity. If you're making any type of other tool call, it would also be set up as an activity. But if it's a step, a step in that process, then that's set up as a workflow. This

12:35 is determinism and non-determinism in terms of workflows are deterministic, activities are non-deterministic. And then of course, you pass in for the worker the plugin for the LangChain.

12:50 In terms of LangChain in particular, we use the interrupt to call out from their node. So, this is part of LangChain and also to be fair, LangChain is more than a framework, but in this case I'm using

13:00 it as a framework. But you use interrupt to call out and then what you're going to do next is use that wait condition in the loop tracking for that interrupt signal to pause that workflow. And then

13:13 the signal will capture whatever the human human responds with to the question that's been asked. Okay, let's see what that looks like.

13:22 I'm going to bring up my demo and show you as they're running. This is the design that I've got. I've got the customer and the fleet agents running on 80K. I've got the dispatch

13:32 agent running on LangGraph. And Temporal is interwoven throughout. So, that's how that's working. Now, let's see if I can show you. In this case, I'm going to drop a

13:48 high-value order. This is an order that will most likely want a human involved. The workflow for the whole system is showing that all these agents are doing

14:00 their job. They're continuing to keep assessing orders. And the way I've got this set up is I have workflows uh child workflows that are split off for each order per framework. So, that

14:12 special order actually had its own assessment done. And here we see that the customer and the fleet agents did their work. They did their assessment. They came back

14:23 with a result. And they sent that off to the dispatch agent. The dispatch agent said, "Hey, okay, I need Oh, well, there you go. >> [laughter]

14:34 >> I need to get uh a human involved in this. So, I'm going to go into a pause state while I'm waiting for the human to respond." And

14:42 we know that humans are not going to respond in 200 milliseconds. If you are, that's great. Maybe there's a there's probably a couple people out there who are. But humans are most likely going to

14:51 take several minutes, maybe days, maybe weeks, maybe longer. And you can you can because this is just pausing that one specific event. And as you can see, the little cars are

15:03 continuing to keep going. So, now what I'm going to try to do is show you what happens if I was to take this service offline. So,

15:13 let's make it bigger for all of us. And kill worker. Cool. So, that takes it offline. You can see it's disconnected. Don't mind me as I do this in a very

15:29 interesting way. All right. If I say approve while it's offline, so what's happened is as I mentioned, this is stuff that's happening off of your service. It's over on its own data

15:41 store. So, we can still keep track of that event and queue it. And then when I bring it back up, yeah, I know. I have this really well organized here here for y'all. But, I'm

15:53 going to I'm going to bring it back up and then we're going to see if it works. Let's see. Make it. And it's back online. And you'll notice

16:01 that the dispatch agent will What's happening is that your event log is getting replayed. It's not redone. It's just replayed up until the point at which it last left off, so it knows,

16:10 here's my current state. And this is now the next step in that state. And it's been queued up to then have that step know that it that the person approved it, and that order got assigned, and

16:22 it's already been delivered, actually. So, there you go. That is durability in your system. Your system can come down, and it can recover.

16:33 And it can have all these other primitives that will help you manage a complex state because these kinds of distributed systems are complex. All right. Last couple things I want to

16:42 mention. When should the human be in the loop? This is the real question. It's a hard one. I will say this. The big thing you want to take into consideration is the

16:52 cost of being wrong is high. Okay, it's a case-by-case basis, and we know we're trying to get to autonomy as much as we possibly can. We also know these models are probabilistic, and they are hard to

17:03 get to a place to be reliable. That's why we got all this stuff happening around harnesses that are interesting. I could go into a whole 'nother conversation about that if y'all want it

17:10 at a later date. But, ultimately, you want to assess the cost of being wrong. What is that worth to you? And especially in a security standpoint. And we know like on the other side of this

17:19 picture, alert fatigue is real. Like a lot of us are seeing, you know, you just start saying yes, yes, yes, yes, yes every time it asks you questions. So, you're trying to balance between those

17:28 two. And we're trying to get the models better so that you can trust them, so that you can address that alert fatigue, but they're still probabilistic models. So, when should they be in the

17:39 When should the human be in the loop? That's the thing you've got to evaluate for the problems that you're solving. At the end of the day though, what you saw there, what I gave you as an

17:48 example, whether it's the human initiating it or the computer initiating it, um you were seeing that we are using two main primitives in terms of you're applying a weight condition to pause

18:00 that workflow, and a signal from the human to be able to continue the work and say, "Here's the result. Now keep moving forward." And And whether you've actually paused

18:11 it and taken it offline or you brought it back online. And I've shown you that we can use multiple frameworks. All right, if you have any questions,

18:21 I'm going to stick around for a few more minutes. Um if you I've got up here a QR code that will take you to some resources about Temporal, as well as some of the demos that I've done, this

18:31 demo included. This is up on GitHub. Uh the slides itself will be up later cuz of course in true form I've been tweaking them up until now. So, I'll be posting those up and linking them to the

18:43 GitHub repo. But yeah, if you have questions, please reach out. Um and you know, have fun building your agent clips. I'm also curious to hear what your what problems you're solving

18:52 with agents and how you're building them autonomously currently. So, feel free to find me. And thank you. >> Woo!