Realtime Voice Agents with Frontier Intelligence — Bohan Li, EliseAI

0:01 [music] >> My name is Bo. I'm going to be here presenting real-time voice agents with Frontier Intelligence. Effectively, going to be talking a little bit about

0:21 how we at Xnor.ai architected our voice agent harness to get real-time voice with the Frontier level of intelligence that we need. Okay. So, before I start,

0:35 I think I wanted to kind of draw some parallels about why we decided to go with cascaded voice agents and especially kind of comparing that to self-driving cars

0:47 which I was working in before. So, to me, cascaded voice agents makes a lot of sense when you view it in lens of kind of breaking it down into perception which is

0:58 for self-driving cars, it's you know, the bounding boxes, the camera, the lidar. For voice, it's going to be the transcription. Basically, effectively

1:07 turning these like signals from the real world into elements of data that the language model or whatever brain you're working on can process.

1:19 Second one is the planning step which is pretty straightforward. This is where the language model will take in the outputs from the perception stage and produce the outputs

1:30 that you want to produce out back and out into the real world. And finally, there's the controls layer where on self-driving, you'd be taking the

1:39 trajectory that the planner would output and kind of turn it into the real controls to kind of build like drive the car. Here, we're turning the text into audio that we use to express our voice

1:53 agent's thoughts. And um yeah, so here I'm going to be like going to diving into each one of these elements and we've made a few kind of

2:04 interesting tricks on each of these areas to improve the speed of our voice agents without sacrificing the intelligence. So, the first one is going to be uh the

2:16 transcriber layer. So, we came up with this concept called like the streaming speculative transcriber where effectively we are layering a fast streaming transcriber like Flux on top

2:27 of or kind of below a uh scribe V2 or a accurate batch transcription which kind of takes in more context. It's a little bit slower, but it will give you more accurate

2:37 detections. So, we're going to walk through a scenario. So, in this in this case the agent just asked, you know, providing can you provide your name and date of

2:46 birth and the user is going to say this and we'll see how that plays out um timing-wise. So, first we're going to get, you know, the short detection. Um we'll get it from we'll get it from the

2:57 streaming layer. The accurate layer uh the corrective layer is not going to fire because it's the same text. Um we're going to get some more

3:07 streaming text detections and in this case the corrective layer is actually canceled because we got new um new text. So, you know, more context, more audio is going to beat the old accurate one.

3:21 And here's where kind of the first correction comes in. So, because the scribe V2 layer understands, you know, the the context of the question, it's able to understand that this is talking

3:31 about name and this is a date of birth. Then a couple more detections, these are just punctuation, we don't care. And so, in the end we kind of release this text over to the agent.

3:44 And moving on um to the language model layer. So, here since we're kind of using these slow but intelligent LLMs, we really want to reduce the number of round trips

3:55 and the thing that causes us to do a lot of inferences is tool calling. So, one way to get rid of that is by having background agents do the tool calling for you and kind of

4:08 um push the tools back into the context of the main agent so that it thinks it made the tool call, but um but it it it really didn't.

4:18 So, uh so, we remember from like detections from before. So, well, what happened is each one of these detections is going to trigger a

4:27 um an early kind of generation of the agent and we but we won't actually emit this out until we're confirming that the user has finished speaking. So,

4:40 in this case, the user says, "Sure." The agent kind of knows that the user is about to say something else. Our background tool calling here, which is going to be helping us find figure out

4:49 the name and the date of birth from the user detection, is not firing. So, nothing much there. Um the next instant detection comes in. It says that,

4:59 you know, still not really a name. Um our agent kind of plays along and continues there. Now, kind of a more more context come comes back. The agent kind of feels like

5:10 there should be a name. It's going to ask to spell it out because it's probably thinking there's some transcription error here. Still no name or date of birth.

5:19 And then finally, this you remember this is kind of our corrected um final instant detection from the transcriber from the Scribe V2. Um here, our eager kind of agent

5:30 generation that was made without any tool calls is going to get canceled because the background agent finally is able to find the name and date of birth it's looking for. So, it's going to

5:40 retrigger and now the the agent actually has the context it needs. Um and you see here, it's kind of we're doing it the tool call here is a little bit um um

5:49 some intelligence there. We're going to be like, you know, correcting mis-transcriptions of name, and doing some like phonetic matching here. Um and yeah, and then we'll kind of

5:59 once we've understood that this is the end of the user utterance, we'll kind of emit it out. So, pretty standard. Okay, and then the next layer here is going to be text-to-speech. So, with

6:10 text-to-speech the goal is to kind of take what the agent said, and the agent's going to be emitting this in a streaming fashion. So, we're going to need to

6:19 um produce audio as quickly as possible. And ideally, what you can do is before the agent has even finished generating the full text you can have the audio play, so it's kind of hiding the latency

6:32 of finishing the generation. So, um I'm going to kind of play the streaming um the stream the streaming uh agent output

6:40 now. So, starts with you. And yeah, actually before I uh further, there's this new concept that we're introducing here called the prefix cache. So, the prefix cache is going to

6:51 be looking at the um agent stream, and seeing if we already have generated audio for that sequence of words um from like a prior generation, or maybe like the same

7:04 generation um in this uh in this call as as well. So, um it sees the word you. Uh we for this prefix cache, we're going to be,

7:15 you know, we don't want to like immediately hit on every single word. We're going to be waiting for a little bit more words. Um so, after three words, the prefix

7:25 cache gets our first hit. And um over here on the right, this is kind of our text-to-speech standard provider, you know, Cartesia is a text-to-speech engine with web socket

7:35 support. So, we're we're piping the agent through the the cache, and also piping it through web socket. Um more tokens come in, more cache, more sending through web socket. Not much to

7:49 say here. And okay, so now we get our first uh kind of first unique thing, which is we found the token that actually causes a cache miss. And it makes sense. If we're

8:02 kind of caching previous generations, um you said your name is is a pretty common thing, but once you we add in the name, suddenly we're that's that's going to result in the cache miss.

8:12 At this point, we're actually going to yield out our cached audio. So, you said your name is is going to be um emitted as the rest of the streaming text is coming back. So, at this point,

8:23 the user hears the agent and user doesn't really know what's going on. They just looks like really fast response times to them. Um and now the kind of remaining text

8:33 flows through. And at this point, we've already emitted from the cache. The cache has done its job. Um the rest we can kind of throw into Cartesia.

8:43 And here's kind of the trick where Cartesia has seen the entire transcript up to this point. It to to Cartesia, like it doesn't know about the existence of this prefix

8:57 cache. It's just going to generate this full sentence with, you know, standard natural prosody. But, what we do is when the generation comes back, since we've already played

9:08 the audio here, we can actually suppress the audio from Cartesia here and just play out the remaining stuff. So, the user, there's might be a tiny bit of a hiccup. You know, I'll play some audio

9:20 later and you'll know that you probably won't be able to notice. And effectively, we're kind of taking this and or taking this part and just emitting it

9:31 directly after this audio has finished playing. So, it looks seamless to the user. And yeah, so I guess at this point kind of this is a lot of technical details.

9:42 I'll I mean the best way is just kind of play a call and see see this in action. So here I recorded a call and let's just hear what happens. >> This is Bo OBGYN Elise speaking.

10:00 >> Hi, I think I might be pregnant and I want to schedule an ultrasound to confirm. >> That's exciting news. Can you provide your name and date of birth?

10:12 >> Sure, it's Elise Trial. Date of birth 2303. >> Thank you. Are you a new patient with us?

10:26 >> Yes. >> Welcome to Bo OBGYN. Can I text you a link to upload your insurance information? >> Sure.

10:39 >> Okay, sent. Let me know when you've received it. I've received your insurance information. The earliest appointment I have is Thursday, July 2nd at 10:00 a.m.

10:55 >> Actually, give me a second. Let me check my calendar. >> Sure, take your time. >> Um do you have something for next week? >> For next week, I can offer an ultrasound

11:11 with Dr. Avery Stone at North Clinic on Tuesday, July 7th at 2:00 p.m. or 3:00 p.m. Do either of those work for you? >> Yeah, 2:00 p.m. works. >> Great. Your appointment has been booked.

11:26 We look forward to seeing you then. >> Thanks. Bye-bye. >> All right. Yeah, that's pretty much it. Um Yeah, you can kind of see our

11:38 all this like uh streaming and you know, a lot of things are happening in the background and and you know, this is what really makes like

11:46 voice agents interesting. And um there's a lot of effort that can be done in the harness to really kind of get a um natural conversation, which is what we're after.

11:59 Uh okay. Yeah, so I guess briefly, you know, in the last part, I want to just talk a little bit about Elise. So I think Elise, you know, our headquarters are in New York and kind of we're trying

12:10 to expand our presence here in the Bay Area. Um we I think it's maybe like a different style of company that I think people are like uh think of when they think about

12:21 AI startups in San Francisco. Where we're actually very focused on um just like helping people and helping people where they need it, like kind of the life's most critical areas. We work

12:33 on housing, health care and uh we're doing really well and you know, here's there's a link here to um you kind of join our team and there's going to we're going to be uh posting a

12:46 lot on Twitter, so you can follow us at EliseAI as well. Um yeah, that's that's it. >> [applause]