BreezeTTS2 - 100% Local Real-Time Voice

0:00 Okay, a brand new openweight TTS model just beat every other open model on the leaderboard. This model came out 6 days ago and I've already got it running locally so I can show you the real

0:10 thing, not just the demo page. >> All right, so that was the new TTS system called Breeze TTS2. And in this video, I'm going to go through what it is, talk about how it's

0:23 very impressive for a lot of different things, talk about some of the sad things that are not ideal here, but also just show some demos of where TTS is at the moment for open weights models. All

0:36 right, so this comes from a company called Breeze Blue. Now, up until a few days ago, I hadn't really heard about them. I didn't really know anything about what they were building, etc. But

0:45 it seems that they are a startup in China that is working pretty much solely at the moment at least on sort of voice and TTS systems and sort of ideas for where these things are actually going.

0:59 So this has got a lot of the sort of standard things that you'd expect from a modern TTS system now, but it's also got a number of really interesting things as well. Now first off, they can do voice

1:10 design. So this is where you're basically creating a distinctive voice from a description of that voice but without any sort of reference audio. So for example, if I describe a character,

1:22 I tell them what the perhaps accent would be like, things like that, it will then create a voice for me. And you can see that they're comparing themselves here to a bunch of different models that

1:33 are out there, including things from 11 Labs. And sure enough, this is doing really well here. If we come in and listen to some of their examples. >> This is Elena Vance reporting live from

1:43 the east block where the crowd just broke the barricade. If anyone can hear me, we need medical teams now. Please, they're getting closer. >> Okay, so you can see that that was

1:52 basically a prompt plus a script. And this is one of the common things that they will have a prompt for describing the voice or the emotion that you want to convey. So you can see this one was a

2:01 young adult mid20s American accent, professional broadcast journalist turning frantic. So we've got clear, articulate, but strained by panic, high tension, rapid pace, breathlessness,

2:13 polished to raw desperation. So that does a pretty good job of explaining that particular voice there. And you can see some of the other examples too of this one where the prompt is for a male.

2:25 Let's just listen to that. I've walked through too many long nights to mistake silence for peace. So when I speak of tomorrow, believe me, brother, I mean a dawn we can finally stand in together.

2:38 Okay, so we definitely have the whole sort of southern accent and stuff like that coming through that they've described in their prompt here. And that's one of the things I noticed with

2:49 the descriptive things is they can do accents very nicely here. All right, so the second thing that they're focused on is voice direction. So this is cloning a voice from audio but steering the tone,

3:02 emotion, and the delivery for it. So for example, here we've basically got a reference voice. Before touching a formula, ask what question am I answering?

3:12 >> And then we've got a prompt here talking about speaking in a hushed intensity, hypnotic and persuasive rhythm. And then we've got the actual script. You feel the energy radiating from it, don't you?

3:22 This isn't just a piece of quartz. It's a conduit for a small offering. You can take it home and finally clear that dark cloud hanging over your future. Trust me, you need this. So, you can see

3:34 there, especially in the bits that perhaps not totally hushed voice, but is actually taking the reference and running with that. So, here's another example of a reference audio.

3:45 >> Let the breath move like the tide. In as the wave draws up the sand and out as it slides back to the sea. Okay. So there you had some kind of guided meditation etc. Now you can see we've taken that

3:59 soft and very sort of gentle passive voice to now become passive aggressive and with sarcasm in it. >> Enjoy your vacation while I stay here and clean up the mess you left behind

4:10 for me. Have a wonderful time. >> Okay. And you can hear that it really does capture the emotion there. That's pretty impressive of how it's actually doing that. So on top of voice cloning

4:22 and voice design and voice direction, it also has the ability to do vocal events, meaning that you can basically pass in things like getting it to laugh, cough, clearing the throat, sighing, a whole

4:35 bunch of different things like that. And we've seen that before in models and sometimes it works really well, sometimes it hasn't. With this model, it does actually come out reasonably

4:45 nicely. Now, another thing that people often are asking for with the TTS is for multilingual speech. And here they've got 50 different languages that you can set it to actually speak in.

5:11 Okay, so I don't really speak a lot of those languages, but it certainly sounds pretty impressive in the pronunciation stuff. And then one of the ones that I think is really impressive here is just

5:22 the low latency. So you can see here that they're comparing themselves against a bunch of the other TTS systems that are out there. And out of the box, it can do full streaming. You've got

5:32 sort of time to first response back being really quick. And that's partly because of the model itself. If we come in and actually look at the model, we can see that this is a three billion

5:42 parameter model. Now the version that they've uploaded is a 16bit version. But there are other versions out there already where people have basically put this on MLX. People have made versions

5:53 for CPP. People have created 4-bit versions of this, 8bit versions of this, etc. And sure enough that they're touting that on the Arena Voice leaderboard here, they're actually

6:04 topping it for openweight models. Now, it looks like they're a little bit behind compared to some of the other models that are not open weights, but undoubtedly this is an impressive model.

6:14 So then the big question before I go into the actual demos that I've made with it is what's the downside? Well, this is the downside. It's that it's basically a research and non-commercial

6:26 license. So they are actually limiting what you can actually do with this. Not only can you not generate outputs for commercial use, but things like you cannot distill from it, you cannot do a

6:37 bunch of different things with it. And I kind of feel if it wasn't for that, this would become the go-to model for doing a lot of different voice generation, whether that's generating audio for

6:46 games or for assistance or for a whole bunch of these different things. So if you're looking to run this by yourself at home, you've got no issues. But if you're looking for to use it for

6:56 commercial things, you're going to want to check out their pricing and quotas on their actual site. But for now, let's jump in and actually try this out. All right, so for the demo, I'm going to be

7:05 running this 100% locally. Thank you to Dell for sponsoring the compute on this. I'm running this on my Dell T2 Pro Max machine with an RTX Pro 6000 in there. This is then all streaming across tail

7:20 scales to the computer where I'm actually doing the recording here. Now, first off, let's just look at sort of voice design. So, you've got two things that you can put in here. You can put in

7:28 what you want them to actually say, and then you've got the actual voice description of what the voice is going to be like in here. So, you can see this one I thought was quite funny.

7:38 >> A world where nothing is what it seems. One developer must ship before Friday. >> So, kind of like the movie narrated for that. The other thing too is that then we can start to play around with some of

7:50 these. So you can see that it actually generates very quickly here. So I'm not using the streaming version here. I'll show you a demo of that next. I think in some ways that's probably even more

8:01 impressive. But here you can see, okay, just describing a voice and giving it a script and then we can generate out. >> Welcome aboard. Your journey begins now. >> Okay, so that's the first thing. The

8:11 second thing you can do is actually clone a voice. So here what I've got is some reference audio. So this is actually a Chinese voice. Okay, so that's literally 4 seconds of

8:24 reference. You need to basically transcribe what they're actually saying and stuff. And then now we can generate out using that here. So you'll see that it generates out pretty quickly. And now

8:35 let's listen to that voice speaking in English. >> It is good to hear your voice again after all this time. >> Okay, so that could be any of the 50

8:44 languages that they support in there. The next thing we can do is we can clone a voice, but we can then guide it. So just like we were doing the voice direction before, you can see now we can

8:55 take that voice and do a direction with it. So we've cloned it, but now we've given it some direction of speak slowly with a strange serious tone. >> We need to discuss what is happening

9:09 with the closed AI labs. >> And so this shows you how you can generate lots of different voices like that. Now you've got a bunch of sort of vocal events that you can use for things

9:20 like cough, laugh, sigh. I think some of them are very hit and miss. So that's something that I think you want to play with. I do think it's very interesting. The direction tends to work really well

9:31 on this. So it's got really nice proity. If I ask it to do a sarcastic version of that voice, >> we need to discuss what is happening with the closed AI labs. You definitely

9:45 can get a different sort of feel there. I think the sort of sarcastic clears throat maybe doesn't go well together. But anyway, let's look at the actual sort of streaming version of this and

9:55 see the streaming version of this where we can focus in on one voice and then basically put it all together. Okay, so in this demo, I'm testing out the actual speed of this. So this is the voice I

10:07 clone. I actually first took one of the generated ones that they made in Chinese. I like the tone of that one, so I cloned it and we're doing it in English here. What I'm going to do is

10:18 I'm going to turn off my mic so that it doesn't get in the way. The challenge I always have with these is that the audio bleeds out into my mic and messes up. So, what I've built here is basically a

10:29 demo of where I can type something and then she's going to speak back to me. And then I've got an auto reply button here basically where it will use a Gemma model, a small Gemma model to generate

10:42 both the replies to her and then what she's saying back. You'll see that when I press this button, it's going to basically feed in my response to her rather than me cutting in sort of saying

10:52 it and stuff like that if I had an ASR system. So, I'm going to start it off simple and then I'm playing this all back real time and you're going to see the stats of how fast it's buffering and

11:01 stuff like that as well. Hey there. I'm doing pretty good. Just tinkering with some new plugins. How about you? Oh, you're running simulations. What kind of models are we talking

11:13 about? I haven't dipped into the latest LLMs today. I'm more into analog gear right now. Not really. I haven't been testing those

11:24 specifically. Everything seems smooth on my end. Stutter in the environment rendering. H I don't have an eye for that kind of technical stuff. Maybe you should check

11:36 the rendering settings. Realistic enough is always a moving target, isn't it? It depends entirely on what you're trying to achieve. >> Okay, so you can hear there

11:46 realistically her response is just milliseconds after me pressing this and that the model has generated what I say to her. So when you combine this with an ASR, you can get a full two-way

11:58 conversation going. And I should stress that this is running on the Dell machine in my office and then streaming over tail scale to the computer where I'm actually doing the recording and stuff.

12:08 So it does show you where you're getting to the point where we can actually have these kind of interactions 100% locally with multiple models. Now at the moment to do it, you still need a beefy GPU,

12:20 but the quality here is pretty amazing considering this is a 3 billion model. I'm running it at full resolution. I could run a quant version of it and it would even be faster than this. Okay, so

12:31 just finishing up. I think that this model is definitely sort of up there around the sort of Quen 3 TTS level. Obviously, the biggest issue here is the license. And my guess this then puts

12:45 pressure on people like Quen to release a newer version of their TTS because we know at the same time you've got a lot of companies whether that's OpenAI with whether that's the Space X team or the

12:57 Gemini team working on newer better versions of these for APIs. So I do think at some point this is going to be interesting to see whether as they bring out Breeze TTS3 do they make this one

13:10 fully open. Now there are some issues with this here. But the way it's been trained is it hasn't removed signal processing before they've actually trained the model. So you get a lot of

13:21 the times where the sound of the voice actually has a particular EQ to it or artifacts like the microphone and where it's recorded. That's especially true if you listen to things like the newsreader

13:36 voices and stuff like that where it's made to actually sound like it's coming out of a TV or out of a small speaker. So, personally, I think that's not ideal. We would like to be able to

13:46 separate the voice from the signal processing and be able to have those run together. But this certainly is a nice model that you can run yourself at home and use for doing real-time streaming

13:58 chats and for just having something where you want your computer to send you alerts or something. It can print out the voice and then send it across Telegram, send it across something else.

14:08 So let me know in the comments what you think of the real time stuff here. While we'd all like to have the models of this quality be the size of Kakoro or something like that, that's probably not

14:18 going to happen with the current architectures that people are using. But I do think for many people running a 4-bit version of this is probably going to get you almost the same standard of

14:28 audio and a very significant reduction in file size and time to actually process that. So, let me know your thoughts in the comments. As always, if you found the video useful, please click

14:38 like and subscribe and I will talk to you in the next video. Bye for now.