Enjoying this issue?
Get tomorrow's AI & engineering digest in your inbox — hand-picked, summarized, and always spam-free.
TLDR
OpenAI's ChatGPT Voice 2.0 introduces a real-time conversational agent that can multitask, control desktop apps, and follow users across screens, effectively acting like a voice-controlled assistant. The update enables hands-free operation, integration with other agentic tools like Hermes, and the ability to perform complex tasks such as web research, file management, and form filling through voice commands.
Key points
- ChatGPT Voice 2.0 allows real-time, interruptible conversations with a voice agent that can multitask and perform actions on the desktop.
- The agent can execute multiple simultaneous tasks like clearing downloads, opening tabs, and building landing pages via voice commands.
- Users can control the agent remotely from a phone, asking it to retrieve desktop information or perform actions while away.
- The voice agent can analyze web pages, explain errors, and recommend products by taking screenshots and using context.
- It integrates with other agentic systems like Hermes, enabling voice-triggered reminders and configuration checks.
- Level five capability allows the agent to click buttons, navigate interfaces, and fill forms based on user context and memory.
- Drawbacks include slower browser control, no live camera support, and limited memory compared to standard ChatGPT.
- The update is available on Mac desktop and mobile apps, with a new faster model expected soon.
Tools mentioned
Techniques
- Voice-controlled multitasking
- Real-time conversational agent
- Desktop and mobile integration
- Browser control and screen analysis
- Agentic tool orchestration
Stop scrolling. Start reading smarter.
Receive the day's most important AI & engineering updates in one concise email. No spam.
Transcript (captions)
OpenAI just dropped ChatGpt Voice 2.0 and it unlocks incredible capabilities. It is essentially Jarvis. It connects all of your tools, works from your phone, and it will change the way that you work, but only if you know how to use it properly and the best use cases. And in [music] this video, I'm going to show you the five levels of Chat GPT voice step by step [music] so you know exactly what to expect at each level, the key mistakes to avoid, and so you can get light years ahead of everybody [music] else and save time. So, if you haven't already, grab that beautiful coffee and let's dive straight in.
Now, open up the Chat GPT app or the Codeex app. And by the way, these use cases are going to blow your mind. They are incredible. Now, I'm going to come down and click on start voice. Very, very cool.
like so and then we can begin the conversation. So it works across our desktop. It we can explore, we can plan, we can learn and it works across our projects with some really cool use cases that you have to understand. So we go ahead, we pick our voice and then we're ready to rock and roll. Hey there, could you do me a favor?
Could you do a quick Google search and find for me who was the tallest man that ever lived and then save a file on my desktop called tallest man and have that information inside please on it. I'll check that and save the file to your desktop. And crucially guys, we can basically mute this, move it around if we want to. We're going to allow this all the time. So we don't have to basically keep on permitting it.
So now we have a basically conversational interface with chat GPT and it can do things on our desktop. Now, as you can see, we have the information right there. Now before we go through level one, we first have to understand what's actually happen here. And by the way, if you're new, I'm Jack. I built a small tech startup with like a gazillion customers.
And now I share here this stuff that actually works. If you find this helpful, feel free to comment, subscribe, uh, down below. Now, here's the thing. The app hasn't changed. The engine behind it actually has, we have two new models already has 150 million weekly voice users.
And instead of having back and forth, you can basically talk to this agent in real time. So, you can interrupt each other and it will absolutely get what's going on. And the reason why it feels smart is simply because it will keep talking to you whilst it's thinking in the background, which means you're kind of waiting 15 working days to get an actual response back. And as you can see, the performance upgrades here are incredible. Now, the cool thing here is we can do this on your current desktop, your laptop, and we can also do the exact same thing from a phone, which is incredible.
So, we're going to cover chat GPT voice across five levels of difficulty, beginning with level one, which is its ability to multitask. Check this out. Let's begin by pressing the toggle down below and lett voice launch. Now, I'm going to give it three tasks simultaneously. Hey there, dude.
Do me a favor. First of all, I want you to go ahead and clear up my desktop. It's looking a little bit messy. Then, I want you to go ahead and open a brand new tab uh inside of the Codeex app. And I want you to do some deep research using firecrawl on the best strategies to grow your ex account in 2026.
Um, make sure you do some good research. Uh, find loads of different competitors and make that fantastic. And whilst you're also at it, could you just build for me a really simple landing page um, for my client's website? U, make it interactive, make it beautiful, lots of different HTML. >> I'll start on those together to the desktop safely and sheet the landing page.
I'm checking your screen in context first. So, this is great actually, guys. And I love the fact I can move this around. It's fun to me that it may not be convenient for me to have it down below. So, we can move this anywhere we go and it will follow us around on any app that we're on.
For example, now I'm over here on a different page. Again, any website I'm on, I can be followed along by this wonderful little icon that we can chat to at any time that we want to. And there's some incredible capabilities that this unlocks in a second, which I'll touch on very briefly. Now, again, it's doing all those different things by itself, and it's asking for a clarifying question. Before I move anything over, watch what should state on your desktop versus go.
Hey dude, what I'd like to do is basically keep everything on my desktop. I just want you to organize them into better folders so I don't have any loose files. In fact, scratch the desktop. Just do this with my downloads. Don't touch anything on my desktop blues.
Okay, cool. I even gave it a slightly contrarian command now. And we're going to see it do these three things simultaneously. Now, the reason I'm using firecrol here is simply because it's going to save me a ton on the tokens that I'm using. And I've always found whether I'm building websites or doing any kind of activity, using the ability to scrape, interact, and pass websites saves you lots of tokens and actually cost, which makes it a faster experience and you get way better results uh because of using this kind of thing.
Now, what's really cool here, the fact that we actually have trackivity on the website, one of the things I can do while I'm here is actually have a conversation with it. So, I can even say, "Hey there, I'm on the Firecrol website. Um, so we're using this, right, to scrape things with our tools. What I want you to do is can you just do me a favor and um explain to me the things I can do on this page please. >> Checking that page now.
You're on firecrawl search playground. You can search the live web and turn results into clean AI ready content. There's also scrape for one URL crawl. Beautiful. So you can see basically now chatbt can be on the website and explain some things to us which is massively helpful.
And again we can bring it anywhere and continue on our conversation which is great. To do that, it does have to take a screenshot of the web page, which is fine, but again, it just saves you that task. And as you can see, chatbt has even opened up a new tab as per my instructions that we can click into and have a look if we want to, which is great. Beautiful. Then within the tabs, we've got everything.
We have the X report, don't report the AI news, tester, earn attention, make provisible, convert intent. Really nice crisp thing. It's done via voice mode. Then we have actually, interestingly, a couple of different options here. We have a website here, which is cool.
Um, we make what tomorrow needs. Really good. It doesn't feel like a classic AI slab website. Going to come down here. You can see proof not promises.
I think this is really good, guys. I just did this on voice mode. Very cool. But this is the kind of thing you can do with it. Give it multiple commands.
Give it the permissions and let it rock. So, now we've given it three different jobs in one voice. We can then start looking at actually taking this a level further actually, which is the ability to actually go ahead and use chat GPT when we're not even at our desktop. Now, all you've got to do is connect your chat GPT account to your desktop and you can download that by s basically um signing into the exact same account. So, let's say for example that you're heading off to the coffee shop.
I can come down and ask it any question about anything on my desktop. I can even get it to build for me. I could say, "Hey there, man. Could you do me a favor and just remind me what was the last thing I saved on my desktop? >> Let me check your desktop now." Checking the most recently saved item.
>> Not only can it retrieve information, it can actually do work for you on your desktop as well. So, now you have the multiple tasks. We have the ability at level two to effectively control this cool um a kind of agentic voice mode anywhere we go. That takes us nicely onto the third level of capability which is the ability to give it a very vague idea and get something significant back. And what I mean by that is once you bring it up in any window, you can actually go ahead and use codecs for different tasks.
For example, let's say that I'm on this page here. I might just have a chat with Codex and have a conversation. So, you could say, for example, hey there, I'm just trying to buy one of these products I can see on screen from 11 Labs. You know a little about my contacts. You know about the build I'm trying to do.
Um, which of these four would you recommend that I go for? Frearter, creator, or what? Okay, I'll check those plans and match them to your use. Got it. I'm looking at the 11 laps plans on screen and given your build, I'm narrowing down which plan fits best.
>> So, as you can see, it already has the context what we're doing. And what I like most about it, guys, is that you can have this conversation and then I could do other things. I could just go on to a separate web page. I could go on to YouTube. I can mute it.
I can do anything I want to and it follows you around the internet. Now, what's also really cool about this is let's say that I click on a random video. Okay? Like so. I can right click on something and actually on the web page itself, I can come down and do inspect.
So, if I'm looking at anything online, I'm doing work. You can see there's loads of errors on the right hand side that happen sometimes when you're building apps. I could for example do this. come over to attract BT and say, "Hey there. Um, I just want you to explain to me the errors on this page.
What's going wrong here?" Sure, I'll check the errors on this page. Beautiful. And then you can carry about your dayto-day doing everything you want to, which helps you build better apps because you have this blue thing following you around everywhere that you go on all the different apps that you're using, which just enables basically just saves you time as you're going through everything. Now, when you do actually load up the chat GBT voice mode, you can pick the voice that you want. Obviously, you go from like GBT 5.6 6 soul terite.
I quite like GBT 5.6 soul high when I do it. I come down and then when I open it, that is the exact model that you're going to be talking to when it loads up, which is massively helpful. And by the way, if all this sounds like I'm speaking Spanish and you do want to go deeper, I'm going to put a link down below for the full coding uh cla code master class we built that takes you through foundations, building websites, memory systems, homes, agent apps, and everything else. All the principles that work across chat codeex are exactly the same in code, and you can go even deeper in some respects. So, I'll put a link down below for that if you want to go ahead and grab that.
Now, what's really interesting is that we talk a lot about Hermes agent right now. If I come over to Hermes agent, check this out. I can actually trigger my Hermes agent by my chat GPT, which is incredible because I run Hermes agent on my computer, which is awesome. But I can trigger it both via voice here and on my desktop. And the reason why this is such a huge hack, whether you're using Open Claw or Hermes agent, it's incredibly beneficial.
Now, check this out. I could say to it something along the lines like this. Hey there. Um, you have access to my Hermes agent. I want to go ahead and tell me what is the model that is powering Hermes agent right now.
>> Checking the Hermes configuration directly rather than guessing. >> Okay, awesome. So, it's going ahead and checking the configuration. Why is this handy? You know, anything you're doing on your computer or if you are just out and about, you can drop these voice messages.
If you happen to be building the chat GPT or you want to connect it to your other agentic systems, we can do that using this software. Hermes is currently configured to use GPT 5.6 Saul the Open AI. >> Beautiful. That's massively handy. Do me do me one more favor, please.
Could you set a reminder? Ask Hermes agent to remind me at 11:00 a.m. in the morning to get my two iced coffees for my flight to toat, please. >> Sure. Setting that up.
>> And so we can also trigger reminders to our agent. >> Remind you at 11 a.m. >> Let's do that tomorrow. Let's do that tomorrow. >> Okay.
Locking that in for tomorrow. >> Beautiful. So that's exactly how we can communicate with our Hermes agent because my Hermes agent runs on my laptop, my computer, my desktop. So because it runs on the same computer, it can all run on the same thing because codeex is there too. And it just unlocks new capabilities.
What I think is really cool is you can have this just following you around on everything you're doing. Especially if you've got a great um you know conversation and you want to have like support with building something, you can just bring it in and in many ways it's a nice kind of like um if you think about it UI update. It really makes it a little bit nicer to use. Then this takes us on to level five, which is a really exciting update and I really like this where effectively you can be completely hands-free and actually have this beautiful floating agent on our screen to do things for us. For example, let's take a look at this road map that I built.
Okay, this is something I built my community to help people with their like goals and strategies. I could come down here and check this out. Hey there, bro. Could you do me a favor? Go ahead and select the ninja and then I want you to click on to the next page.
All right, doing that now. Ninja is selected and the next button is enabled. >> That sounds cool. Go ahead and analyze my profile. >> Got it.
Analyzing your profile. The profile analysis page is open and ready to run. >> Cool. And now just click that button for me to analyze my profile. >> Okay.
Running profile analysis now. So what's really interesting here, guys, isn't the fact that it can click buttons. It's the fact that it can look at the screen, do various different things, analyze what's going on, and go to the next step. But where it really takes to a new level is the fact that it can actually take all of the context and information that has about you and then specifically make decisions. What's really cool and interesting here, if I just come down and just say $10,000 as an example, what I can do is actually get chat GPT to fill in this information based on what's happening.
So I could be like here and say, "Hey, you know a lot about me and it's asking a lot of questions. I'd love you to go through this and just help me select what you think would be most relevant based on everything about me and my situation, please. Sure. I'll match the choices to your profile and goals. And what's really interesting here, guys, is it will go ahead and fill out information for us based on everything that we know, which is great and does make things a million times easier.
Now, we talk about realistically a couple of the drawbacks that are great to know. The first one to know is that this is not the fastest thing on the planet if it's doing browser control. It's great if you want a hands-free experience and run through things, but it isn't insanely fast. They are releasing a new model soon, I believe, that is even faster, which is going to be incredible. So, a couple of good caveats to know, guys, is that it doesn't have a live camera, so you can't point your phone at something and ask about it whilst GPT live.
Um, the screen is Mac desktop only, so there's no screen sharing into a live voice session, but the Mac OS desktop app does have appshot, so it can read whatever's on the screen. Now, it has a memory, but not quite as comprehensive as using chat GPT. So, if you're asking it questions, it's good to refer to specific files sometimes to get that extra level of detail. So, now we know exactly how to use this chat GBT voice agent. The next thing that we need to do is learn how to take this to a completely new level, which we're going to do together using Hermes agent.
And we can do that in this video right