Google Just Gave Your Coding Agent a Real Phone.

0:00 It's Friday afternoon. Your coding assistant reports the login fix is done. Test screen. DIFF clean. A very confident summary. But nobody opened the app. Nobody tapped sign in. That signin

0:12 button now sitting just under the keyboard is still there patiently waiting for a real user to find it on Monday. So here's the honest question. What would it actually take for that

0:20 assistant to open your app, look at the screen, press the button, and then tell you what happened? with evidence you can inspect yourself rather than a sentence you have to trust. Google published a

0:31 repository called Arteimus. Apache 2.0 licensed. In its own words, "It turns natural language instructions into reliable Android automation. It automates endto-end workflows, captures

0:42 logs, and integrates with AI coding assistants like anti-gravity, codecs, and cloud code. One thing before we go further, I did not install this. No device was connected, no task was run,

0:53 nothing was reproduced. Everything ahead comes from Google's documentation and the benchmark paper behind it. When I hit a number I can't verify, I'll stop and say so. The story starts with a

1:03 sentence. In the repo's own workflow screenshots, step one is a person typing a plain language request into their editor. Something like open the app, sign in with the test account, confirm

1:14 the dashboard loads. That's it. No selectors, no test IDs, no page object. But typing in an editor doesn't move a phone. So, how does text in a window become a thumb on glass? Artemis ships a

1:26 native model context protocol server. One command, Artemis MCP install, and your assistant suddenly has new tools it didn't have before. MCP is an open standard for connecting AI applications

1:38 to external systems. Its own documentation compares it to a USBC port for AI apps, which is why one Arteimus server can plug the same physical phone into claude code, cursor, windsurf,

1:49 codecs, or anti-gravity. The port doesn't care which cable you bring and the assistant isn't improvising down there. It calls a small set of named tools. Mobile_run_task

2:01 mobile_manage task mobile_get device state mobile_insspect_trace mobile diagnose. Run a task. Check what the device is doing. Read the run back

2:14 afterwards. Diagnose it when the whole thing sulks. On the far end is a phone, a real one with USB debugging switched on, or an emulator if you'd rather not hand over your daily driver. Setup pulls

2:26 in the helpers. It needs ADB, Scarsby, FFmpeg, and Python through UV. Nothing exotic. You don't strictly need the editor either. There's a CLI and Artemis Doctor checks whether your setup is the

2:38 problem, which with USB debugging involved, it frequently is. And the primitives underneath are documented by Android itself. ADB exec out screen cap for a PNG. ADB shell screen record for

2:50 video. Logcat for the system log. Remember those three. The artifacts this thing eventually hands you look a great deal like what those commands produce. Now the loop itself. Step one, perceive.

3:02 Android exposes an accessibility hierarchy. A live tree of the elements currently on screen. Read that tree and you can know there is a button labeled sign in and roughly where on the glass

3:12 it sits. Except modern interfaces are unhelpful to that tree. Compose, flutter, a handrolled canvas. They paint pixels and the node tree comes back thin or empty. So, Artemis combines

3:24 accessibility hierarchies and OCR with visual models specifically for custom canvas compose and Flutter interfaces. Tree when there's a tree, eyes when there isn't. Step two, decide. The model

3:37 now holds the tree, the text OCR pulled out, and the picture. It picks one target. And in the pro profile, there's an extra beat before anything happens. The chosen target gets checked against

3:47 the live UI tree and the pixels before that individual action is dispatched. Step three, act. That decision becomes a real touch on a real screen, not a mocked call inside a test harness, but

3:59 the device your user holds, ending where their thumb also ends. How that touch is actually dispatched, the repo doesn't spell out. What it does document is that Flash has no ADB shell at all. So

4:10 there's more than one path down there. Then the phone does something rude. A permission dialogue over the top of your app, which is ordinary. Android's own UI automator exists to test an app from

4:21 outside its process with APIs for user apps and system apps. Watch for the permission dialogue. Click allow. That out of process model is long established on Android. Whether Artemis leans on

4:32 that particular library isn't documented, but its cross app runs are doing the same kind of job. Step four, and this is the one people skip. Verify. Did the screen actually become what we

4:42 expected? Pro runs checkpoint verification and produces a final report. And here is where the two documented profiles genuinely part ways. Flash is one model watching the live

4:52 screen, thinking, and acting. A fast token efficient reactive loop roughly 3 to 5 seconds per step with no graph orchestration. Pro is a planning and verification workflow built as a multi-

5:04 aent graph at roughly 15 to 40 seconds per step. Those timings are readme figures, not mine. They'll shift with your model, your network, your device, and flash's speed has a cost the repo

5:15 states plainly. No pre-execution safety net, no checkpoint verification, no final report, and no ADB shell. So, Flash is a scout. Quick, cheap, perfectly willing to tap the wrong thing

5:27 and never mention it. Pro is slower, and writes everything down. For a smoke test, you rerun 40 times a day. That's a genuine trade, not a ranking. Step five, collect. The run hands back screenshots,

5:39 video, and logcat. And now you know why those three feel familiar. They aren't exotic instrumentation. They're the same things ADB has always been able to give a developer by hand. The evidence isn't

5:50 a special feature. It's the debugging tools kept. And step four of the documented workflow is the report. What was attempted? What the screen did? Where it broke. That is the answer to

6:01 Friday's problem. Not done. But here's the tap. Here's the screen after it. Here's the exception underneath. If it did fail, mobile_insspect_race lets the assistant read its own runback

6:13 step by step. and mobile diagnose is for when the plumbing itself is broken. The cable, the authorization, the device that vanished midrun. Zoom out and the whole system is one sentence long.

6:25 Assistant MCP server perception decision device artifacts back. The official architecture diagram is the same loop we just walked only drawn with boxes instead of a phone on a desk. Now the

6:37 headline. Artemis reports a 99% plus completion rate on Android world described as Google researches benchmark spanning 20 plus apps and 100 plus multi-step tasks. That's a real

6:48 published claim. It is also a very specific claim about a very specific thing. So let's look at the thing. Android world is 116 programmatic tasks across 20 real world Android apps. Each

7:00 task ships its own initialization, success checking, and tearown logic. That's what makes it scorable at all. The environment already knows in code what success looks like and it runs on

7:11 one image. The setup specifies an emulator named Android World A DA a Pixel 6 Android 13 API 33. Same apps, same starting state every single run. That's a strength for a benchmark and

7:25 the exact reason a score on it isn't a score on your app. For scale, in the Android world paper, the author's own best agent completed 30.6% 6% of those tasks. So 99% plus is an enormous jump

7:38 on a fixed course with the gates already marked. Here's what I could not find. Android world results live in a maintained Google sheet and the project doesn't describe an independent

7:48 verification process. Third party mirrors list these entries as self-reported. I didn't rerun the benchmark and I found no independent reproduction of the 99% plus figure.

7:58 None of which makes the number false. It makes it a benchmark score. And Artemis' own phrasing, a completion rate on Android World is exactly right. The trouble only starts if you read it as a

8:08 promise about your app. What stays unresolved is the interesting part. Your production app, real hardware, a custom OEM skin, a logged account with somebody's actual data in it, a flow the

8:19 benchmark never contained. Nothing inside 116 fixed tasks tells you how any of that goes. And consider the blast radius. You're handing a model accessibility control of a phone plus in

8:31 pro and ADB shell. The repo documents that shell as pro only and the accessibility helper as local only. I haven't audited either one, so don't assume a sandbox. Assume a phone. The

8:43 readme names multimodel backends, Gemini, Claude, GPT40, Quenv fairly loosely. Which are required? What keys you'd need, what a long prorun costs, and tokens. That's a gap in the docs,

8:56 not a number I'm withholding. Budget for it before you wire this into CI and one open thread include it because leaving it out would be dishonest. A third party vendor has publicly raised attribution

9:07 and leaderboard emission complaints about Artemis. I haven't verified them. That's an unresolved dispute, not a finding, and I'm not going to present it as one. If you test one thing, change

9:17 one thing. Same phone, same single flow. Say sign in on a compos screen, 10 runs on Flash, 10 on Pro, everything else held still. My hypothesis, untested. Pro's pre-execution check catches wrong

9:31 target tabs that flash commits silently. Maybe it doesn't. That's the point of running it. So, the decision adopt it if your interface is Compose, Flutter, or canvas. That's the perception problem it

9:42 was specifically built for. And if what you want back is artifacts rather than reassurance, start on an emulator with a throwaway account on pro on exactly one flow. Wait if the only phone available

9:54 is your loggedin daily driver or if a check mark you can't inspect is honestly good enough for now. And the question I'd take into that first run when the report says passed, what did it compare?

10:04 The pixels, the tree, or its own memory of what it meant to do. Open the trace and find out.