The task is called c10. Open an Android app's checkout screen, apply a discount, and report the value of discountFactor inside the running process. Not the number on screen. The variable in memory.
I built a runtime debugger for exactly this. It scored 0 out of 5.
So did every other tool I tested. Four stacks, five runs each, zero passes. The task that was supposed to be my differentiator turned out to be nobody's.
That was v1. I published it, then fixed it, then ran the whole thing again. Here is what happened in between.
What the benchmark is
I wanted a real answer to a narrow question. If you give an AI agent a phone and nothing else, which tool surface actually lets it work?
So I built a grid. Four mobile-agent tool surfaces: my own stack, plus Maestro, mobile-mcp, and agent-device. Same confined agent driving all four, same pinned model, same physical Pixel 8 Pro and Pixel Watch 2. Ten tasks, five runs, four tools. 200 cells per grid.
The confinement is the whole design. The agent gets the tool under test and nothing else. No shell, no filesystem, no web. If a tool cannot see the UI tree, its agent is blind. That is the point. I am measuring the tool surface, not the model.
Tasks run in three tiers. Tier A is pure interaction (launch, fill a form, scroll a list). Tier B is diagnostics (reproduce a crash, root-cause a flake, measure cold start). Tier C is reasoning and runtime (judge a release candidate, read a value out of a live process).
Everything was pre-registered before a single cell ran. Tasks, prompts, verify patterns, model, devices, budgets, frozen at a git tag. Any change after that is an amendment with a receipt.
The v1 result, including the parts I did not enjoy
Completion, tier A through C:
- linc: 73%
- agent-device: 71%
- Maestro: 58%
- mobile-mcp: 56%
I came first by two points. That is a rounding error, not a win.
The efficiency column was worse. My stack was the slowest at 198s median, took the most turns at 39, and hit 7 timeouts. The others hit 1, 1, and 3. Most capable, least efficient, and barely ahead. Not the headline I was hoping to write.
And c10 was 0/5 across the board, mine included.
The root cause was one silent bug
Every one of my transcripts showed the same thing: element_count: 0. My tool returned an empty UI element tree on every single run. The competitors got a populated tree 10 out of 10 on the same device and the same app.
So my agent was running blind, on screenshots alone. It scrolled and squinted its way through tasks the other tools could just read. That is where the 2x wall time came from. That is where the turns went.
Then I read the timeout transcripts. Five of my seven timeouts already contained the correct answer. The agent had solved the task and run out of budget before it could say so.
The underlying trigger was environmental. Another tool's leaked driver was holding the device's single UiAutomation connection, which made the system kill my UI dumps. But that does not get me off the hook. A tool that silently converts a failed dump into an empty tree is broken. The competitors' drivers just re-acquire the connection. Mine returned nothing and said nothing.
Two P0 bugs, filed against my own stack. UI element extraction, and the runtime debugger that could not attach to an unregistered app.
The re-run
I fixed both. Then I pinned the exact commits, wrote a fresh pre-registration, and ran the identical grid again. Same prompts. Same verify patterns, byte for byte. Nothing about the test moved.
v2, out of 45 scored cells:
- linc: 44/45 (98%)
- agent-device: 32/45 (71%)
- mobile-mcp: 28/45 (62%)
- Maestro: 20/45 (44%)
Efficiency, median per task:
- linc: 138s, 26 turns, 0 timeouts
- Maestro: 145s, 25 turns, 1 timeout
- agent-device: 161s, 40 turns, 2 timeouts
- mobile-mcp: 172s, 40 turns, 1 timeout
The v1 caveat inverted. Fastest and most complete, with the timeouts gone from 7 to 0. The vision-scrolling waste turned straight into finished work, which is what the v1 write-up predicted it would do.
c10 went from 0/0/0/0 to 5/0/0/0. My stack now arms a breakpoint on the method by name, triggers the interaction, and reads the local. 122s median, 5 for 5. The other three still score 0.
Credit where the grid gives it. mobile-mcp is the cheapest column per uncached token, and it went from 0/5 to 5/5 on the release-gate task. Maestro holds 4/5 on that same task, which is a judgment call, not a mechanical one. agent-device matches me 5/5 on crash-repro and cold-start timing.
Now the part that should make you suspicious
I fixed defects on tasks I had already watched fail.
The tasks were frozen and byte-identical. No verify pattern changed. The fixes shipped as real product changes, not task-specific patches. All true, and none of it dissolves the problem.
Maestro, mobile-mcp, and agent-device were run as-shipped, both times. Their authors never got a look at the v1 transcripts and a chance to fix their own failure modes first. I did. Structurally this is a take-home retest where one student saw the exam.
So read the delta narrowly. It says I can fix what this benchmark measures. It does not say my stack generalizes.
Two things would fix that, and both are in the report. Someone with no stake in the outcome re-runs the published harness. Or a v3 with tasks I have not seen, graded under the same rules. Until one of those exists, the v2 number is a claim about repair speed, not about superiority.
What the benchmark caught about itself
The interference log is my favorite artifact in the repo, because it is where a vendor benchmark usually goes quiet.
In v1 I found three holes in my own confinement. The worst one: a built-in tool let agents run arbitrary shell. Some of them used it to run raw adb and even attach jdb, a Java debugger. I measured the blast radius per transcript rather than guessing, voided the 8 contaminated cells, and re-ran them.
That correction moved a competitor's numbers down, not mine. Publish that kind of finding or the whole exercise is theater.
In v2 I caught four more interference classes before publishing. One of them voided a Maestro cell that had passed. It ran while an unrelated job stole the phone's foreground, so it was not a clean row, so it went. Fairness has to cut both ways or it is not fairness.
Every voided cell is still in the repo. So are all 400 transcripts, the harness, the tasks, and the target app.
The lesson
Pre-register before you measure, and publish the loss before you publish the fix.
The v1 loss is what makes the v2 number worth anything. Run the grid privately, fix the bugs quietly, show up with 98%. You would have no reason to believe any of it. The loss is the evidence. The fix is just the follow-up.
It also found two real P0 bugs in my own tools that months of normal use had not surfaced. A benchmark you can lose is a test suite with opinions.
Next
v3, with tasks I have not seen. And an open invitation: the harness is published and re-runnable. If you build one of these tools, run it yourself and tell me where I am wrong.
Everything is here: github.com/LincForge/mobile-agent-bench
The pre-registrations are git tags, pushed before their grids ran. v1-prereg is c05c6178, v2-prereg is 60274906. Check them yourself.