LINC Innovations

Blog

We ran the unseen-task rematch. Callstack tied us 34/35 on UI + logcat, and Sonnet 5 solved a debugger task with algebra.

Andrey Pogorelyy · October 14, 2026

In July I published a mobile-agent benchmark where my own stack scored 0/5 on the one task my debugger existed for. I fixed the two bugs that caused it, re-ran the identical 200-cell grid in v2, and scored 44/45.

Then I wrote down the reason you should not trust that v2 number. I had fixed defects on tasks I had already watched fail, while the three competing stacks ran as shipped. It was a take-home retest where one student saw the exam.

I promised a v3 rematch on unseen tasks against the latest competitor releases, scored once on a frozen build with zero post-fix re-runs. All 200 runs of v3 finished this morning on a physical Samsung Galaxy S24 and Galaxy Watch7 (600 runs total across v1, v2, and v3).

Here is what happened when nobody got a second take.

The headline: 34/35 tie on UI + logcat

First, what changed before the freeze (v3-prereg, git tag 2dc566d):

  • linc: frozen at the public PyPI release linc-nerve 0.1.0 (LincForge/nerve@5477863), paired with git-pinned crucible@cf04279 and spectra@c471dad.
  • agent-device (Callstack): upgraded from 0.12.2 to 0.21.27.
  • @mobilenext/mobile-mcp: upgraded from 0.2.2 to 1.0.10 (which adds mobile_get_device_logs and mobile_list_crashes).
  • maestro (mobile.dev): upgraded from 2.5.0 to 2.11.0.

We kept 4 byte-identical anchor tasks from v1/v2 (b5, b6, c9, c10) and added 6 unseen tasks (a5, a6, b9, b10, c11, plus the b11 Wear OS watch row) that had zero prior agent runs on any stack. Every run used the same confined claude-sonnet-5 agent with no shell, no filesystem search, and no web access.

Seven of the nine phone tasks test UI automation and logcat triage (a5, a6, b5, b6, b9, b10, c9 - 35 runs per tool). The other two (c10 and c11) require inspecting local variables inside a running JVM stack frame.

On the 7-task UI + logcat phone subset (35 runs per tool):

  • linc: 34/35 (97.1%)
  • agent-device 0.21.27: 34/35 (97.1%)
  • mobile-mcp 1.0.10: 18/35 (51.4%)
  • maestro 2.11.0: 14/35 (40.0%)

Callstack's agent-device 0.21.27 matched my stack run for run. We beat them by one run on b5 (5/5 vs 4/5). They beat us by one run on b6 (5/5 vs 4/5). We tied at 5/5 on the other five tasks (a5, a6, b9, b10, c9), including turning their v2 c9 score from 0/5 into 5/5.

Per-task efficiency on those 7 shared tasks split down the middle:

  • Wall time favored agent-device 0.21.27 on 5 of the 7 tasks (64.1s vs 80.6s pooled median, 20 vs 21 median turns). linc was faster on b5 (304.7s vs 322.5s) and b6 (232.1s vs 470.6s).
  • Token cost favored linc on 6 of the 7 tasks (2.60M vs 3.95M total uncached tokens, $34.22 vs $47.93 total API spend, or $1.01 vs $1.41 per passing run).

And if you narrow further to the 3 pure UI phone tasks (a5, a6, c9 - 15 runs per tool, no logcat or debugger required), linc, agent-device, and mobile-mcp all went 15/15, and maestro 2.11.0 went 13/15 (14/15 if you count maestro/a6/run-2, which found the right phone token and only failed the regex because it mentioned the watch token by way of contrast).

Where the 44/45 vs 34/45 gap comes from

On the full 9-task phone grid (45 runs per tool), the totals are:

  • linc: 44/45 (97.8%)
  • agent-device 0.21.27: 34/45 (75.6%)
  • mobile-mcp 1.0.10: 20/45 (44.4%)
  • maestro 2.11.0: 14/45 (31.1%)

On the b11 Wear OS watch row (Galaxy Watch7, reported separately and never averaged into the phone total): linc (50.8s median), maestro (61.0s), and agent-device (62.5s) all went 5/5, and mobile-mcp went 4/5.

Every single run of the 10-run gap between linc (44/45) and agent-device (34/45) comes from c10 and c11. That is a capability boundary, not a UI automation gap. linc ships a JDWP runtime debugger (spectra) that can pause a method by name and read local variables out of the JVM frame. The other three stacks do not.

How Sonnet 5 solved a debugger task without a debugger

The best transcript in all 200 runs is mobile-mcp/c10/run-4.

Task c10 asks for the exact runtime float discountFactor (0.755) inside CheckoutViewModel.applyDiscount() when item quantity is 7 and discount code BENCH15 is applied. The screen only renders the rounded integer dollar total ($50).

With linc, the agent attaches spectra_method_breakpoint to CheckoutViewModel.applyDiscount, taps Apply Discount, and reads discountFactor = "0.755" from the stack frame in 100.2s median and 3,703 output tokens (5/5).

mobile-mcp has no debugger. In v1 and v2 it scored 0/5 on c10.

In v3 (mobile-mcp/c10/run-4 and run-5), claude-sonnet-5 refused to give up. It tapped Apply Discount at every quantity q = 1..7 and recorded all seven rounded integer totals from the UI: (9, 17, 25, 32, 39, 44, 50). Then it wrote out the seven rounding intervals 9.37 * q * f(q) in [total - 0.5, total + 0.5), solved the system of inequalities for the linear step f(q) = 1 - 0.035q, and derived f(7) = 0.755.

It took 579.1s and 34,512 output tokens in run-4 (855.0s and 46,356 output tokens in run-5 - roughly 10x the output tokens and 6x to 8.5x the wall time of reading the frame directly). And it passed 2/5.

Task c11 (c11-caught-exception-state, unseen in v3) closes that algebraic path by design. Redeeming promo code BENCH26 throws a PromoExpiredException whose catch block shows a static "Promo code expired" banner for every input while holding auditToken (PROMO-BENCH26-7956) and rebateCents (1355) in local variables that have no UI-observable channel. On c11, linc went 5/5 (119.5s median) and the other three stacks went 0/15 combined.

One logcat flag cost mobile-mcp 10 timeouts

@mobilenext/mobile-mcp 1.0.10 shipped two new observability tools: mobile_get_device_logs and mobile_list_crashes. Those tools unlocked b6/run-1 (1/5, up from 0/5 in v2) and solved b5 in 198.1s once crash buffers were clean.

Yet on the two unseen logcat tasks (b9 silent feature-flag gate and b10 main-thread stall audit), mobile-mcp 1.0.10 hit the 900-second timeout on all 10 runs (0/10).

The clue showed up inside mobile-mcp's own unfiltered log output in results_v3/mobile-mcp/b10/run-1/transcript.jsonl:

adbd service requested 'shell,v2,TERM=dumb:export ANDROID_LOG_TAGS=''; exec logcat '-v' 'threadtime,year' '-T' '1''

Instead of dumping the historical ring buffer and exiting (logcat -d), logcat -T 1 emits only the last 1 line of historical logcat and keeps streaming future lines until limit matches arrive. When an agent taps a button (Run Health Check in b9 or Compact Local DB in b10) and then calls mobile_get_device_logs with a message filter (filter: ["message=blocked_ms"]), the synchronous Log.w inside onClick has already fired before the logcat stream opens. With no future matching lines arriving and no per-call timeout, the tool call hangs until the 900s harness kill.

Both linc-nerve (nerve_logcat) and agent-device (logs) read the historical ring buffer (logcat -d). Both went 10/10 across b9 and b10.

The bugs v3 caught in our own frozen stack

Because we froze the stack at v3-prereg and scored once without re-running, every defect v3 surfaced in linc stays in the published numbers:

  1. The b6/run-4 miss (our sole failure out of 50 phone + watch runs): Across 16 calls in v3, agents passed the Android package name (tag='dev.lincforge.benchtarget') or a message keyword to nerve_logcat's tag parameter. Because adb logcat -s <tag> is case-sensitive and tag-only, those calls returned 0 lines. In b6/run-4, the agent saw 0 lines, abandoned logcat, and guessed a random startup draw.
  2. Ten UI-thread ANR dialogs on c10 and c11: Even though linc went 10/10 on c10 and c11, arming spectra_method_breakpoint on a UI-triggered ViewModel method paused the app's main thread inside onClick while nerve_tap was still waiting for adb shell input tap to return. Every single c10 and c11 run hit a 10-second nerve_tap read timeout and triggered Android's 5-second "Wait" ANR dialog before spectra_method_locals auto-resumed the process.

Both fixes are now queued for the next release. Neither touches the v3 scoreboard.

Check the receipts

A vendor benchmark where competitors never improve is marketing copy. Publishing the 34/35 UI + logcat tie with Callstack's agent-device 0.21.27, their 5-of-7 wall-time lead, and our own 10 ANR dialogs is what makes the rest of the table worth reading.