Last week I pointed my device bench at three real Android devices for the first time: a Pixel 8 Pro, a Samsung Galaxy S10e, and a rugged Samsung Tab Active3. Same scorecard my agents had been running against emulators for weeks. The emulators had been telling me a cleaner story than the phones did.
The scorecard checks three things about an app. Does it survive in the background? How fast does it cold start? Do sensors keep streaming when the OS tries to put the device to sleep?
What passed, and it should bore you
The boring results first, straight from the run log:
- Background survival: pass on all three devices, 100% of runs.
- Cold start: 416 to 606 ms across the matrix. The rugged tablet was the fastest at 416 ms, which I did not predict.
Good. Boring. That is what a passing check should look like. The interesting part is everything the emulator runs had quietly gotten wrong.
Lie #1: "idle" while charging
Android has a test hook that forces a device into Doze, the deep idle state where the OS defers work to save battery. My bench used it. The verdicts came back clean for weeks.
Here is the thing the docs won't shout at you: force-idle silently no-ops while the device is charging. It does not error. It just doesn't idle.
A bench device lives on a charger. So every "how does the app behave when the phone sleeps" measurement I had was a measurement of a fully awake phone. The fix was mechanical: cut USB power first, verify the device actually reports idle, then measure. But nothing in the tooling tells you to do that. The emulator never surfaced it because an emulator's battery is a config value, not a physics problem.
Lie #2: the harness itself had only ever met emulators
One check verifies that motion sensors keep streaming through forced sleep. On real hardware it came back "inconclusive" on all three devices.
I read the raw logs. The sensors were fine. They streamed at full rate through the entire doze window. The verdict logic was the problem: its pass/fail heuristic keyed on behavior that only emulators exhibit. Real hardware also broke a rate assumption. The config expected sensor batches at 200 Hz, and the devices, asked for their fastest delivery mode, sent something else entirely.
The test harness is software too. And until that week, it had also only ever run against emulators. Both scoring gaps are now filed and getting fixed, but the lesson stands: a verdict is only as real as the hardware it was calibrated on.
Lie #3: one "doze" is not the other
My forced-idle window was 60 seconds. Samsung's real app-killing behavior operates on windows that are hours to days long. That is the "sleeping apps" regime One UI applies to apps you haven't opened in a while.
Those are different beasts. Passing a 60-second forced doze says almost nothing about surviving a Samsung weekend. There is a reason dontkillmyapp.com has a per-OEM page. The next tier of the bench is longer windows on the Samsung side of the rack.
One genuine surprise going the other way: One UI honored force-idle correctly once the device was unplugged. The OEM with the reputation for killing apps behaved exactly by the book in this test. The book just covers less than people think.
What this adds up to
CI green means your code compiled and your logic held on a Linux box. An emulator adds a rendered screen. Neither one has a battery, a charger, a thermal envelope, or an OEM's opinions about background work.
I knew that in the abstract. Now I have run logs where the same suite produces different truths on silicon versus simulation, three devices deep. The gap between "passes CI" and "works on the phone in your customer's hand" is not a slogan. It is measurable. I spent a week measuring a slice of it.
The bench runs against real hardware now, not just emulators. The phones keep disagreeing with the simulation. I'll keep writing down what they say.
Numbers in this post come from my own bench's merged run logs, 2026-07-10. Self-reported, happy to walk through them.