Sixty-seven percent of every CI failure event in nextcloud/android's last 90 days traces to one line of test code. Not a flaky line. A line that cannot pass on any Android version that exists.
I found it by running my full test-suite diagnostic against an open-source app that never asked for one. Here is what that looked like, including the part where I got my own numbers wrong.
Why audit a stranger's CI
I sell a diagnostic engagement for mobile test suites. Before asking anyone to pay for it, I wanted proof the method survives contact with a codebase I had never seen, on data I don't control.
So I ran the whole engagement end to end against nextcloud/android: a file-sync app with 1M+ downloads on Google Play, an active maintainer team, and a fully public CI history. Public data only. Their GitHub Actions history, their issue tracker, and a debug build compiled from their public source.
One rule framed the exercise: the app is a subject, not a defendant. The maintainers received the full report before anyone else saw a word of it. This post exists because they responded well to that.
What 14,449 runs actually contain
The 90-day window (2026-04-10 to 2026-07-09) held 14,449 workflow runs. 3,799 failed. That is 26.3% overall, and 33.0% on pull-request runs.
Sit with that second number. A contributor opening a PR against this project sees a red check one time in three, as the default experience.
I decomposed the 3,799 failed runs into 16,539 individual failure events, then clustered them by failure signature: exception type, failing frame, normalized message. Deliberately ignoring which test surfaced the error, so one root cause failing 400 tests counts as one problem, not 400.
The 16,539 events collapsed into 252 clusters. The top 10 clusters account for 83% of all events. The single biggest accounts for 67%.
That concentration is the finding that matters. It is not 252 problems. It is ten, and mostly one.
The one line
Eight test classes grant two permissions through a raw GrantPermissionRule: WRITE_EXTERNAL_STORAGE and POST_NOTIFICATIONS, together.
POST_NOTIFICATIONS does not exist below API 33. The app's own manifest caps WRITE_EXTERNAL_STORAGE at API 29. So on API 32 and below, the first grant throws. On API 30 and above, the second one does. There is no Android version on which both succeed.
The project's CI emulators run API 28 and 29. Every executed run of both instrumentation lanes fails at test setup, before any test body runs. Suite-level retries and per-test retry rules blur the pattern, so for months it has read as "flaky CI."
I did not trust the history alone. I looped the affected classes on my own bench: an API 28 emulator, an API 34 emulator, and a physical Android 12 tablet. 31 iterations, zero passes. Deterministic on every level I could get hardware for.
The repair is not heroic. The repo already contains the correct SDK-conditional grant pattern in another rule. Extending it and replacing eight call sites is an hours-scale change, and the maintainers had already started on the permission problem in PR #17268 before my report arrived.
A second config-level find rode along: the screenshot-test lane had executed zero tests since mid-June. 598 runs failed at Gradle task selection, each one booting an emulator first. Their own #17143 tracked the cause. I added the measured impact.
Those two mechanical fixes remove roughly half of all red CI events in the window.
The finding that wasn't a test problem
My bench loops kept ending in process death. 17 iterations across three test classes on two API levels. Five captured stack traces, all identical: an uncaught AccountNotFoundException thrown from a background upload-retry coroutine whose scope has no CoroutineExceptionHandler. Nothing catches it. The app process dies.
Two of the three crashing classes contain no permission rule at all, so this is an independent app-code fault, not a side effect of the test defect above.
The part that makes it a real product bug: the retry fires passively. App start, a connectivity change, plugging in a charger. No user interaction required. In CI history it hides inside generic "Process crashed" noise.
Filed upstream as #17337, with stacks and a repro loop that currently kills the process 11 times out of 15.
I got two numbers wrong first
Before anything left my desk, I re-derived every number in the report from the raw evidence files. Two claims failed that check.
My first draft said the crash hit 2 of 25 iterations of one test class. The raw ledgers said 17 iterations across three classes. I had understated my own strongest finding by roughly 8x, because my harness counted "crashed" separately from "failed" and my summary silently dropped the crashed column.
Worse: my production story for the crash was wrong. "The user removed their account" is guarded in their code. With zero accounts, the crashing call is skipped entirely. The real precondition is narrower, an account row that exists while its server-URL field reads back null, and I only found that by reading the decompiled library their build pins.
Both corrections shipped in the report as a visible revision note. That earned a standing rule: nothing goes upstream until every number is re-derived from raw evidence. The first draft was wrong in exactly the finding a maintainer would check hardest.
The strongest objection
A fair skeptic would say: this is an open-source project, maintained by a small team alongside a huge product surface, with CI anyone can inspect. Finding noise there proves nothing about a funded product team's suite.
Partly agreed. The failure rate is not the transferable part. The structure is.
Every failure-heavy suite I have measured, including my own fleet's, shows the same shape: a few deterministic defects wearing a flakiness costume. At 33% red, "just rerun it" becomes culture, nobody reads the noise, and the two things in this window that deserved eyes (a true regression and the process-killing crash) were invisible inside it. The concentration means the fix list is short. The noise means nobody can see the list without clustering the failures first.
And the maintainers here are not a cautionary tale. The mobile team lead replied within three days, a permission fix was already in flight, and they asked for the issues to be filed directly so they could act on them. Issues went in the way they preferred. That is the response you want pointed at your own repos.
What this means for your suite
If your CI is red one run in three, the base-rate assumption should not be "we have hundreds of flaky tests." It should be "a handful of deterministic defects are mass-failing runs, and retries are laundering them into noise." Finding them is mechanical once failures are clustered by signature instead of by test name.
That diagnostic is now something I run as a fixed-scope engagement. The method above is the method: history, clustering, bench reproduction on real devices, and every number re-derived before it ships. Details at lincinnovations.com/audit.
The nextcloud maintainers got the whole thing for free, first, with no ask attached. That was the point of the rehearsal. The next one runs on somebody's suite that isn't public.
Numbers in this post come from the public GitHub Actions history of nextcloud/android (2026-04-10 to 2026-07-09) and my own bench iteration ledgers. The full report went to the maintainers on 2026-07-11. Public receipts: #17337, PR #17268, #17143.