Sample Diagnostic Report
Flaky-Suite & Device-Test Diagnostic: run against our own fleet
This is the Week-1 half of the real deliverable. I produced it by running the full method against my own engineering org: 51 repositories at the time of the run, 90 days of CI history, over eleven thousand CI runs. I won’t sell a diagnostic I haven’t run on myself.
Names and links are my own fleet’s. Everything else is exactly what a client report looks like.
A full engagement adds Week 2: the top offenders reproduced and root-caused on physical devices, with screen video, logs, UI-tree diffs, and debugger evidence per finding (format shown in §5).
1. Executive summary
Over 90 days (2026-04-02 → 2026-07-01), this org’s CI executed 11,338 workflow runs, of which 900 failed (7.9%, or 7.0% on PR-gating runs). Those failures decompose into 967 distinct failure events, which collapse into 139 root-cause clusters. The top 10 clusters account for 62% of all failure events.
The headline: half of all failed runs (49.6%) were transient. The same pipeline went green within 24 hours with no relevant change. That is the number that erodes trust in red CI. And the biggest sources weren’t test bugs at all:
- 17% of failure events were one pipeline-automation defect: a nightly status-publishing job pushing to a branch that branch protection rejects (GH006), failing 149 times across 13 repos in 4 weeks. One structural fix (publish to a sidecar branch) ended it. The data shows the cluster dying on the exact day the fix landed (2026-06-26).
- 13% was one CI-config defect: multi-repo builds failing dependency resolution (Distribution not found) whenever a sibling checkout was missing: 130 events, 13 repos, 5 workflows, 23 active days.
- Genuine test/build regressions were 21% of events - real bugs, but they were drowning in the noise above. Example: one shared-module compile break propagated to 2 downstream apps within hours (15 events in 2 days) - visible immediately once clustered, invisible in a sea of unrelated red.
Fixing the top 5 clusters (~3 engineering-days total, per-item estimates in §4) would have removed ~42% of all failure events in this window.
2. What was analyzed & method
Inputs: GitHub Actions run history for 51 repositories (Kotlin Multiplatform mobile apps, Python services/tooling), 2026-04-02 → 2026-07-01: 11,338 runs, 900 failures, job-level metadata + failed-step logs for every failure. Coverage caveats (stated, not hidden): ~4% of failed runs had already-expired logs (provider retention), so they were classified from job/step metadata only. One high-volume repo’s history hit the provider’s 1,000-run listing cap, so its window is effectively ~75 days.
Method: every failure event → canonical record (test id, exception type, normalized message, failing frame/step, branch, trigger, timestamps) → clustered by failure signature(deliberately ignoring which test surfaced it, so one root cause failing many tests counts once) → classified by evidence rules → ranked by event count and blast radius. In a client engagement, clusters that resist classification from history alone go to the Week-2 bench.
Confidence labels: Confirmed (mechanism verified) · Strong (history evidence consistent) · Hypothesis (best available explanation, verification path stated).
3. Where the failures actually came from
| Class | Events | % | What it looked like here |
|---|---|---|---|
| Quality-gate rejections | 224 | 23% | Intentional hard gates (complexity ratchet, lint, isolation checks) doing their job - policy friction, not flakiness. Belongs in a different conversation than “flaky tests” |
| Environment / infra | 221 | 23% | Dependency-resolution failures in multi-repo builds, expired logs, emulator/device hiccups, checkout auth |
| True regressions (test+build) | 206 | 21% | Real bugs, incl. a shared-module compile break hitting 2 downstream apps |
| Pipeline automation | 169 | 17% | One misconfigured automation push pattern (GH006), plus token-supply defects |
| Needs deeper evidence | 136 | 14% | Job-level failures whose logs lack test detail - in an engagement these are Week-2 bench candidates (honest bucket, not hand-waved) |
| Test-authoring debt | 11 | 1% | Race-pattern assertion failures recurring across days |
Two observations a team can act on without me. 1) When nearly a quarter of “red CI” is quality gates working as designed, the fix is gate placement/expectations, not test repair. 2) The transient rate (49.6%) versus the true-regression share (21%) quantifies exactly how much signal the team is losing to noise, and why “just rerun it” became culture.
4. Ranked fix plan (top 5 of 139 clusters)
| # | Cluster | Class | Events | Repos | Root cause | Confidence | Effort |
|---|---|---|---|---|---|---|---|
| 1 | Protected-branch push (GH006) | pipeline-automation | 149 | 13 | Nightly status-publisher pushes to protected main. Rejected every night, org-wide | Confirmed - fixed 06-26 (publish to sidecar branch). Cluster’s last-seen = fix date | S (0.5d) |
| 2 | Distribution not found (path deps) | environment/infra | 130 | 13 | Multi-repo editable path-dependencies require sibling checkouts the workflow didn’t always provide | Confirmed - generator now emits multi-repo checkout | S–M (1d) |
| 3 | Complexity-ratchet gate failures | quality-gate | 91 | 10 | Hard quality gate rejecting new uncovered complexity - working as designed. Recurring failures = teams learning the gate | Confirmed (by design) | policy, not code |
| 4 | Shared-module compile break | true-regression(build) | 15 | 2 | One XrayReportingModule.kt API change broke downstream composite builds | Strong - resolved within 2 days in-window | M (1–2d incl. contract test) |
| 5 | Token-supply failures | pipeline-automation | 20 | 7 | Workflow steps invoked without required token input after a workflow refactor | Strong | S (0.5d) |
(A client report continues to ~10 items and includes the full 139-cluster table as a CSV appendix, plus a quarantine list with re-enable criteria.)
5. What Week 2 adds (format sample)
For each agreed top offender, the engagement delivers a bench evidence page:
Cluster #N - [name]
Reproduction: client debug build on physical [device/OS]; scripted repro; reproduced m/n iterations (or: not reproduced in n iterations → stated hypothesis + where the evidence points).
Root cause (Confirmed): mechanism narrative written for the engineer who will fix it.
Evidence: screen recording with the failure moment timestamped · device log excerpt (the 5 lines that matter) · UI-tree diff at the failure step · debugger findings (breakpoint, stack, the variables that tell the story).
Recommended fix + effort + verification recipe (e.g. a 30-iteration repro loop, included).
This sample stops at Week 1 because it’s my own fleet analyzed for calibration. On an engagement, Week 2 runs on my physical device bench (Android, iOS, Wear OS, watchOS hardware) with runtime debugging attached to your build.
6. Keeping it this way
Regardless of engagement: 1) track the transient rate weekly (it’s one query once failures are signatured). 2) hold a quarantine policy with re-enable criteria. 3) re-run the taxonomy quarterly - this report’s signature table makes that mechanical.
Prepared with the same pipeline used on client engagements. Client data never leaves an encrypted per-engagement store and is deleted 30 days post-delivery.