Your engineers adopted AI coding agents. Your CI is green. Now answer these, tonight, from records, not from memory:
- Who authorized the last merge an agent made?
- Is the code the agents write making main better or worse - measured, not felt?
- What does an agent-merged change cost you, per change?
- If an agent goes wrong at 2am, which switch do you pull, and who is allowed to pull it?
Green CI answers none of these. CI proves the tests pass. It does not prove the change was authorized, in-scope, reviewed by something other than its author, reversible, or affordable. That's a different layer: the control plane between "agents produce PRs" and "changes reach main."
2026 made this concrete. In May, attackers compromised the TanStack npm packages (12.7M weekly downloads on react-router alone). They poisoned a CI cache via pull_request_target and scraped an OIDC token out of runner memory. The 84 malicious package versions they published carried valid SLSA provenance attestations (TanStack's own postmortem). The lesson for this rubric: cryptographic provenance certifies where an artifact was built, not whether the change behind it was authorized. Authorization is a control-plane property. It's what the first four dimensions below grade.
I run a fleet of ~25 production repos where AI agents propose, review, and merge changes daily, governed by exactly this kind of control plane. This rubric is the one I use to grade it. At the end I'll show you my own scorecard, including the ugly rows.
The maturity scale
Grade each dimension separately. Be harsh. The scale is designed so L3 is hard.
| Level | Name | Meaning |
|---|---|---|
| L0 | Absent | No mechanism exists. |
| L1 | Ad-hoc | Humans do it by hand, when they remember. |
| L2 | Defined | Documented and partially wired, or wired but its integrity is unproven. |
| L3 | Enforced | Automated, fail-closed, agents cannot bypass it, and it has fired for real at least once. |
| L4 | Proven | Enforced AND longitudinally measured: 30+ days of records, metrics, rehearsed recovery. |
The trap to avoid: a policy document is L1, not L3. A prompt telling the agent to be careful is L0. A prompt is a request. A required check is a control.
The 10 dimensions
1. Identity & credential separation. Every agent acts as a distinct non-human identity with its own credentials, never as a human's account. Test: pick an agent-authored commit. Can you prove from the platform's own records which agent made it, and that no human credential could have been used instead?
2. Change-approval gates. Nothing reaches main without passing gates the author cannot waive: branch protection, required checks, no admin-bypass merges. Test: can the agent that wrote a PR also approve it? If yes, you have a rubber stamp, not a gate.
3. Protected-surface controls. The governance machinery itself (the gates, the checks, the permission config) is a protected surface that agents cannot modify. Test: what happens if an agent opens a PR against the gate's own code? (This is the dimension almost everyone scores L0 on. It happened to me, and the gate held. That story gets its own post.)
4. Human override with audit trail. When a human overrides a gate, the override is explicit, attributable, expiring, and replay-resistant (e.g. a dated approval token in the record, not a quiet admin-merge). Test: find your last override. Who, when, why, and could it be replayed next week?
5. Verification & quality gates. Automated correctness bars that agent code must clear, same as human code or stricter: tests, lint/complexity ratchets, mutation checks, device/end-to-end verification. Test: name the bar an agent PR must clear that a confident-but-wrong PR would actually fail. For the L4 longitudinal proof, track your Rework Ratio: the share of commits in the last 30 days that revert, amend, or refactor a commit also under 30 days old. It's the 2025 DORA Report's fifth metric, built for exactly this. Under ~10% is a healthy human baseline. Unmanaged agent fleets have been measured above 40%. It's computable from git alone, no vendor required.
6. Observability you can trust. You can see fleet state (what's running, what merged, what's blocked) AND the signals carry freshness contracts. A dashboard that can silently go stale is a liability: the worst pathology is a metric that reads "healthy" for weeks after the thing behind it died. Test: for each green light, when did it last prove it can turn red?
7. Rollback. You can revert what the agents did (not just one PR, but "undo the last N agent merges") and you have drilled it. Test: when did you last rehearse a rollback? Undrilled rollback is L1 by definition.
8. Cost governance. Per-agent spend is metered, budgeted, and alarmed. Agents that retry, loop, or research autonomously can burn real money silently. On subscription plans the failure is double-sided: quota you burn too fast, and quota you pay for but never use. Meter both. Test: what did your agents spend yesterday? If the answer requires opening a billing console and guessing, this is L0-L1.
9. Blast-radius limits. Every unit of agent work carries a declared scope (which paths, which repo, one claim at a time), enforced by checks, so a misbehaving agent damages a bounded area. Test: what stops one agent task from touching 40 files across 3 repos?
10. Learning & incident loop. Failures become captured lessons that change the system (postmortems, compiled rules, gate adjustments) rather than tribal memory. Test: show the artifact produced by your last agent incident.
How to score it
Grade each dimension L0-L4 with evidence, not intentions ("we're about to add it" scores as absent today). Then read it as a profile, not an average: one L0 in cost governance can hurt you faster than six L3s can help. Most teams I talk to that have adopted AI coding grade L0-L1 across the board. That's not an insult. It's twelve months of normal adoption speed outrunning controls.
My own scorecard (2026-07-01)
Honesty is the point of the instrument, so here's mine. The fleet: ~25 repos where three autonomous agent roles operate daily (an observer/memory node, a triage/review node, and a dispatcher/executor node).
| Dimension | Grade | One-line evidence |
|---|---|---|
| Identity & credential separation | L3 | Distinct GitHub App identity per agent node. Launch wrappers strip human credentials and fail fast if the bot key is missing |
| Change-approval gates | L3 | Branch protection fleet-wide, no admin merges. A fail-closed AI content-review gate with a bounded retry loop that escalates to a human |
| Protected-surface controls | L3 | Pattern-protected governance modules + scope checks. Live proof: the gate blocked an AI edit to its own module |
| Human override w/ audit trail | L3 | Dated, expiring approval tokens tied to a separate human identity. Exercised once for real, not yet routine |
| Verification & quality gates | L3 | Complexity ratchets, per-PR mutation testing, on-device verification gates. Efficacy (post-merge escape rate) not yet measured |
| Observability | L2 | Live fleet dashboard, but I've documented a stale-green incident (a metric read "achieved" for 69 days while the thing behind it was dead). Integrity work in progress |
| Rollback | L1 | Per-PR revert, human-initiated, never drilled. Named it, scheduled it |
| Cost governance | L2 (updated 2026-07-23, was L0 on 07-01) | Per-agent spend meter now built and merged: ledger, aggregation, threshold signal, CLI report. Dashboard tile and automated alert not yet wired, so by this rubric's own definition it's L2, wired but unproven. The ~$277 silent-burn scar that motivated it stays in the story |
| Blast-radius limits | L3 | Declared scope on every claim, path-allowlist required checks, isolated worktrees, one claim per tick |
| Learning / incident loop | L3 | Every incident -> captured lesson -> compiled operating rules the agents load |
Aggregate: solid L3 with one crater (rollback L1) and two L2s climbing (cost, where the meter shipped after the 07-01 grade, and observability, where integrity work is in progress). The gap between my L3s and L4 is a single ingredient: longitudinal proof. 30-day records, measured escape rates, rehearsed recovery.
What to do with your score
If you graded mostly L0-L1: you're normal, and you're exposed exactly where an auditor, a security review, or a bad Tuesday night will look first. The order I'd fix things: identity separation -> approval gates -> cost metering -> protected surfaces. The first three are days of work each with GitHub-native primitives (Apps, branch protection, required checks). None require buying anything.
I also run this as a fixed-price, two-week AI-SDLC Governance Audit: I grade your setup on these 10 dimensions with evidence, and hand you a scorecard plus a remediation roadmap in priority order. If that's useful, my inbox is open. If you just want to argue about the rubric, that's useful too.