How to Measure GitHub Copilot ROI (Without Trusting Acceptance Rate)

Start With the Right Question.

The question a CFO renews on is not "are engineers using GitHub Copilot?" It is "is this seat spend producing attributable value, measured against how we shipped before?" Those are different questions, and most measurement stops at the first one because it is the easy one.

The method below is deliberately tool-agnostic. Whether your team is on GitHub Copilot, Cursor, Claude Code, or Windsurf, the metrics that prove or disprove ROI are identical. The only thing that changes per tool is where the usage telemetry lives. For GitHub Copilot, that is the GitHub Copilot admin console plus your organization's GitHub Insights. The tool is the search term; the measurement is universal.

One rule holds throughout: anchor every claim to independent research, not to the vendor's marketing. A 10x productivity claim is a slide, not a measurement. The strongest evidence, an experienced-developer randomized trial from METR in July 2025, found developers were 19% slower with early-2025 AI while believing they were 20% faster. Start from that humility, not from the pitch deck.

The GitHub Copilot Admin Console: Useful, But a Weak Signal.

The GitHub Copilot admin console gives you two things quickly: seats assigned versus seats active, and acceptance rate (the share of suggestions developers keep). Pull both. But be clear about what each is worth.

Acceptance rate is a usage-depth signal, not a value signal. A developer can accept a suggestion, then spend twenty minutes verifying and rewriting it. Acceptance counts the keystroke, not the outcome. Treat acceptance rate as evidence that the tool is being touched, never as evidence that it is paying off. Any ROI story built on acceptance rate alone is measuring the wrong end of the pipeline.

• +Seats assigned vs active (30-day): the input to your idle-seat math.
• +Acceptance rate: usage depth only. Do not present it as ROI.
• +Cross-reference with your identity provider (Okta, Entra) for licensed-but-dormant users the console alone may miss.

Idle Seats: Your First Hard-Dollar Number.

Before any productivity analysis, compare seats you pay for against seats active in the last 30 days. The gap is pure waste, and it is often the fastest number to produce and the easiest to act on.

Multiply idle seats by your per-seat price and you have a concrete, defensible figure: dollars spent on licenses nobody is using. For many teams this single line item is large enough to fund right-sizing the contract at the next renewal, independent of any deeper productivity question. It is also the least controversial finding you will present, which makes it a good place to start building credibility.

The Metrics That Actually Measure Value.

Real ROI shows up downstream of the keystroke, in delivery. Measure these against a pre-AI baseline (the same teams, a comparable window before rollout), pulled from GitHub Insights or your engineering-intelligence tool:

• +Pull-request throughput: merged PRs per developer per week, now versus your pre-GitHub Copilot baseline. This is the headline "is it working" number.
• +Pull-request size trend: rising PR size means larger, harder-to-review batches. Growing size is a warning, not a win.
• +Review and merge time: the most common place value leaks. As AI increases code supply, review becomes the bottleneck. Even Faros AI, a measurement vendor whose business is making AI adoption look good, measured review time rising 91% in one 10,000-developer study (vendor telemetry, cited here precisely because it cuts against the vendor's interest).
• +Change-failure and revert rate: the quality check. DORA's 2024-to-2025 research found throughput can improve while stability still degrades. Speed that ships more incidents is not ROI.
• +Spend vs value by team: seats times unit price (the denominator) set against the throughput and quality deltas above (the numerator). This per-team cross-tab is the table a finance leader actually renews or cuts on.

The Perception Gap Is Part of the Finding.

Ask your team how much faster they feel with GitHub Copilot, then ask how much of that speedup survives review and rework. The gap between the two is one of the most useful things you can measure, and it is usually large.

This is not a knock on your engineers. The METR trial found developers felt +20% while measuring -19%, a roughly 40-point swing between perception and reality. Perceived productivity is an unreliable instrument because the verification and integration cost of AI-generated code is invisible in the moment and expensive in aggregate. When you present measured deltas next to felt ones, the contrast is disarming and it reframes the whole conversation away from vibes and toward numbers.

If You Cannot State a Baseline, That Is the Finding.

Many teams run this exercise and discover they cannot answer basic questions from memory: What was our review time before GitHub Copilot? Our change-failure rate? Our PR throughput per developer? If those numbers do not exist, no honest ROI verdict is possible yet, for any tool.

That is not a failed measurement. It is the most important result you will get. The recommendation writes itself: instrument the handful of metrics above before scaling spend further, then re-measure against a real baseline in a quarter. A team that cannot state its baseline is not in a position to renew or cut with confidence, and naming that gap plainly is more valuable than any manufactured number.

From there, every engagement lands in exactly one of four honest places: right-size and cut idle seats; keep and consolidate if adoption and lift are both real; keep the tool but fix the downstream system if throughput is flat while review time balloons; or cut honestly if there is no adoption and no path. Independent evidence points to that third case being the most common, because AI amplifies whatever delivery system it lands in. Redesigning review and workflow, not swapping tools, is the lever McKinsey found most correlated with impact.

Straight answers

Questions leaders ask.