Cycle time: speed that survives to production
Cycle time is how long work takes to move from "we should build this" to "it is running in production and doing its job." It is the closest thing to a real productivity signal, because it captures the entire path, not one heroic step in the middle.
That distinction matters more with AI than without it. A model can help a developer produce a first draft of a change in a fraction of the usual time. If that draft then spends three extra days in review because it is subtly wrong, or bounces back from QA twice, the cycle time did not improve — it just moved the slow part downstream where it is harder to see. We measure the full loop precisely to catch this. Faster typing that produces slower shipping is not a win; it is a bottleneck relocation.
The right framing is throughput of finished, working software over time, compared honestly against the baseline before AI tools arrived. If cycle time drops and stays dropped, something real happened. If only the drafting step got faster while the overall loop held steady or slowed, you have learned something equally valuable, just less flattering.
Quality: the gain that cannot come at quality's expense
Speed is only worth having if the software works, so quality is measured alongside cycle time, never instead of it. The signals are the ordinary ones, and their ordinariness is the point: defect and bug rates, production incidents, test coverage and pass rates, the amount of rework a change requires after it ships. Increasingly, for AI features specifically, eval scores belong here too — the evaluation suites that tell you whether the model layer is still doing its job.
The number we watch most carefully is the trend, not the snapshot. If cycle time is falling while defect rates and incidents hold steady or improve, AI is delivering genuine value and you should keep going. If cycle time is falling while quality quietly degrades, you are not moving faster — you are borrowing speed against a debt that comes due later, usually at the worst possible moment. Reading those two metrics together is the whole discipline. Neither one alone will tell you the truth.
Risk: the debt that hides
Risk is the slipperiest of the four because it accumulates silently and shows up late. AI-assisted development can generate a lot of code quickly, and volume has a way of concealing accumulating problems: architectural drift, inconsistent patterns, security gaps, brittle spots, and the general erosion of a codebase that no human fully understands anymore because it grew faster than anyone's ability to hold it in their head.
We treat risk as a first-class measure rather than a vibe. Security findings over time, architectural consistency, the share of the codebase under meaningful test coverage, and how well the team actually understands the systems they own — all of it trends in a direction, and the direction is what you manage. A codebase that grows faster than the team's comprehension of it is a liability dressed up as productivity, and no amount of cycle-time improvement makes that trade a good one. The whole reason this series has insisted on evals, observability, and human accountability is that they are how you keep AI-driven speed from turning into AI-driven risk.
Cost: the whole bill, honestly
Cost is where the analysis either becomes real or becomes marketing. The obvious line item is tooling and model usage, and it is worth watching — model calls carry a marginal cost on every invocation, and at production volume that adds up in ways prototypes never reveal. Guidance on running and scoring model behavior, such as OpenAI's evaluation best practices and its agent evals material, is useful here not just for quality but for understanding what you are actually paying for per run.
But the honest cost picture is bigger than the invoice. It includes the human time spent reviewing and correcting generated code, the effort to maintain evals and observability, and the eventual cost of any risk that materializes into an incident or a painful refactor. We have seen teams celebrate a lower tooling bill while quietly spending more senior-engineer hours cleaning up than they saved in drafting. That is not a cost reduction; it is a cost transfer to your most expensive people. The value calculation is total cost against total benefit, over a real time horizon — not the sticker price of the tools.
What This Looks Like in Practice
- Measure the full cycle, not the fast step. Track time from idea to working production, so gains in drafting cannot hide losses in review, QA, or rework.
- Read cycle time and quality together. Falling cycle time with steady-or-better quality is real value. Falling cycle time with degrading quality is borrowed speed.
- Trend risk deliberately. Security findings, architectural consistency, test coverage, and team comprehension of the codebase are tracked over time, not assumed.
- Count the whole bill. Tooling and model usage plus review time, eval maintenance, and the eventual cost of materialized risk — against total benefit, over a real horizon.
- Establish a baseline. Measure before-AI so after-AI claims mean something.
Common Mistakes
- Vanity metrics. Prompts sent, licenses activated, lines accepted, demos given. Activity dressed as value, and a direct invitation to optimize the wrong behavior.
- The single-step illusion. Celebrating faster code generation while ignoring that the overall delivery loop did not move.
- Quality-blind speed. Tracking cycle time without tracking defects and incidents, and calling debt accumulation a productivity gain.
- The invisible bill. Comparing tooling cost against benefit while omitting review hours, maintenance, and risk — the parts that actually dominate.
- No baseline. Making before-and-after claims with no honest "before."
