Confidence is not competence
The defining characteristic of AI-generated code is that it is confident. The model does not hedge. It does not say "I think this handles the edge cases, but you should probably check the concurrency behavior." It produces clean, well-formatted, plausible code with the serene assurance of something that cannot experience consequences. And that is precisely the problem, because human reviewers unconsciously read confidence as competence. Neat code feels correct.
A model optimizing to produce output that looks right will, reliably, produce output that looks right—which is not the same as output that is right, and occasionally is its exact opposite wearing a convincing costume. It will invent a method on a library that does not have it, because a method with that name should exist and the model is pattern-matching a plausible world. It will write authentication that reads correctly and quietly permits the thing it was supposed to forbid. It will handle the inputs you demonstrated and silently assume the ones you did not. None of this announces itself. It all looks like the surrounding, correct code.
The failure modes have a shape
Once you have cleaned up enough of it, you start to recognize the recurring patterns. They are worth naming, because naming them is how you learn to look for them.
The happy-path special. The generated function does exactly what the example implied and nothing more. Empty inputs, nulls, timeouts, partial failures, unexpected types—the unhappy paths that constitute most of real engineering—are simply absent. The code is not wrong so much as unfinished in a way that passes a casual glance.
The plausible hallucination. An API call, a config key, a function signature that does not exist but absolutely sounds like it should. These are insidious because they are reasonable. The model has produced the code that would exist in a slightly better-designed universe, which is cold comfort in this one.
The subtle mismatch. Data shaped one way here and consumed another way there; an off-by-one in a boundary condition; a timezone assumption that holds until a user in Auckland does something ordinary. The system runs, the tests you happened to write pass, and the defect waits patiently for production traffic.
The silent architectural drift. Each generated piece is locally reasonable and globally incoherent. New patterns accrete beside old ones instead of extending them. No single diff is wrong; the sum is a codebase nobody holds a clean mental model of. This is the most expensive failure mode precisely because it never triggers an alarm.
Why "it works" is a trap
The most dangerous three words in AI development are "it works." What people usually mean is "it ran once, on my machine, with the input I tried." That is a demonstration, not a verification, and treating it as verification is how invisible debt gets a mortgage.
Working software is a much taller claim. It means the code behaves correctly across the inputs you did not try, degrades sensibly when its dependencies misbehave, holds up under concurrent use, resists the inputs a hostile user will eventually send, and remains changeable six months from now by someone who was not in the room. AI generation gives you the first, cheap version of "works" almost for free. Every other, more valuable meaning of the word still has to be earned the old-fashioned way. The platform providers themselves are candid about this: guidance from OpenAI and Anthropic consistently frames model output as something to validate, not something to trust on sight.
Our point of view at Chapter Two
We treat AI-generated code exactly as we would treat a first draft from a fast, knowledgeable, slightly overconfident colleague—one who has read an enormous amount and been personally accountable for none of it. That framing is not an insult to the tools; it is how you get the most out of them without getting burned.
A first draft from a strong colleague is genuinely valuable. It saves hours. It surfaces approaches you might not have considered. It is also not something you ship unread, because your colleague, however capable, does not carry the responsibility for what happens when it fails at scale. You do. So the draft goes through the same gauntlet everything else goes through: it gets read by a human who understands the system, checked by tests that assert real behavior, and reviewed by someone other than whoever—or whatever—produced it. The generation got faster. The standard did not move.
This is also why we are dogmatic about independent verification. When the author of a change is a model optimizing for plausible output, self-review is worth even less than usual, and it was never worth much. The check has to come from a different vantage point with different incentives. That is not bureaucracy; it is the mechanism that converts confident-looking code into code you can actually stand behind.
What This Looks Like in Practice
In our workflow, generated code is guilty until proven innocent, and we mean that cheerfully. When an agent produces a change, we do not ask "does it run." We ask what it does with the inputs nobody demonstrated, whether every API and method it references genuinely exists, whether the data shapes line up across the boundaries it touches, and whether it extends the existing architecture or quietly starts a competing one.
We run it against real and adversarial inputs, not the flattering ones. We lean on tests as the objective arbiter, because a passing test is evidence and a confident comment is not. And we keep the review honest by ensuring the reviewer is not the builder—human or model. Systematic evaluation practice exists for exactly this reason: when your inputs are probabilistic and self-assured, the only durable defense is structured, repeatable checking. The result is that we move fast on generation and stay slow, on purpose, on trust. That asymmetry is the whole trick.
Common Mistakes
- Reading confidence as correctness. Clean, assured code is not more likely to be right; it is only more likely to be believed.
- Accepting "it works" as verification. Running once is a demo. Behaving correctly under real conditions is the actual bar.
- Trusting referenced APIs without checking. Models hallucinate plausible methods and config keys that do not exist.
- Letting generated code drift the architecture. Locally reasonable pieces can sum to a globally incoherent system.
- Skipping independent review because a model wrote it. If anything, model-authored code needs the independent check more, not less.
If AI-generated code is a draft that requires human judgment to become trustworthy, then the obvious follow-up is: which parts of engineering can never be handed off? The tools are getting better at generating code astonishingly quickly. They are not getting better, at anything like the same rate, at the judgment that decides whether the code should exist at all. In the final article of this series, we map exactly where humans remain indispensable.
