Evaluation: know whether it works before your users do
You cannot improve what you cannot measure, and you certainly cannot ship it with confidence. Evaluation — "evals" — is how you turn "seems good" into "here is the number." An eval is a repeatable test that scores the model layer's output against expectations, run before you change a prompt or model and continuously after you ship.
This is a discipline with real depth, and we lean on primary guidance rather than inventing our own metrics from scratch. OpenAI's evaluation best practices is a sensible starting point for building an eval suite; their guidance on agent workflow evals and the cookbook example on evaluating agents are useful when your feature does more than answer in one shot. The vendor is not the point; the habit is.
A few things we have learned about evals that the tutorials underplay. First, your eval set is a product artifact, and it should grow. Every real failure your users hit becomes a new test case, so the same mistake cannot return unnoticed — the AI equivalent of a regression test. Second, evals must reflect the job, not a proxy for it. "The answer contained the keyword" is easy to measure and frequently meaningless; "the answer was correct and did not fabricate a source" is harder and actually worth knowing. Third, run evals in your pipeline, not in someone's notebook the week before launch. A prompt change is a code change. It deserves the same gate.
The uncomfortable truth is that a model that scored well last month can score worse this month after a change you did not make, because the model itself was updated upstream. Continuous evaluation is how you find out on your terms rather than through a support ticket.
Observability: when it breaks, you need the tape
When a user reports that an AI feature gave a bad answer, the first question is "what actually happened," and the second is "can we even find out." For a distressing number of AI features, the answer to the second question is no. The logs say "AI feature invoked" and stop, which is roughly as helpful as a flight recorder that captures only takeoff.
Real observability for an AI feature means capturing the whole chain: the input the user gave, the context and documents that were retrieved, the exact prompt assembled, the model's raw output, any tools it called and what they returned, and where the deterministic layer accepted or rejected its proposal. With that tape, a bad answer becomes a solvable bug — you can see that retrieval returned the wrong document, or that the model's output was fine but validation was too strict, or that a tool timed out and the feature papered over it.
There is a privacy dimension here that we take seriously and you should too. This telemetry often contains user data, so it needs the same access controls, retention limits, and redaction as any sensitive store. Observability is not an excuse to build a shadow copy of your customers' information in your log aggregator. Log what you need to debug, protect it like production data, and expire it on a schedule.
Fallbacks: fail like a professional
Deterministic systems fail in predictable ways, and we design around them without thinking — retries, timeouts, circuit breakers. The model layer needs the same courtesy, because it will fail: it will time out, return low-confidence output, produce something that fails validation, or occasionally return an answer that is confidently, articulately wrong.
Designing fallbacks means deciding, in advance, what happens in each of those cases. A low-confidence classification might route to a simpler deterministic default. A failed validation might fall back to a cached prior result, or to a plainer non-AI experience, or to an honest message that the feature is temporarily unavailable. A high-stakes action — anything that spends money, deletes data, or contacts a customer — should not be executed on the model's say-so at all; it should be staged for human approval. Keeping a human in the loop for irreversible actions is not a lack of ambition. It is what "reliable enough" means when the cost of being wrong is real.
The failure mode we hunt for hardest is the silent one. A model call that errors and returns an empty result which then renders as a blank-but-plausible answer is far more dangerous than a call that fails loudly, because nobody gets paged for a wrong answer that looks fine. We would rather a feature say "I couldn't do that" than confidently hand a user something invented.
What This Looks Like in Practice
- Build an eval suite before launch and run it continuously. Start from established guidance, score the actual job rather than a proxy, and gate prompt and model changes on it the way you gate code.
- Turn every production failure into a test case. The eval set grows with reality so old mistakes cannot quietly return.
- Capture the full chain. Input, retrieved context, assembled prompt, raw output, tool calls, and the control plane's decision — protected and expired like the sensitive data it is.
- Design fallbacks for every failure class. Low confidence, validation failure, timeout, and high-stakes actions each get a defined path. Irreversible actions wait for a human.
- Hunt the silent failure. Prefer a loud "couldn't do that" over a plausible-looking fabrication.
Common Mistakes
- Shipping on vibes. Launching because the demo felt good, with no repeatable measure of quality and no way to notice regressions.
- The blind feature. Logging only that the feature ran, then trying to debug a specific bad answer with nothing to look at.
- Trusting confidence. Treating a fluent, assured model response as a correct one. Fluency and accuracy are unrelated variables.
- Auto-executing risky actions. Letting the model directly trigger irreversible operations because adding a human step felt like a step backward. It is the opposite.
- Telemetry sprawl. Logging everything, forever, unprotected, and creating a privacy problem in the name of reliability.
