The stack has two temperaments
It helps to stop thinking about tiers and start thinking about temperament. Parts of your system should be stubborn. Given the same input, they must produce the same output, every time, forever. Authentication, authorization, payments, record writes, quota enforcement, the code that decides whether a customer can see another customer's data — these are your deterministic control plane. They are boring on purpose. Boring is a feature. You can test them exhaustively, reason about them, and hand them to an auditor without flinching.
Other parts of your system should be allowed to exercise judgment. Summarizing a messy document, ranking suggestions, drafting a reply, classifying an ambiguous request, deciding which of several tools might help — these are your probabilistic model layer. They are useful precisely because they can handle inputs you did not anticipate. That flexibility is inseparable from the fact that they will sometimes be wrong.
The core discipline of AI-native design is refusing to let those temperaments blur. The model layer proposes; the control plane disposes. A model can suggest that a refund is warranted. It must never be the thing that moves the money. If you internalize one idea from this piece, make it that sentence, and then go check whether your architecture actually enforces it or merely hopes for it.
Draw the trust boundary first
Before choosing a model, a vector database, or an orchestration framework, we draw a single line on the whiteboard: everything above it can guess, everything below it cannot. This trust boundary is the most important artifact in an AI-native design, and it is usually the one teams skip.
Concretely, that means model output is data, not instruction. When a model returns a decision, it crosses the boundary as a proposal that must be validated by deterministic code before anything irreversible happens. Structured outputs help here — constraining a model to a schema turns free text into something you can check — and the major platforms now support this directly, whether you are working with OpenAI's APIs, Anthropic's, or Google's Gemini API. But a schema only guarantees shape, not truth. A well-formed JSON object can still assert that a banned user is in good standing. The boundary is where you re-check the claim against the record of truth you actually control.
The same logic applies to tools. Giving a model the ability to call functions is how AI-native systems get useful, and standards like the Model Context Protocol have made tool wiring more consistent across the industry. But every tool a model can call is a hole in your boundary. Read-only tools are cheap to grant. Tools that write, spend, delete, or notify a human belong behind the same authorization and validation you would demand of any other client — because from the control plane's point of view, that is exactly what the model is.
Data and retrieval decide the ceiling
Teams love to argue about which model is smartest. It is a fun argument and, for most applications, a secondary one. The ceiling on an AI-native system's quality is usually set by the data it can see and how well it retrieves the right slice at the right moment. A capable model reading stale, poorly chunked, or mis-permissioned context will produce confident nonsense. A modest model reading clean, well-scoped, correctly filtered context will quietly do its job.
So we treat retrieval as a first-class engineering surface, not a bag of embeddings we shove text into. That means owning the boring parts: how documents are chunked, how freshness is maintained, how access control is applied before retrieval rather than hoped for afterward, and how you know when the retrieval step returned garbage. A retrieval layer that silently returns nothing is more dangerous than one that errors loudly, because the model will happily fill the silence.
There is a permissions trap worth naming. Retrieval that ignores the requesting user's access rights is a data breach with extra steps. If your control plane enforces that user A cannot read user B's records, but your vector store cheerfully hands the model user B's documents to summarize for user A, you have built a very sophisticated way to leak data. Permissioning belongs in the retrieval path, enforced by the deterministic layer, not left to a prompt politely asking the model not to look.
Assume the model layer will fail, and design for it
Deterministic code fails in ways you can enumerate. The model layer fails in ways you cannot fully predict, which means the design question is not "will it fail" but "what happens when it does." AI-native systems that survive contact with real users have three things the demos never bother with: evaluation, observability, and fallbacks.
Evaluation means you have a repeatable way to measure whether the model layer is doing its job, ideally before you ship a change and continuously after. This is its own discipline, and it is the subject of a later article in this series, but the architectural point is that your stack needs a seam where evals plug in — a place where inputs, outputs, and judgments can be captured and scored. Observability means that when a user reports something wrong, you can reconstruct what the model saw, what it returned, which tools it called, and where the control plane accepted or rejected its proposal. If your logging stops at "the AI feature ran," you are debugging blind.
Fallbacks mean the system degrades instead of collapsing. A timeout, a low-confidence response, or a failed validation should route to a defined path: a simpler deterministic behavior, a cached answer, a queue for human review, or an honest "we couldn't do that right now." The worst fallback is the invisible one, where a failed model call quietly returns an empty result that looks like a real answer.
What This Looks Like in Practice
Here is how we approach it in engagements:
- Map temperaments before tools. We label every capability as deterministic or probabilistic and put the trust boundary on the diagram before anyone opens a model provider's docs.
- Treat model output as untrusted input. It is validated by deterministic code against sources of truth before any irreversible action. Structured outputs constrain shape; the control plane confirms substance.
- Engineer retrieval and permissions together. Access control runs in the retrieval path. Freshness, chunking, and empty-result handling are owned, tested, and monitored.
- Build the seams for evals, logs, and fallbacks from day one. Retrofitting observability into an AI feature after launch is roughly as pleasant as adding a basement to a finished house.
Common Mistakes
- The model as spinal cord. Routing every request through a model, including ones that are pure deterministic logic, makes the system slower, costlier, and less reliable for no benefit. Not everything needs judgment.
- Confusing schema with safety. A validated structure is not a validated fact. Teams see clean JSON and assume the content is trustworthy.
- Prompt-based permissions. Asking a model to respect access rules in its instructions instead of enforcing them in code. Instructions are suggestions; the retrieval query is the law.
- No degraded mode. Designing only the happy path and discovering the failure modes in production, narrated by customers.
