Four architecture decisions that get enterprise AI to production.

95% of business AI programs fail to deliver measurable business value (MIT NANDA, 2025). The instinct is to blame the models. The evidence points elsewhere. The models are rarely the bottleneck. The architecture around them is.

Most stalled programs share a pattern: a capable demo, an ambitious roadmap, and an underlying design that quietly mixes responsibilities (orchestration tangled with model access, knowledge retrieval doing the job of a live system query, a workflow that leans on the model to do work the model was never the right tool for). The result is a system that looks impressive in a controlled setting and becomes unmaintainable, untrustworthy, or unaffordable the moment it meets real volume, real data, and real auditors.

This paper sets out four architecture decisions that separate production AI from expensive experimentation. Each is drawn from real engineering review, and each maps to a dimension of Return on Intelligence: the measurable business value generated per dollar spent on AI. They are not exotic. They are the disciplines that decide whether an AI program reaches production at all.

A pilot has to be convincing once. A production system has to be correct every time, at cost, inside the systems the business already runs, with an audit trail when someone asks why it did what it did. Those are not the same engineering problem, and treating them as one is the most common way an AI program dies between the demo and the deployment.

The four decisions that follow are where that gap is won or lost.

The fastest way to make an AI system unmaintainable is to let its layers blur into each other. Three responsibilities in particular get collapsed when they should be kept separate:

The orchestration layer decides what happens and in what order: the sequence of steps, branching, retries, and the flow of a task from start to finish.

The model gateway decides how the system talks to models: routing a request to the right model, managing keys and rate limits, handling fallbacks, and metering spend.

The business workflow engine decides what the business actually does with the output: entering a record, triggering an approval, updating a downstream system of record.

Tools across the market (orchestration frameworks, model gateways, retrieval and vector stores) are each useful, but they solve different problems. When one tool is stretched to cover all three jobs, every change becomes a risk: a routing tweak breaks a workflow, a workflow change leaks into model handling, and no one can reason about the system end to end.

Give each layer one clear job and the architecture becomes something a team can maintain, test, and extend without fear. This puts joined-up intelligence and trust into practice: AI integrated cleanly into the systems the business already runs, with governed, predictable behavior rather than a single brittle layer doing everything at once.

Retrieval-augmented generation (RAG) is powerful and frequently overextended. The discipline is knowing what belongs in retrieval and what does not.

Good candidates for retrieval are stable knowledge: standard operating procedures, scripts, product specifications, and policy documents. This content changes slowly, tolerates approximate matching, and benefits from being surfaced in context.

Poor candidates for retrieval are live operational state: pricing, inventory, orders, purchase orders, and customer or account status. This data is volatile, and being approximately right is the same as being wrong. Pushing it into a vector store produces answers that are stale, unverifiable, and impossible to audit.

Live operational data should be reached through live system queries: direct, validated calls to the system of record, with the result checked and the lookup logged. The test is simple: if being slightly out of date or slightly approximate would cause a real business error, it does not belong in RAG.

In practice, this means deterministic, governed outputs with validation and auditability, not probabilistic guesses about facts the business already holds with certainty.

Consider an order-entry workflow, a classic Order to Cash (O2C) problem. An order arrives as a document; it needs to end up as a clean, correct record in a downstream system. It is tempting to frame this as 'can the model read the order?' That is the easy part. The hard part is everything around it: classification (what kind of document is this, and does it belong in this workflow at all), document parsing and structured extraction (pulling fields out reliably across messy, inconsistent formats), SKU and customer matching (resolving what the document says to the entities that actually exist in the business), exception handling (what happens when something does not match, is ambiguous, or is incomplete), human approval (where a person reviews, corrects, and signs off), and clean entry into the downstream system (committing the result without corrupting the system of record).

The model contributes to several of these steps. It is decisive in almost none of them. This is why the work that determines success is evaluations and confidence thresholds: measuring how accurately each step performs against real cases, and setting the bar above which the system acts automatically versus routing to a human. Get the evals and thresholds right before expanding the workflow, and the system scales safely. Skip them, and every new edge case is a production incident.

Together, this is accuracy benchmarked against the real workload, not a lab demo, with governance gates and a human in the loop where the stakes require one.

Running models on local or private infrastructure has real appeal, usually framed as a data-control or risk decision. But making it a default architecture choice often imports significant operational burden (hosting, scaling, patching, and maintenance) to solve a risk that can frequently be addressed with less effort elsewhere.

Local models make sense as a targeted tool for specific cases: genuinely sensitive workloads, or high-volume tasks where the economics clearly favor it. As a blanket default, they are usually the wrong trade.

For most of the risk that drives teams toward local models, enterprise cloud model contracts, redaction, request routing, and access controls solve more than people expect, with far less operational overhead. The right question is not 'cloud or local?' but 'what is the specific risk, and what is the lightest control that retires it?'

This is cost discipline and trust working together from day one: the right model for each task rather than the default, and governance achieved through controls rather than infrastructure the team then has to carry.

The 95% failure rate is not a verdict on what AI can do. It is a verdict on how AI is built. Separate your layers. Be deliberate about retrieval. Earn your accuracy with evals and thresholds before you scale. Use local models where they actually pay, and controls everywhere else. Do those four things and you are building for the 5% that reaches production.