Enterprise AI Needs More Than Models and Agents. It Needs an Orchestration Layer.

“Just add an agent” is becoming the new “just add a chatbot”. It sounds like an answer because it names a technology. It usually avoids the harder question: what, exactly, is responsible for getting the work done?

The first wave of enterprise AI was understandably preoccupied with the models. Companies wanted to know which one was most capable, which had the largest context window and which supplier was safe enough to build around. Those questions still matter, but they have never described the whole system.

The conversation has now moved towards agents: give the model tools, allow it to plan, and let it carry a task through from beginning to end. That is useful. It is also where the same category error can reappear. An agent can be an important component of an enterprise AI system without being the enterprise AI system.

When we first tried to make AI useful inside ProdPad, we ended up building a lot around the model: state, routing, permissions, validation, retries and the hand-offs back to a person. I previously called that an AI OS because it did some operating-system-like jobs. The industry has since converged on a plainer term, and frankly a better one: the orchestration layer.

You will also see people call parts of this a harness or a control plane. OpenAI now describes the harness as the control plane around the model: it owns the agent loop, tool routing, hand-offs, approvals, tracing, recovery and run state. That is useful technical language. I prefer orchestration layer for the broader enterprise problem because the system must coordinate more than the model loop. It has to coordinate deterministic software, business workflows and people too.

What became clear during that work was that the difficult part was not getting a model to do something impressive once. It was making the result dependable enough to become part of somebody’s work.

A demo only has to work once

Most AI demos are happy paths wearing a lab coat. The context is clean, the task is narrow, and the example has usually been selected because it demonstrates the capability well. The prompt has been refined, the tools are available, and somebody who understands the system is close enough to intervene when it wanders.

There is nothing inherently wrong with that. A demo proves that something is possible. The trouble begins when we confuse that proof of possibility with evidence that the same system is ready to operate inside a business.

Production has missing context, inconsistent data, unclear requests, permissions, tool failures and exceptions that nobody wrote down because everyone involved already knew the workaround. It also has people who expect the system to work on the tenth attempt, with a different user and less convenient data, not merely during the sales demonstration.

Enterprise controls such as access, approval, traceability, ownership and exception handling are not details to add once the agent works. They are part of the thing that has to work. And they cannot live only in policies and control documents. They have to be reflected in the architecture: in what each step can see, what it can do, how the result is checked and when the decision returns to a person.

Agents are flexible, which is both their value and their problem

Agents are attractive because they can work out how to pursue a goal rather than following a completely fixed route. That flexibility is useful when the work is genuinely ambiguous, but it also makes cost and behaviour harder to predict.

The same request may lead an agent to retrieve different amounts of context, call a different sequence of tools or repeat a reasoning step that adds little value. It may produce a plausible answer adjacent to what the workflow required. It may continue working when it should have stopped and asked for help.

None of these outcomes necessarily looks like an obvious model failure. The result may read very well. That is why reviewing the prose is a poor way to decide whether the workflow succeeded.

At enterprise scale, every extra tool call, oversized context window and unnecessary retry becomes part of the cost of the process. Every result that needs somebody to inspect whether the agent actually did the right thing reduces the productivity gain that justified the automation.

The outcome needs to be dependable, not the wording

Large language models are probabilistic. We cannot turn them into traditional deterministic software by writing an increasingly elaborate system prompt, and we do not need to.

If the system is synthesising customer feedback, the wording can vary. What matters is that it used the relevant evidence, respected permissions, distinguished a recurring pattern from an isolated request and did not invent support for its conclusion.

The same applies to a guided workflow. The conversation can vary according to the user, but the system still needs to recognise what has been completed, trigger only permitted actions and know whether the workflow can move to the next step.

Enterprise AI does not need deterministic prose, but it does need dependable workflow outcomes.

Variability is acceptable where it helps the model reason or communicate. It is much less acceptable around authority, evidence and completion.

Make the work smaller before making the agent bigger

Anthropic makes a useful distinction here: workflows follow predefined code paths, while agents dynamically decide how to pursue a task. Enterprise systems will usually need both. The mistake is assuming that adding more autonomy removes the need for workflow design. In practice, autonomy makes the surrounding orchestration more important, not less.

One of the most effective ways to manage model variability is to stop asking one agent to carry the whole process.

Planning is not the same job as gathering data. Gathering data is not the same as interpreting it. Producing a recommendation is not the same as having authority to execute it. When those responsibilities are collapsed into one open-ended loop, it becomes difficult to see where the result went wrong or what should be improved.

We have found it more useful to break work into bounded steps. Each step can have a clear purpose, limited context, constrained tools, an expected output and an explicit definition of completion. Some steps are read-only. Some prepare an action. A smaller number may change product data, and consequential decisions still return to the user.

The model remains probabilistic, but its uncertainty is contained inside a system whose boundaries we can understand and test.

The orchestration layer is also the cost-control layer

Recent research tested the same 22 enterprise tasks across six foundation models while changing only the harness around them. The orchestration layer reduced cost per task by 33–61% across every model, cut tokens per task by 38% and reduced median latency by 44%, while overall task quality remained broadly at parity.

That matters because most input tokens are not chosen by the model. The application decides which system instructions, history, tool definitions, retrieved material and intermediate results are sent on every turn. A naive agent loop can replay all of that repeatedly. A disciplined orchestration layer can cache stable context, compact older history, offload bulky results and stop a failed workflow before it burns through another series of retries.

Model pricing matters. But model choice cannot compensate for an architecture that sends the model far more than it needs.

What we had to build around the model

Much of the work that made Conductor reliable was unglamorous. We clean empty or default arguments before calling a tool. Then we validate structured output rather than assuming valid-looking JSON is semantically correct, check identifiers and permitted values, retry or repair output that fails validation, and stop when the system lacks enough information to proceed safely.

These details matter because models fail in ordinary software-shaped ways as well as unusual AI-shaped ones. They omit required fields, use the wrong identifier, choose an invalid option or confidently produce a structure that is technically well formed but inappropriate for the current workflow state.

Asking the model to be more careful helps only up to a point. Reliability comes from designing for these failure modes rather than pretending they will disappear with the next release.

Testing the workflow rather than the script

We cannot judge an AI workflow by checking whether it repeats a particular sentence. A good model may express the same conclusion in several acceptable ways.

Instead, we test scenarios. Can users express the same intention differently? Can they skip ahead, change their mind or complete part of the work elsewhere in the product? Can an empty result be handled without collapsing the flow? Can the system recover when a model response fails structural validation?

The aim is not to force every user through the same script. It is to establish that varied and realistic paths still lead to the correct workflow state, or to a sensible request for human help.

Governance works better when it starts with the workflow

Enterprise AI needs auditability, monitoring, authority management, revocation and a route for people to challenge important decisions. But governance is not a substitute for deciding how the work should be structured.

There is a tendency to imagine one broadly capable agent with access to everything, then put a control framework around it. That begins with too much authority and tries to manage the risk afterwards.

A better first control is to decide what each part of the workflow actually needs. Give retrieval read access to the relevant sources. Give execution only the tools required for that action. Keep human approval where judgement or consequence demands it. Governance can then observe a system whose authority is already bounded by design.

Why orchestration matters

Orchestration holds these decisions together. It determines which context is relevant, which model or tool should be used, what the current workflow state allows, how the result is validated and whether the system should continue, retry, ask or stop.

This is what Conductor does inside ProdPad. It coordinates model capability with user intent, product context, business rules, workflow state and interface actions. The aim is not to remove the judgement of the product team, but to make sure AI participates in the workflow without being asked to invent and govern the workflow at the same time.

Models matter. Agents matter. But neither is the system that makes the work dependable. That job belongs to the orchestration layer – the part that decides what the AI sees, what it may do, what counts as done and when a person needs to take over.

The enterprise ends up with an agentic estate whether it plans one or not

IBM has started using “agentic estate” for the collection of agents, models, tools and workflows spread across an organisation. It is useful language because most companies will not design that estate neatly from the beginning. Teams will experiment, vendors will add agents to existing products, and useful prototypes will escape into daily work.

The estate therefore exists before anybody has agreed how it should be operated. That is why IBM is also talking about an agentic control plane: a central way to see what agents are doing, govern how they behave, manage credentials and trace their work across frameworks.

That control-plane problem sits beside the orchestration problem rather than replacing it. The control plane gives the enterprise visibility and authority across the estate. The orchestration layer decides how a particular piece of work moves between models, tools, software and people. Conductor is where that work is made dependable inside ProdPad.

Conductor is the orchestration layer inside ProdPad. It decides what CoPilot PM sees, what it may do, and when the work is done, so you can trust the output. See CoPilot PM in action