LLMs Shouldn’t Replace Deterministic Systems

One of the easiest mistakes to make when building with large language models is to confuse something they can do with something they should do.

We ran into this repeatedly while building ProdPad CoPilot and Conductor. Need to understand an ambiguous request? The model is often exactly the right tool. Need to count records, enforce a permission, calculate a score or decide whether a workflow step has been completed? It may still produce an answer, but that does not make it the right place to put the logic.

The models are capable enough that using them for everything can look surprisingly convincing. Give them more context, allow them to reason for longer and perhaps add another agent to check the first one. The demo works. The prompt slowly becomes the application.

That is the architectural mistake. The useful question is not simply which model should perform a task. It is which parts of the task genuinely require a language model at all.

Capability is not the same as suitability

Software engineers have always been able to stretch a technology well beyond the job it was designed for. You can use a document store as a relational database, put business logic into a user-interface component or build an entire system around whatever happens to be fashionable that year. It may even work for a while.

The problem appears later, when the system has to be tested, explained, changed and operated at scale.

LLMs are now becoming the default answer to several quite different problems. In some systems, the model is being asked to act as the search engine, analytics engine, rules engine, calculator, workflow engine and user interface at the same time. Every new requirement becomes another instruction, another tool description or another chunk of context.

That is impressive in a demonstration. It is a fragile place to put the logic of a business.

Language models are useful because language is ambiguous

LLMs are particularly useful where there is no single mechanical route to the answer. They can interpret an incomplete request, infer what a user probably means, connect concepts expressed in different ways, synthesise qualitative evidence and explain a result in language appropriate to the reader.

Those are not peripheral abilities. They are what make the technology valuable.

If I ask for a summary of the main themes emerging from customer feedback, several answers may be defensible. The themes may be described differently, the examples grouped another way and the emphasis changed according to the product context. Probabilistic reasoning is useful because the task itself contains judgement.

The mistake is assuming that every step inside the same workflow contains the same kind of uncertainty.

A database should still do the counting

Consider a different question: which customer segment generated the highest number of unresolved feedback items linked to roadmap initiatives during the last quarter?

The user has expressed the request in natural language, but the underlying work is not primarily a language problem. It requires filtering, joins, grouping and counting. A database or analytical engine such as DuckDB can perform those operations exactly against the underlying data.

The query can be inspected, tested and repeated. The result can be cached. Given the same data and query, it will return the same answer.

An LLM may also arrive at the right conclusion. It may instead omit a record, misunderstand the grouping or produce a persuasive explanation of a calculation it did not perform correctly. Worse, it can be difficult to tell which happened from the prose alone.

The model should understand the question and explain the result. The database should establish the answer.

Mathematics already has better tools

The same principle applies to mathematics.

Modern language models can solve a surprising range of mathematical problems, particularly where the job involves interpreting a vaguely worded question or explaining an approach. But exact symbolic manipulation, numerical computation and verification are not new problems waiting for a language model.

Specialist mathematical systems were built for them. They are deterministic, reproducible and usually much cheaper to run. Their output can be checked in ways that a long chain of model reasoning often cannot.

The LLM can work out what the user is trying to calculate, select the appropriate operation and explain the result. The mathematical engine should do the mathematics.

Rules and workflow state belong in software

Business rules are another area where flexibility can become a liability.

A model can interpret a policy written in ordinary language, identify which parts may apply and explain the policy to a user. It should not become the only place where the organisation’s eligibility rules, approval thresholds, permissions or safety controls are enforced.

Those rules need to be explicit, versioned and tested. In higher-risk systems, the market is increasingly describing this as deterministic control or deterministic policy enforcement: the model can reason around the rule, but it cannot quietly rewrite the rule while applying it.

The same is true of runtime state. A chat transcript may contain clues about what happened, but it is a poor source of truth for whether a required action was completed, which approval is outstanding, what side effects have already occurred or what the system is allowed to do next.

Use the model where interpretation is valuable. Keep authoritative rules and state in software.

The LLM should often be the interface to the analysis

This leads to a rather different architecture from the one where raw data is pushed into a large context window and the model is asked to work everything out.

Suppose a product leader asks: ‘Which strategic themes have the strongest demand from enterprise customers, and which of those are not represented on the roadmap?’

The LLM can interpret the request and identify the concepts involved. The orchestration layer can map those concepts to the relevant product entities, select the tools and decide which steps should be probabilistic and which should be deterministic. DuckDB or another analytical system can perform the joins and aggregation. Validation can check that the result is structurally complete. The LLM can then explain the findings, point out useful caveats and help the user decide what to investigate next.

Salesforce now describes this as a blended orchestration model spanning agents, humans, automation tools and deterministic systems. The terminology is new. The architectural principle is not: use the right kind of system for each part of the work.

The model is doing the parts where language and judgement create value. It is not being paid, in tokens, to imitate a database.

Context engineering should remove work, not hide bad architecture

There is a growing and useful discussion around context engineering: deciding which instructions, data, tools and memories the model should see for a particular step.

That is a much better framing than simply making the context window larger. But good context engineering is not only about finding a cleverer way to show the model more information. It should also identify information and computation that the model never needs to see.

If a tool can return an exact aggregation, send the result rather than thousands of source records. When workflow state already says that a step is complete, do not ask the model to infer it from the transcript. And where a deterministic validator can reject an unsafe action, do not spend another model call debating whether the action looks sensible.

The cleanest context is often the context you did not have to create.

Throwing more tokens at the problem has poor economics

This matters because AI costs can grow much faster than the value organisations receive from them.

A recent study of the same enterprise tasks across six foundation models found that changing the orchestration layer reduced cost by 33 to 61 per cent across every model tested. Tokens per task fell by 38 per cent and median latency by 44 per cent, while headline quality broadly held. On that workload, architecture moved the bill more than changing from the cheapest model to the most expensive one.

That does not mean every optimisation will reproduce those numbers. It does show where a large part of the cost is created: context assembly, repeated history, tool exposure, retries, waiting and failure handling. Those are orchestration decisions, not properties of the foundation model.

A surprising amount of inference is simply compensating for an earlier architectural choice: asking the LLM to perform work another system could complete in milliseconds with a predictable result.

The suppliers and their customers optimise for different things

The frontier labs have understandable incentives here. They build and sell model inference, so their examples naturally emphasise larger models, longer contexts, more reasoning and increasingly autonomous agents. Those techniques can improve difficult model tasks, and the labs are right to develop them.

Their customers have a different optimisation problem.

A model provider is rewarded when more inference is consumed. An enterprise is rewarded when the work is completed reliably, visibly and at lower total cost. The most capable model configuration for a benchmark is not automatically the most economical architecture for a business process.

This does not require a conspiracy. Vendors tend to demonstrate the world through the product they sell. It is the responsibility of the customer and the product team to decide where that product actually belongs.

Orchestration should reduce the amount of LLM work

This is one of the less obvious benefits of an orchestration layer.

Orchestration is not simply a way to coordinate more model calls or more agents. It should decide when a model is needed, which model is sufficient, how much context the step requires, which deterministic controls apply and whether the work would be better performed by ordinary code or a specialist system.

A bounded classification may be handled by a small model. An exact aggregation should go to the database. A policy check should use deterministic rules. A mathematical calculation should use a mathematical engine. A difficult qualitative synthesis may justify a frontier model. A consequential decision may require a human.

This is the part of the current enterprise language I find most useful. The orchestration layer is not there to replace deterministic systems. It coordinates agents, humans, automation and deterministic software into one workflow, while runtime state, traces and validations make that workflow operable.

Good orchestration makes the whole system more capable partly by using less LLM inference. It reserves probabilistic reasoning for the places where probability is an advantage rather than a defect.

Use the right intelligence for each part of the job

LLMs are not replacing deterministic software. They are giving people a much more flexible way to work with it.

That distinction matters. The language model can turn an ambiguous human request into a structured task, choose or prepare the appropriate tool call and make the result understandable. Deterministic systems provide the precision, repeatability and auditability the model does not.

The next generation of useful enterprise AI will not be defined only by larger models, longer contexts or more agents. It will be defined by better decisions about where models should be used and, just as importantly, where they should not.

The right question is no longer ‘Which model should do this?’ It is ‘Which parts of this problem genuinely require a language model?’