What an agent actually is

There is one question that separates the four kinds of software people currently call an agent: who decides what happens next, and when was that decision made. In a script, you decided, at design time, and the software follows. In a chatbot, the software answers and a person acts. In a copilot, the software drafts inside a person's workflow and that person commits the result. In the fourth case, the software chooses the sequence while the task is running, calls the systems that hold the state, and finishes with a changed record — a credit note raised, a case closed, a supplier record updated — rather than a message on a screen.

Most of the time, the correct answer is the script. Anthropic's own engineering guidance, which you would expect to argue the other way, says to find "the simplest solution possible, and only increasing complexity when needed. This might mean not building agentic systems at all" (anthropic.com/engineering/building-effective-agents). The test is whether you can draw the flowchart. A nightly reconciliation between two systems with fixed schemas, no exceptions, is a cron job and some SQL. It will be cheaper, faster, deterministic, and auditable line by line. Putting a language model in that path buys you variance you did not need and a bill per run.

Scripts stop working for a specific reason, and it is worth naming it precisely. Thoughtworks, writing about robotic process automation, puts it plainly: "If anything changes with the interface, the data, or any other aspect of the legacy app, then the RPA breaks" (thoughtworks.com/en-us/insights/articles/four-bad-ways-use-rpa). A script encodes one path through one version of the world. When the variation is in the input rather than the interface — a supplier who writes the purchase order number in the email body rather than the reference field, a claim that mentions a second policy — you are maintaining a growing pile of exception branches, and each one was written by someone who has since left.

Chatbots and copilots each have a correct use. A chatbot is right when a person needs an answer and already holds the authority to act on it. A copilot is right when the judgement is genuinely irreducible but the typing is the bottleneck: a solicitor marking up a lease, an underwriter writing a rationale. Neither of these carries a task to the end, and neither should be sold as if it does.

The fourth thing earns its cost in one narrow case: the path varies case by case, the inputs are unstructured enough that you cannot enumerate the branches, and the finish line is a state change in a system you control. Outside that case it is an expensive way to do something a script already does.

Five properties decide this, and they are cheap to check before anyone writes code. The first is written rules that are current. If the rule for when a credit note needs a second signature exists only in the head of the person who has done the job for eleven years, you are not automating a process, you are paying to discover one. That discovery is usually worth doing on its own merits, but it should be budgeted and staffed as a process-definition project, not hidden inside a software estimate.

The second is readable systems. Can software see and change the state, through an API, a database, or at worst a stable screen you are contractually permitted to drive? Go and check the licence terms of the vendor application in question before the meeting, not after. A process that can only be completed by a human inside a hosted session that forbids automation is not a candidate at any price, and finding this out in week nine is a common and avoidable expense.

The third is the one most often skipped: a checkable outcome. After the work is done, can something cheaper than a human tell you whether it was right? "The refund amount equals the sum of the disputed invoice lines and the credit note references the original invoice" is checkable by a query. "The customer was handled well" is not, at least not without building a second system to judge the first. Without a cheap check you cannot measure the thing, which means you cannot run it unattended, which means what you are buying is a copilot.

Fourth and fifth are volume and the cost of an error, and they interact. Low volume with a high cost per error is a copilot: keep the person, speed up their hands. High volume with a low cost per error is the strongest candidate there is, because errors are recoverable and the arithmetic works. High volume with a high cost per error is possible, but be clear about what you are purchasing: an evaluation programme and an oversight rota, of which the software is one part.

One piece of external evidence is worth holding here. The MIT NANDA report that circulated in 2025, based on 150 interviews, a 350-person survey and an analysis of 300 deployments, found that roughly 95% of pilots produced no measurable effect on profit and loss, and that the returns that did appear came from back-office work rather than the sales and marketing tools taking over half the budget. Treat the headline figure with the caution any interview-based study deserves, but the direction matches the five properties above: back-office processes are where the rules are written down and the outcome is checkable.

The model you choose will be replaced within a year, probably by you, probably on a Tuesday. The set of test cases with known-correct outcomes is the thing that survives that swap and tells you whether the replacement was an improvement. That is the asset. If a proposal — ours or anyone's — spends more time on which model it uses than on how correctness is measured, the proposal is thin.

The numbers that matter here are public. On τ-bench, a benchmark of customer-service tasks built by Sierra, GPT-4o scored roughly 50% on the retail domain at a single attempt, and roughly 25% when the same task was attempted eight times with the same underlying meaning phrased differently (sierra.ai/blog/benchmarking-ai-agents). That gap is the whole subject. A demonstration is a single attempt on a case someone chose. Your Tuesday is the eighth attempt. On TheAgentCompany, a simulated software firm built at Carnegie Mellon, the most capable agent completed 30% of tasks autonomously (arxiv.org/abs/2412.14161). On Salesforce's CRMArena-Pro, leading agents managed around 58% on single-turn tasks and about 35% once the interaction ran over several turns, and showed near-zero awareness of confidentiality unless prompted for it — at which point task performance fell (arxiv.org/abs/2505.18878). That last finding is the kind of trade-off no vendor slide contains, and you should ask about it directly.

Measurable correctness, in practice, means something narrower than it sounds. For each test case you write a predicate over the end state, not over the transcript. Not "the agent explained the refund policy", but "a credit note exists, its value equals X, its reference points to invoice Y, and no second credit note was raised". The predicate runs without a human. You can then run the same fifty cases against a new model in twenty minutes, which is the only reason anyone can honestly say an upgrade helped.

Hamel Husain's widely read piece on evaluation makes the organisational point: most teams work only on changing the system and never on measuring it, which "prevents them from improving their LLM products beyond a demo" (hamel.dev/blog/posts/evals). He describes three layers — cheap assertions, human and model review of traces, and live comparison — and one discipline that matters more than any of them: "You must remove all friction from the process of looking at data." Someone has to read the failures, weekly, by name.

So the questions to put to any supplier are these. How many labelled cases do you have, who labelled them, and what did they disagree about? What is the score when the same case is run eight times? What is the score on cases drawn after the system was built? A supplier who cannot answer the third has told you their number is a description of the past.

Human oversight is usually set by fear, which produces one of two thresholds: nothing is automated, or everything is, with a person nominally approving all of it. Both are decisions made without arithmetic. The honest version is a comparison between the expected cost of an error that escapes — its financial and regulatory cost multiplied by the rate your evaluation actually measured — and the cost of the review that would have caught it.

The second half of that comparison is where most designs quietly fail, because a review that exists on the process map is not the same as a review that happens. Parasuraman and Manzey's review in Human Factors (2010) found that operators monitoring automation of constant reliability detected failures 33% of the time, against 82% where reliability varied, and that up to sixty minutes of training did not reduce the effect. Experienced pilots and air traffic controllers were no better than novices (journals.sagepub.com/doi/10.1177/0018720810376055). A person approving four hundred items a day is not a control. They are a signature.

The design conclusion is to make the human see fewer, harder things. Have the software abstain when its own checks do not pass, and route that minority to a person along with the specific evidence — the clause it relied on, the record it matched, the figure it could not reconcile. Sample the remainder rather than nominally reviewing all of it. Thirty difficult cases a day is a control that works; four hundred routine ones is one that does not.

Article 14 of the EU AI Act is a useful checklist here even where it does not bind you. It requires that the person overseeing a high-risk system can understand its capacities and limitations, remain aware of the tendency to over-rely on its output, interpret that output, decide not to use it or to override or reverse it, and stop it in a safe state (artificialintelligenceact.eu/article/14). Article 14(5) goes further for biometric identification and requires two people to confirm before action. Read those five capabilities as build requirements. "Reverse the output" in particular means a real procedure in a real system, written down and tested, not an intention.

Finally, set the threshold as a parameter you revisit rather than something welded into the architecture. METR's measurements put a frontier model's 50%-success time horizon at roughly fifty minutes of human work, doubling about every seven months since 2019 (arxiv.org/abs/2503.14499). Whether that trend holds is genuinely unknown. Either way, the boundary you draw this quarter is a boundary for this quarter.

The first failure is the impressive demonstration. Every demonstration is a single attempt on an input the builder selected, which is precisely the measurement τ-bench showed to be worth about half of what it appears. The counter is simple and slightly rude: ask to see the failures. Ask for ten cases the system got wrong, with the traces. A team that has been measuring honestly will have them to hand and will be relieved to be asked. A team that cannot produce them has not been looking, and you have learned what you needed to know in one question.

The second failure is the process nobody agrees on. You will find it when three experienced people describe the same process three different ways and each is confident. Automating over that disagreement does not resolve it; it encodes one person's version silently and moves the argument to a place where nobody can see it. Thoughtworks put the general case well: "If you have poor business processes, burying them underneath an RPA solution might help you sleep better at night but hasn't solved the underlying problem." The disagreement is a finding, and it is often the most valuable thing an automation project produces. It should be surfaced and settled by the people accountable for the process, in writing, before any software is built.

The third failure is the system with no way out. This takes several forms: an action that cannot be reversed, a vendor platform with no API and no audit trail, or — most commonly — software that acts but leaves no per-step record of what it read, what it called and what it wrote, tied to a case identifier. Without that record you cannot investigate a complaint, satisfy an auditor, or improve the system, because you cannot tell which step went wrong. If an action cannot be reversed and cannot be traced, it does not go unattended at any accuracy level, and no evaluation score changes that.

The costs worth stating plainly, because they are frequently left out: inference cost per task, which is real and recurring and rises with the number of steps; the ongoing work of maintaining the evaluation set as the rules change, which does not stop; the oversight rota, which is a standing staff cost; and the integration work against systems whose owners have other priorities. Against those, the honest benefit case is usually about cycle time and consistency rather than headcount, at least at first.

If you take one thing from this module, take this: the question is never whether the software is impressive. It is whether you can tell when it is wrong, how quickly, and what you do then. A process where you can answer all three is a candidate. One where you cannot is a research project wearing a business case.