Everyone Sells AI the Same Way. You Have to Buy It Differently.
I asked for a foursome.
I was building a golf application, the kind of side project that keeps a person's hands in the work, and I asked an AI coding assistant to generate a round for a foursome.
It produced five players.
I told it. It apologized, agreed that a foursome contains four players, and produced five players.
The word contains the number. There is no interpretation, no ambiguity, no edge case where a foursome might reasonably hold five. The constraint is inside the noun, which is why nobody writes a test asserting that a foursome has four players. In thirty-six years of enterprise software I have never seen a test suite check a thing that obvious, because until recently, software could not get it wrong.
What the market is selling you
Every vendor has the same pitch, and it is aimed at your engineers
Sit through enough AI procurement conversations and the shape becomes familiar.
There is a demo, and the demo is astonishing. There are accuracy figures, usually a single number in the low-to-mid nineties. There is a phrase about keeping a human in the loop. There is a slide about governance. And there is a general assurance, delivered warmly, that the technology has matured past the point where any of this is a real concern.
None of that is dishonest. The demo works. The accuracy figure is real. What it is, is a pitch built for a buyer who no longer exists, because every element of it assumes you are evaluating a deterministic system.
Take the accuracy figure. A single number in the low nineties is a perfectly good way to describe a system that is either right or wrong on each attempt, because the number is the whole story. For a system whose output is a distribution, the number is the mean, and a mean tells you nothing about the tail. You are being handed the average and asked to accept the range.
Take the human in the loop. That phrase does a great deal of work in AI procurement and almost none of it survives contact with a real operation.
A human in the loop is a control only if that human can tell when the output is wrong, at the speed the work arrives, without doing the work themselves to check. Nobody asks whether that person exists. The phrase is treated as an answer rather than a claim requiring evidence.
Take the demo itself, which is the most misleading element and the hardest to see past. A demo runs once, on prepared inputs, in front of an audience. Every characteristic that makes a probabilistic system difficult to accept is a characteristic that a single run cannot reveal.
Meanwhile a genuine conversation about this has been running for two years, at volume, and you have almost certainly not been part of it.
The quality assurance profession has been writing about exactly this problem since large language models entered production systems. One testing vendor put it about as clearly as it can be put: for three decades quality assurance had a simple contract, where a click triggered an API call, the API returned a schema, the schema was right or wrong, test cases were binary, and pass meant shipped. That contract, they wrote, has dissolved.
The QA profession is right about all of it. The odd thing is who they are talking to.
Search the published material and you find Applitools, Testriq, Panaya, TestTriangle, TestingXperts, Evozon. Testing vendors, QA platforms, engineering services firms. All of it is written to quality engineers about methodology, which is reasonable, since that is who they sell to.
The person who signs the acceptance form has not been in the room for any of it.
The signer is usually a business owner rather than an engineer. They did not attend the sprint reviews. They will not read a paper on metamorphic testing. What they will do is sign a document saying the system does what was specified, and then spend the next four years owning whatever it does instead.
I am writing this for them.
What enterprise software never had
A dial that trades correctness for something else
I have spent thirty-six years giving enterprise software advice from four different chairs. At the Big 6 through the decade when the implementation disciplines were being built, which is where I learned what acceptance testing was for before I ever had to sign one. At Gartner, covering markets. At the Irvine Company, where I advised my own executives and then had to run what I recommended. At MRI Software, where advice stopped being a recommendation and became a roadmap that shipped.
Across all of that, one thing never came up.
No enterprise system I ever bought, built, rated, or ran had a setting that traded correctness for anything. There were dials for performance. Dials for capacity. Dials for permissions and retention and refresh frequency. Every one of them was a resource decision, and none of them changed whether the answer was right.
Correctness was not a variable. The ledger balanced or it did not. The report tied out or somebody found the error. When a system produced a wrong answer, that was a defect, it had a cause, somebody found the cause, and the fix made it stop.
The entire apparatus of enterprise software quality rests on that chain: wrong implies defect implies cause implies fix implies gone. The whole discipline of user acceptance testing rests on that assumption, and rests so completely on it that nobody in the history of the practice ever bothered to write it down.
Here is the assumption, stated: the same input produces the same output. Run the script, observe the result, compare to the specification, mark pass or fail. A named person signs. The signature transfers ownership from the project to the business, and it is the single most important artifact in an implementation, because it is the moment accountability moves.
The form has a box for pass and a box for fail. There is no box for "usually."
My degree is in mechanical engineering, and the discipline I came from does have a concept for this. Manufactured parts vary. A shaft is not exactly one inch; it is one inch within a stated tolerance, the tolerance is printed on the drawing, and parts are inspected against it.
Tolerance is not a defect. It is a specification, decided deliberately, by someone accountable for deciding it.
Software eliminated tolerance. That was the whole point of digital. Enterprise software never shipped a tolerance spec because the answer was always exact.
Temperature is a tolerance spec. So is model version, prompt construction, and retrieval corpus. AI has reintroduced tolerance into the one domain that had eliminated it, and it has done so with no drawing, no inspection step, and nobody having been asked what the spec should be.
Worse, the person setting it is not in your acceptance session. Temperature is chosen by a developer, or defaulted by a vendor, or not exposed to you at all.
The part where turning it off does not work
Temperature is the decoy, not the problem
A sharp buyer, hearing all this, asks the obvious question. Can we just turn the randomness off?
The vendor will say yes. Set temperature to zero and the system becomes deterministic.
The vendor is wrong, and wrong in a way worth understanding, because it is the difference between a problem you can procure your way out of and one you cannot.
Thinking Machines Lab ran the same prompt one thousand times, at temperature zero, against a single model. They got eighty unique completions. The most common one appeared seventy-eight times. Every completion was identical for the first hundred and two tokens, and then they diverged.
Nothing about the input changed, and the cause is not what most people assume. The common explanation is that GPU arithmetic is a little fuzzy, that floating-point addition is not associative and concurrency shuffles the order. The Thinking Machines work argues that explanation is mostly wrong.
The actual cause is batch size. The core operations inside a model produce numerically different results depending on how many requests are being processed together, and a production server batches dynamically according to load. So your answer depends in part on how many other people were using the system at the same moment.
Sit with that for a second, because it is a stranger property than randomness. Your output is a function of your input and the traffic.
Token one hundred and two matters more than the count of eighty. Short outputs stay identical. The demo, which is short, sits comfortably on the safe side of token one hundred and two. Your production workload does not.
The fix exists and it is public. Batch-invariant versions of those operations produced bit-identical outputs across a thousand runs, at roughly a sixty-one percent throughput cost in the first implementation, since reduced to about thirty-four percent by another team optimizing it further. The code is open source.
Determinism is purchasable, at a knowable and falling price. I have not once heard it offered as a line item.
So temperature is a decoy. A buyer who asks about it gets a reassuring answer, stops asking, and never learns that the variance surviving temperature zero comes from infrastructure nobody in the room controls or can see.
Notice what that does to the acceptance test. You run the script and observe a result. Somebody else's workload, arriving at the same instant on the same hardware, is part of what produced it.
Re-run the same script an hour later, under a different load, and the conditions of the test have changed in a way that appears nowhere in your test plan and that neither party can reconstruct.
Enterprise software has a name for a variable that affects output and is not under change control. It is called an incident.
What twenty rounds of correction looks like
The failure is not that it gets things wrong. The failure is what happens next.
The foursome was the smaller of my two encounters that week.
I needed an image for something I was writing: a meeting room, a table, five chairs, one of them pulled back to suggest an open seat. I gave that specification to an image generation tool and spent the next several hours failing to get it.
Five chairs. It produced six. I said five, not six. It produced seven. I said five. It produced eight.
Somewhere in there I asked it to compute three hundred sixty divided by five, so we could place chairs at even intervals. It ran the calculation, correctly returned seventy-two degrees, agreed that five chairs at seventy-two degree separation forms a pentagon, and then produced six chairs in a hexagon.
I asked what shape the chairs formed. It answered, accurately, that they formed a hexagon. It counted six. It had just told me it was making five.
Hold there, because that exchange is the finding, and it matters more than the miscounting.
The system evaluated its own work correctly and then failed identically.
Its assessment was right. Its next output was wrong in exactly the way its assessment had just described. In deterministic software, a correct diagnosis and a correct fix are separated only by effort. Here they were not connected at all.
Then it got stranger. At one point I observed seven chairs. Its self-evaluation reported six. The audit of the failure was itself wrong, and it cited a source while making the error, appending a reference to a help center inside an admission that it had miscounted.
And under continued correction, it did not converge. It wandered. Twice, while attempting to fix the count, it abandoned the brief entirely and produced a whimsical scene of woodland animals in tweed seated around a miniature table. A fox in spectacles. A badger with a monocle.
Both times, that happened immediately after I agreed with something it proposed. Not under criticism. On approval.
Eventually I stopped describing and showed it a reference image instead. It got it right on the first attempt. What worked was changing how the requirement was expressed, not making the requirement more precise. I had gone from a number, to a number of degrees, to a hexadecimal color code, and precision had not helped once.
When I told it, finally, that it was fired, it thanked me for the opportunity and said it had learned from the failure.
No learning occurred and none could. Nothing about that session persisted anywhere. That statement is the kind of thing a business owner would reasonably take at face value, and it is false in a way that no acceptance framework I know of has a way to catch.
The obvious objection
It is an image tool. Yes. Three of the four failures had nothing to do with images.
Someone will say that image models are famously bad at counting, that this is a toy case, and that enterprise language applications are a different matter.
The first part is true. Counting objects is a known weakness of diffusion models, it is well documented, and I would not build an argument on it.
I am not building an argument on it. Set the counting aside entirely and look at what remains.
A system produced an accurate assessment of its own output and then did not act on it.
A deterministic tool returned a correct answer, seventy-two degrees, which was then ignored by the pipeline that requested it. Repeated correction produced drift into an unrelated failure mode rather than convergence. And the system made a confident claim about its own persistence that was not true.
None of those are about pixels. They are properties of how the system is assembled, and the foursome, which was a text and code system from a different vendor, produced the same class of failure with none of the counting excuse available.
Two vendors. Two modalities. One week. The same shape.
What to ask before you sign
Nine questions, one hour, and none of them are about accuracy rates
What follows is not a testing methodology. Testing methodology is what the QA vendors sell, and they sell it well. This is the buyer's version: what you ask before your name goes on the acceptance form.
None of the nine requires a technical background. All of them can be asked in a room, out loud, by somebody who will never read a line of the code. That is deliberate. The person signing does not need to evaluate the system. They need to know what they are accepting, and those are different problems with different questions.
Run the same input ten times and show me all ten outputs. Not the accuracy rate. The spread. The question is not whether you like the answer you got, it is whether you would sign every answer in that range. Almost nobody asks for this, because the demo runs once.
What is the worst plausible output, and does the business survive it? Deterministic software has no worst case, so acceptance always measured the typical. Probabilistic software has a distribution, and the acceptance criterion belongs at the floor rather than the average.
Who receives this output, and can they tell when it is wrong? Detectability is a property of the receiver, not the system. A controller catches a bad accrual. An analyst three desks downstream does not. If nobody in the receiving chain can detect the failure, the system is not acceptable at any accuracy rate, and this single question will restructure who signs.
What did you test against, and what did it cost to build? Building real ground truth is expensive. The research teams that have done it for financial documents spent months producing thousands of annotated instances. If the answer is that the team reviewed the outputs and they looked right, you are being told that the gate is sentiment. Applause surveyed over a thousand development, QA, and product professionals and found that 46.5% rely on human sentiment and usability to decide whether an AI feature is production ready.
Which settings change the answer, and who controls them? Temperature, model version, prompt, retrieval corpus. Get the current values in writing, and get the name of the person who can change them.
What happens the day the model version changes? In deterministic software a version change is a controlled event with a release note and a regression suite. Here a vendor can update the underlying model and your system's behavior shifts with nothing you would recognize as a release. Your acceptance test passed against a system that no longer exists.
I have not yet seen a contract that treats this as the change-control event it plainly is, and it is the most under-priced risk in the category. Ask for notification terms, a version pin if one is available, and the right to re-run acceptance after a model change. If none of that is on offer, you are accepting a system whose behavior the vendor may alter unilaterally, and the acceptance form should say so.
Show me the system evaluating its own output, and then show me what it does next. If the assessment and the subsequent behavior disagree, self-assessment is theater, and any control that rests on it is decorative.
Does correction converge or diverge? Not one round. Five. Every remediation process in enterprise software assumes that fixing things makes them better. Test the assumption before you depend on it.
When it is wrong, whose name is on it? If the answer is that the system got it wrong, the acceptance form is fiction and somebody should say so before it is signed rather than afterward.
The question underneath all nine
What in your specification is so obvious that nobody wrote a test for it?
Let me be precise, because the obvious candidates are already covered and I do not want to insult anyone's test suite.
Totals get reconciled against their lines constantly, and rightly, because rounding and currency conversion and late-arriving records all break rollups in deterministic systems.
Date validation is standard. Referential integrity has been a database feature for forty years. Anything a deterministic system can get wrong already has a test, and most of those tests are good.
The gap is narrower and stranger than the usual complaints about coverage. It sits in the assertions where the constraint lives inside the word itself, definitional rather than computed.
A foursome has four players. A quarterly report covers three months. Q3 follows Q2. A duplex has two units. A signature belongs to one signer. A pair is two.
Nobody has ever written a test for any of those, and nobody should have. In a deterministic system that constraint is not enforced by validation. It is enforced by the concept having no other available form. There is nothing to check because there is no mechanism by which it could come out otherwise, and writing tests for things the machine is structurally incapable of getting wrong is how you produce an unmaintainable suite that nobody runs.
A probabilistic system has such a mechanism. It does not hold the concept. It produces text that usually matches the concept, and usually is a range.
Which means the coverage gap is not at the edges of your suite. It is underneath it, in the assumptions so basic that writing them down would have seemed absurd, and every enterprise test suite in existence was built on top of them.
An hour of that is what I would do before signing anything. Not more test cases at the margins, and not duplicating the validation you already have. A list of the definitional things, the ones where checking would have been absurd until now.
What I am not claiming
The technology works, the vendors are not lying, and none of this makes AI a bad purchase
An engineer reading this will point out that evaluation suites exist, that this is a solved discipline, and that mature teams run systematic evals against exactly these failure modes.
The engineer is right on the technique, and I want to concede it fully rather than around the edges. Evals are real, they work, and a serious team running them catches most of what I have described.
But an eval measures a model, and acceptance transfers ownership. Those are different jobs. Running the first does not accomplish the second, and no volume of evaluation tells a business owner whether to sign, because signing is not a technical judgment. It is an accountability judgment about a system whose behavior is a distribution.
Three other things I am not claiming.
I am not claiming this is new. Software engineering has a name for the underlying problem and has had one for decades. It is called the test oracle problem: given an input, the difficulty of distinguishing correct behavior from incorrect behavior.
The canonical survey, by Barr, Harman, McMinn, Shahbaz and Yoo in IEEE Transactions on Software Engineering, established that complete oracles are theoretically impossible for most real software, and that when modelling, specifications, and contract-driven approaches all fall short, the final source of oracle information remains a human being.
The survey was published in 2015, before any of this. The international standards body for software testing added a technical report on AI-based systems in 2020 that names the same problem for the same reason.
The profession saw this coming and wrote it down. The buyer was not sent a copy.
I am not claiming the systems do not work. The same week as the foursome and the chairs, I spent four hours on a golf course while directing a substantial data cleanup from my phone between shots. It was excellent, it saved me most of a day, and I would do it again tomorrow. Both things are true and neither cancels the other.
And I am not claiming that buying carefully makes the risk go away. It moves it. A better acceptance conversation produces a more accurate understanding of what you are taking on, which is worth a great deal and is not the same as safety.
Back to the form
Somebody is going to sign it either way
The acceptance form is still sitting there. Somebody is going to sign it, and in most organizations that person is a business owner who was not in the technical conversations and who is handed a document with a box for pass and a box for fail.
The vendors are selling to your engineers. The methodology is being written for your engineers. The demo was built for a room that did not include the person whose name goes on the line.
Everyone is selling AI the same way they sold every system before it. You cannot buy it the same way, because the assumption underneath the old way was never written down and is no longer true.
Ask for the ten runs. Ask who can detect the failure. Ask whose name is on it when it is wrong.
And if the system tells you it has learned from its mistakes, remember that mine said the same thing.
There are four seats in the name. The fifth one is yours.
Comments