Tolerance Spec

The settings that change what a system produces, who controls them, and what range of outputs the business is willing to sign for.

Manufacturing has always had tolerance specs. A shaft is not exactly one inch. It is one inch within a stated tolerance, the tolerance is printed on the drawing, and parts get inspected against it. Tolerance is not a defect. It is a specification, decided deliberately, by someone accountable for deciding it.

Enterprise software eliminated tolerance, because the ledger either balanced or it did not. AI reintroduced it to the one domain that had removed it, with no drawing, no inspection step, and nobody asked what the spec should be.

Where it came from

A vendor is asked whether the randomness can be turned off and says yes, set temperature to zero. One research team ran the same prompt a thousand times at temperature zero and got eighty different outputs, identical for the first hundred and two tokens and diverging after. The cause was not randomness. Core operations produce different numbers depending on how many requests are batched together, and a production server batches by load. The output depends in part on the traffic.

When to run this

Before acceptance. The whole point is that the acceptance form has a box for pass and a box for fail and no box for usually.

The steps

Run the same input ten times and look at all ten outputs. Not the accuracy rate. The spread.
Ask whether you would sign every answer in that range, not whether you like the one you got.
Identify the worst plausible output and ask whether the business survives it. Deterministic software has no worst case, so acceptance measured the typical. This one has a distribution and the criterion belongs at the floor.
List which settings change the answer. Temperature, model version, prompt, retrieval corpus. Get the current values in writing.
Get the name of the person who can change each one.
Ask what happens the day the vendor changes the model version, and whether you get notice, a version pin, or a right to re-run acceptance.

What a bad answer looks like

A single accuracy figure. For a system whose output is a distribution, a number in the low nineties is the mean, and a mean tells you nothing about the tail.

An assurance that temperature is set to zero, offered as though it settles the question. It does not.

A human in the loop, offered as a control. That phrase is a control only if the human can tell when the output is wrong, at the speed the work arrives, without doing the work themselves to check.

What to do with the output

Write the spec down. For each output, the acceptable range, who set it, and who can change it. That document is the thing enterprise software never needed and now does.

Then take it to the contract. If notification terms, a version pin, and a right to re-run acceptance are not on offer, you are accepting a system whose behavior the vendor may alter unilaterally, and the acceptance form should say so.