AI projects tend to stall in one of two ways. In the first, a team evaluates for a year: requirements, a vendor shortlist, a steering committee, a pilot that never quite ends. Nothing reaches production. In the second, a team ships the demo because it looked good in a meeting, and then spends months on support tickets nobody can reproduce.

The two look like opposites, but both teams are missing the same thing. Nobody wrote down, before the build, what "good enough" means.

Why "iterate fast" is not enough

A model gives different answers to inputs that look alike to a person. Change a prompt to fix one case and you can break three others without noticing. If you have no fixed set of cases to check against, iteration is guessing, and production feedback arrives as complaints rather than numbers.

Shipping fast only helps if you can tell whether this week's version is better than last week's.

Why waiting for perfect does not work either

No AI system reaches zero errors. A review with no exit criteria never ends, because there is always another edge case to discuss. More meetings will not produce certainty. The people who decide on go-live need a number to decide against.

Agree the failure budget up front

A failure budget is a written agreement about what the system may get wrong, how often, and what happens when it does. The owner of the process, the person who will be on call, and whoever signs off on risk agree it before the build starts.

Illustrative failure budget for invoice processing: routine invoices from approved vendors only. The system may send up to one invoice in ten to a person for review. It may never post a payment to the wrong vendor; one such error stops the system until the cause is found. Cost per invoice stays under an agreed ceiling.

Notice what the budget does. It turns "is it ready?" into a check anyone can run. It separates errors that are a nuisance from errors that stop the line. And it tells the team what to build: a path for the invoices the system should not touch.

Evals on the tasks users actually hit

An eval set is a collection of real inputs with known correct outcomes. Build it from last quarter's work: the invoices, tickets, or contracts the system will see, including the messy ones. Synthetic examples written by the build team tend to test what the build team already expected.

Run every change against the set. When production turns up a failure the set missed, add that case. Over a few months the eval set becomes the most accurate description of the work your company has.

The harness is the product

The model is a component you may swap next year. What stays is the harness around it: the evals, the contracts for each tool the system can call, traces of what it did and why, a cost ceiling, and the rule that stops it when the failure budget is spent. Exceptions go to a named person, not to a log nobody reads.

With a harness in place, changing the model or the prompt is a measured change. Without one, every change is a bet.

Ship the smaller system that passes

If the system does not pass its evals on the full scope, cut the scope until it does. Narrow the inputs it accepts. Remove actions it may take on its own. Send more cases to a person. A smaller system that passes evals beats a large one that demos well.

Then widen one slice at a time, and only when the evals on the next slice pass. That is how you ship early without arguing about it each time.

What this asks of the organization

  • One person who decides scope and go-live, and who can make that call in days.
  • The people who do the work today, reviewing exceptions and supplying eval cases.
  • A short weekly review where production failures become new eval cases and the team decides what to fix.

Where to start

Pick a high-volume workflow with written rules. Measure it as it runs today. Write the failure budget, build the eval set from real work, and scope the first release to what passes.

If your demo works and production does not, the Harness Engineer is the seat that builds the evals, traces, and failure budget around the model. If you need the whole feature built and your team trained to run it, the AI Pivot Squad structures first value for four weeks on a fixed milestone. Either way, book a scoping call and bring the workflow you have in mind.