This week a software agent quietly spent fifty thousand dollars of someone else's money on cloud compute before anybody pulled it off the job. The industry filed the story under cloud costs. I recognised the specimen: a positive-feedback loop with no stop condition, discovered when the credit-card bill arrived. I first met that failure mode on a shop floor, where it wore a part number. An unattended machining cell, refeeding itself since midnight; we found out at shift start, from the scrap bin. Nobody called that a FinOps incident.
The runaway is a failure mode, not an anecdote
Strip the novelty away and an agent that cannot stop is a process that cannot stop. Three ingredients, all familiar. A success metric with no terminal condition – keep optimising, keep provisioning, keep going. No tiers of authority, so the same credentials that let the thing read a log let it spend money. And an optimisation loop that feeds itself: the agent consumes resources to evaluate work that justifies consuming more resources. The agent was, in fairness, the most dedicated worker on the payroll. That was the problem.
Every quality engineer has met this animal. Overnight auto-refeed runs. Unattended cells. Presses cycling on a stuck sensor because an interlock was bypassed for changeover and never restored. We interlocked every one of them precisely because nobody was watching. This week the boards discovered the principle too – governance commentary everywhere, enterprise-risk framing, sober essays on third-party agents. All of it correct, and none of it asks the question a line designer asks without thinking. Where is the stop condition, and who owns it?
Jidoka for agents – the stop cord comes first
Jidoka rests on one insight: autonomy is what you grant, containment is what you engineer. At SNOP I built the quality department of a 900-plus employee greenfield plant from zero; before that came WITTE Automotive. Both plants ran stop-the-line authority down to the operators – a temporary worker on day three could halt a press. Vendors now ship software agents holding more authority than any operator in the plants I ran quality for, and no cord at all.
Autonomy is granted in a meeting; containment is engineered on the line.
Translated into agent terms, the andon spec is short. Hard-wired spend and action caps: a ceiling on money and on action count, enforced by infrastructure, not by the agent's own good behaviour. Release-authority tiers, so an agent may propose a work order or a parameter change, but releasing anything above a defined threshold is a human-signed gate.
Then the part vendors skip. Automatic escalation with a named owner – a person on the org chart before autonomy goes live, not after the first incident. Least privilege on the machine identity: the minimum permissions the task needs and nothing more, an operator badge that opens one cell, not the whole plant. Security people call that discipline. On the line we call it obvious.
I build these systems myself. MultiPS, the orchestration platform I designed and built, runs sixty-plus models in parallel and synthesises their outputs into a consensus. That consensus layer is verification, and it earns its keep. It is not a control. Verification tells you the answer was checked; a control tells you the loop stops. The first thing I wire is the halt condition, not the prompt, because a wrong answer that stops is a defect, while a wrong answer that continues is a recall.
When the agent touches the product
The bill is not the exposure. Fifty thousand dollars is recoverable. An escaped batch is not. A fleet action is not. A certification hold is not. The same architectures that burned the cloud budget are being sold today to release work orders, adjust process parameters and approve deviations – decisions that in my world sit inside PFMEA, APQP and 8D discipline, with signatures attached.
Hold agents to the evidentiary standard of a torque value. Every action leaves a record: what was authorised, by which authority tier, on what data, with which stop conditions armed. A defect caught at the cell costs €80 of scrap. The same defect caught at the customer costs €80,000 and a fortnight of containment. Caught across a fleet, in aerospace, it costs a certification hold and years. When an algorithm has adjusted a weld schedule, an auditor working to IATF 16949 or AS9100 will not accept "the agent decided", and neither will a court. The company owns the risk of its agent's decisions. That is not negotiable.
When a vendor pitches a third-party agent, ask for PPAP-grade evidence: documented authority limits, halt conditions, decision logging, a failure-mode analysis of its own runaway modes. If what arrives is a slide deck, they are selling you an unattended cell without interlocks.
Key takeaways
- Wire the stop condition before the prompt: hard spend caps, action caps and halt logic are the agent's andon cord.
- Tier the release authority – an agent may propose; above a threshold, a human signs.
- Apply least privilege to machine identities exactly as you do to operator badges: minimum permissions for the task, nothing more.
- Demand PPAP-grade evidence from third-party agents – authority limits, halt conditions, decision logs – and reject the slide deck.
That fifty-thousand-dollar invoice was tuition, and the tuition was cheap. The next runaway will not send one. It will exercise release authority nobody capped, on a batch nobody contained, in a system nobody can interrogate after the fact. The line solved this seventy years ago with a length of cord and the discipline to pull it. Design the stop cord before you grant the autonomy – before the autonomy touches the product, not after the invoice arrives.