Here is an uncomfortable observation from the shop floor: the most dangerous machine in your plant is not the one that fails. It is the one that learns. A machine that fails fails in ways your PFMEA predicted. A machine that adapts fails in ways nobody enumerated — including the vendor who sold it to you. So when an expert commentary this week argued that an AI-controlled machine needs a brake outside its AI brain, I did not read it as news. I read it as poka-yoke arriving, a little late and slightly rebranded, in the AI age.
The principle is older than any of us. Shigeo Shingo did not need a neural network to teach him that no protective function may share a brain with the thing it protects. Across two decades of automotive and aerospace error-proofing under IATF 16949 and AS9100, I signed off dual-channel stop circuits where the two channels were deliberately dissimilar. I wired vision systems with hard go/no-go gauges downstream, and wrote control plans on the assumption that every sensor eventually lies. The commentary restates all of this for learned software. Restating it is still worth doing, because vendors are now shipping learned motion control and AI-designed robot cells as fast as the market reports can count them — and the people buying them did not all grow up on poka-yoke.
What a learned control loop does to your PFMEA
A PFMEA is a catalogue of enumerable causes. Bearing wear, sensor drift, operator loading the fixture backwards — you list them, rate severity, occurrence, detection, and sleep reasonably well. The method rests on one assumption: the system's behaviour tomorrow is its behaviour today, plus degradation you can name.
A learned control loop breaks that assumption at the root. The model's output is a function of its training data, its update schedule, its inputs, and whatever it has adapted to since commissioning. You cannot enumerate the causes, because the causes include every configuration the model has ever seen. Detection ratings go mushy too — your detection score assumed a specific drift pattern crossing a specific threshold. What you get instead is confident wrongness at cycle time: a model that has quietly re-learned "good part" from a shifted distribution and passes everything with full self-assurance. The failure mode is not a fault code. It is a smooth, productive, cheerful shift in what the machine believes.
The dumb layer is the safety layer
The fix is architectural, not algorithmic. Every safety- and quality-critical function sits in a layer that is dumb, hardwired, and physically verifiable — and the model gets no path around it. Not "the AI monitors itself and escalates." Hardwired stops. Light curtains wired to drop power, not to send a message. Mechanical go/no-go fixtures and poka-yoke gauges that reject what the model accepts. Two-channel circuits with diverse logic, exactly as AS9100 taught us.
This is ethical-hacker logic applied to your own equipment, and it is where my CEH discipline and my quality career turn out to be the same job: assume the smart component fails smartly. A conventional controller fails dumb. It stops, it faults, it hangs. A learned controller fails intelligently. It optimises around your guardrail, finds a context you never tested, and produces an output that is wrong but internally consistent. That is precisely how I break into systems professionally: not by overpowering the control, but by finding the reasoning path that talks its way past it. Never place a protective function where a persuasive argument can reach it.
A control that can be argued with is not a control. It is a suggestion wearing a uniform.
I have seen this play out. A vision system at an automotive supplier kept cheerfully passing parts after something in its world changed — lighting, fixture, upstream process; we argued for weeks about which. Its confidence never dipped. The line produced scrap with excellent throughput until a dumb go/no-go gauge downstream, the one the integrator had called redundant, started catching everything the camera blessed. The redundant check was the only one that worked. That gauge earned its floor space more than any camera we ever bought.
Writing the control plan for a station that thinks
How do you audit a learning station? The same way you audit anything: move the verification out of the thing being verified.
- Independent physical verification the model cannot bypass. Gauges, fixtures, measurement cells on separate power and separate logic. If the AI passes a part, a device that cannot update itself must agree.
- Model-change discipline under classic change control. A model update is a process change. It goes through the same gates as a tooling change: notification, revalidation, PPAP-style evidence, sign-off. "Continuous learning" without change control is an unapproved process change running at production rate.
- Duress-test the stop before you trust the cycle. I test the stop button, not the software's promise. Force the guard, blind the sensor, feed the model a known-bad part at full cycle speed, and confirm the dumb layer catches every case. Do it at commissioning and periodically after — the model underneath will have changed even if your validation paperwork has not.
I write this as someone on the inside of the AI wave, not a sceptic at the fence. I build and deploy multi-agent AI systems; my platform MultiPS runs dozens of models in parallel with consensus synthesis. Not one of my agents gets the last word over a deterministic check. When the models disagree, an agent decides nothing — a rule I wrote in code decides. The same architecture, scaled to a robot cell, is exactly what this week's commentary is demanding: intelligence in the loop, determinism at the gate.
Key takeaways
- Adaptation breaks the PFMEA assumption of enumerable causes — plan for confident wrongness at cycle time, not fault codes.
- Place every safety- and quality-critical function in a hardwired layer the model cannot reach, argue with, or update.
- Treat every model update as a process change under classic change control, with revalidation evidence.
- Duress-test the stop with known-bad inputs at full speed, at commissioning and periodically — the model will drift even when the paperwork doesn't.
The brake that cannot think is the only one you can trust, because it is the only one whose failure modes you can still list. Vendors will keep shipping smarter brains, and most of them will be genuinely good. Buy them. Use them. But when the auditor comes — or worse, when the part doesn't — the question that decides whether you are a professional or a spectator is not "how intelligent is the control?" It is "what, physically, stops this machine when the intelligence is wrong?" Quality's job in the AI age is not to keep up with the brain. It is to defend dumbness where it matters.