The AI lab wrote the standard. Nobody wrote the safety case.

Here is an uncomfortable observation. A standard for AI agents that command manufacturing equipment is being drafted right now, and I have yet to see a name attached to it that I recognise from a shop floor. No 8D signatory. No PFMEA defended in front of a customer audit team. Nobody who has stood at a press at five in the morning waiting for the tooling technician. The metrology trade press carried the story this week: Anthropic, a foundation-model laboratory, is writing the rulebook for agents that will drive machines. GE Appliances is reported to be running some 800 agents already, and the conference agendas promise fully autonomous operations. Standards used to be written in blood. This one is being written on a release cycle.

what a safety case actually assumes

Every functional-safety certificate on your floor – ISO 13849 on the machinery side, IEC 61508 on the process side – rests on three assumptions. The logic is deterministic. The failure modes are enumerable. The arithmetic holds. Performance Level d, SIL 2, Category 3 architecture, MTTFd, diagnostic coverage: the whole calculation is bookkeeping over a finite list of ways the system can fail. You count the failures, rate their probability, design diagnostics to catch them, and the certificate states the safety function will behave exactly as specified. Every time.

A probabilistic agent voids those assumptions the moment it touches an actuator. Its logic is not a ladder diagram you can trace; it is a distribution. You cannot enumerate the failure modes of a system whose next output is a sample from a few hundred billion parameters and last quarter's training data. Confabulation – fluent, confident, wrong – is not a line item in an FMEA. It is a behaviour of the substrate.

Now the part that should keep a quality director awake. Your certificate covers the PLC. It does not cover the new, uncertified link issuing commands to it. The audit trail stops at the rack.

A safety case is a promise that every failure has already been counted; a foundation model is a machine for producing failures nobody counted.

standards written without the failure memory

ISO and IEC committees look slow from the outside. They are slow because they are doing something a sprint cannot: encoding a century of blood and audit findings. Guard-door interlocks with solenoid locking exist because hands were lost. ISO 13849 exists because practitioners found the earlier categories wanting – after incidents, over years. Every clause in those documents is an epitaph with a clause number.

Two decades of change control under AS9100/EN 9100 and IATF 16949 taught me what the shortcut costs. In my SNOP years a maintenance crew swapped a failing photoelectric sensor for a compatible one off the shelf – same make family, different response time. The guard's reaction distance shifted by centimetres. Nobody raised a change request. It was a Friday, and the line had to run. A customer audit later found the safety validation no longer matched the physical machine: major nonconformity, three days of revalidation, north of €30,000 in engineer-hours – all to protect a modification that had saved one afternoon. If a €40 sensor fitted outside change control can generate that, an over-the-air model update that silently changes the behaviour of whatever drives the machine is the same nonconformity with one new feature. Nobody even notices a change occurred.

We have run this film before. In the 2010s integrators bolted operational technology onto corporate networks, security was somebody else's assumption, and the industry spent a decade retrofitting segmentation and monitoring onto plants never designed for it. The trade press is already talking about shadow agents and identity for the agentic enterprise – same shape, arriving faster. If quality and functional safety stay out of the room while the agent standard is drafted, we run the retrofit again. This time the uncontrolled component has actuators on the other end.

how you validate what you cannot prove

Security solved this problem years ago; quality should steal the playbook. I hold a CEH among thirty-odd security certifications, and red-team practice never assumed determinism – penetration testing exists precisely because you cannot enumerate a complex system's failure modes. You attack it, on purpose, before someone else does. Treat the agent as an attack surface: prompt injection through job data, spoofed sensor values, drift, reward hacking. Tell it to hit a throughput target and watch what it optimises. Fuzz it. Pay engineers to break it. Document every way it breaks. That documentation, not a certificate, is your safety argument for a non-deterministic component.

Then architect around what you cannot prove. Three rules that hold today:

• Hard interlocks the agent cannot reach. Category 3/4 hardware safety – light curtains, guard locking, safe torque off – physically outside the agent's address space. The agent may request; the hardware decides.
• Consensus before actuation. I designed and built MultiPS, an orchestration platform running 63+ heterogeneous models in parallel with consensus synthesis. First-hand finding: every agent confabulates, fluently and with total confidence. Requiring independent models to agree before a command leaves the software layer is poor man's triple modular redundancy – one channel can lie, and the vote catches it.
• Every model update is a process change. Pin versions. Treat the weights as a controlled process parameter. Gate updates behind revalidation – the same discipline you apply to a tooling modification, because that is what it is.

The experts quoted this week saying fully autonomous factories remain some distance away are right about the distance and wrong about the comfort. That distance is the window – shorter than it looks – in which the people who know how machines fail can still shape the assumptions. The choice was never whether agents drive machines; procurement and the demo floors are settling that already. The choice is whether the safety case gets written by people who have counted the ways machines maim, or improvised later, in a retrofit, in blood, a decade late.