Physical AI Safety: The Board's Guide to Assurance

If your AI moves a robot, drives a vehicle, or controls a machine, a software bug becomes a physical event. Boards now own that risk. Before any embodied or agentic system goes live, you need a documented safety case: structured evidence, traced to a recognised standard, that the system is acceptably safe in the real world, not just in the lab. Most organisations test these systems for performance and skip the assurance step. That gap is where the lawsuits, recalls, and regulator fines sit.

A digital model that hallucinates produces a wrong answer. An embodied system that fails produces a collision. The stakes change completely once AI crosses from screen to hardware, and so does the governance question a board has to answer: can we show, on paper, that we did the work to make this safe?

Why physical AI is now a board-level risk

Cyber-physical AI covers anything where a model's output drives action in the world: warehouse robots, autonomous and assisted-driving vehicles, surgical and clinical devices, drones, and the new wave of agentic systems that book, buy, and operate without a human in the loop.

Three things make this a board problem, not just an engineering one.

The cost of failure is physical and often irreversible. You can roll back a bad database write. You cannot un-injure a person.

The liability is direct. Regulators and courts treat physical harm from an automated system as a product-safety and corporate-accountability matter, not a technical footnote.

The evidence requirement is rising fast. New standards and laws expect you to hold a documented safety argument before deployment, and to keep monitoring after. No argument on file is itself a finding.

The Cruise case shows what happens when this breaks down. In October 2023, a Cruise robotaxi in San Francisco ran over a pedestrian who had been thrown into its path, then dragged her about 20 feet while trying to pull over, because its detection system did not register a person underneath it (NHTSA). California suspended the company's driverless permit within weeks. Cruise later admitted filing a false report to federal investigators and agreed to a $500,000 criminal fine, with a separate $1.5M NHTSA penalty for failing to fully report the crash (US Department of Justice). The technical failure was an edge case. The corporate failure was governance.

What's different about testing AI that acts in the world?

Standard AI evaluation checks accuracy on a dataset. Physical AI assurance has to deal with the messy interface between a model and reality. Three problems sit at the centre of it.

Problem	What it means	Why standard testing misses it
The reality gap	A system can be near-perfect in simulation and unsafe on the road. Real sensors are noisy, environments vary without limit, and hardware drifts over time.	Lab benchmarks reward simulation performance, which doesn't transfer cleanly.
Physical adversaries	Inputs can be manipulated in the physical world: stickers that fool perception, interference that blinds a sensor, objects placed to trigger a wrong action.	Digital security testing checks the network, not a tampered stop sign.
Irreversible action	Many physical actions can't be undone, so a single rare failure can be catastrophic rather than annoying.	Aggregate accuracy metrics hide the rare, high-consequence tail.

The practical takeaway: a high benchmark score is not a safety case. It's one input to one. Treating it as proof of safety is the most common, and most expensive, mistake we see.

Which standards should govern this work?

You don't need to invent the framework. A set of recognised standards now defines what good looks like, and aligning to them is both the safer engineering choice and the stronger legal position. These are the ones that matter for a board.

Standard	Covers	Why it matters to you
NIST AI Risk Management Framework (AI RMF 1.0)	Govern, Map, Measure, Manage. The cross-sector backbone for AI risk.	The common language regulators, insurers, and buyers increasingly expect. Voluntary, but the default reference.
ISO/IEC 42001:2023	The first certifiable AI management system standard: governance, risk, impact assessment, supplier oversight.	Certification you can show a board, an auditor, or a customer. Organisation-wide, not per-model.
UL 4600	Safety case method for fully autonomous products: vehicles, robots, drones operating without human oversight.	Defines the documented safety argument itself. Goal-based, so it fits novel systems.
ISO/PAS 8800:2024	Safety and AI specifically for road vehicles. Extends the existing automotive safety standards to cover AI behaviour.	If you build or buy automotive AI, this is now the industry reference for data quality, reliability under varied conditions, and the assurance argument.
ISO 10218-1/2:2025	Industrial and collaborative robot safety, now folding in the human-robot collaboration requirements previously in ISO/TS 15066.	If robots work near people, the 2025 revision shifts focus from safe hardware to the safety of the actual application.

Across all of them the common thread is the same. The deliverable isn't a passing test run. It's a structured, evidence-backed argument that the system is acceptably safe, written down before launch and maintainable after.

What does a real assurance programme look like?

A credible programme runs in stages, gaining real-world exposure only as evidence accumulates. Skipping stages is how the reality gap bites.

1. Component simulation. Test each subsystem in a virtual environment.
2. Integrated simulation. Run the whole system against hard, automatically generated edge cases.
3. Controlled physical testing. Move to a closed, real-world setting where failures are contained.
4. Monitored limited deployment. Restricted real operation under close human supervision.
5. Staged rollout. Expand the operating domain only as the evidence supports it.

Hardware-in-the-loop and digital-twin methods bridge the stages: real hardware against simulated environments, or a virtual replica of a specific deployment site fed with live data. They let you probe dangerous edge cases safely. They don't replace real-world validation, because the reality gap is exactly the thing simulation can't fully close.

The testing inside those stages has to target the failure modes standard QA ignores: sensor degradation and interference, adversarial physical inputs, extreme weather and lighting, emergency-stop and fail-safe behaviour under component failure, and graceful degradation when a sensor drops out. Redundant sensing, hard safety limits enforced independently of the main model, and runtime monitoring are the engineering controls that make the safety case defensible.

How does the law treat physical AI failures?

The regulatory floor is rising, and it's no longer optional reading for a board.

The EU AI Act classes AI embedded in regulated products, including vehicles, medical devices, and machinery, as high-risk. Providers must run a documented risk management system, maintain technical documentation and automatic logging, ensure human oversight, and pass a conformity assessment before the CE mark goes on. Under the May 2026 Digital Omnibus agreement, the deadlines moved: use-case high-risk obligations (Annex III) shifted to December 2027, and product-embedded high-risk obligations (Annex I) to August 2028 (European Commission). More runway, same destination. The work to be ready takes longer than the time you think you have.

Beyond the AI Act, product-liability and duty-of-care law already applies. If a system you deployed causes foreseeable harm, the question in any investigation is simple: what did you do to prevent it, and can you prove it? A documented safety case, traced to a recognised standard, is the answer. Its absence is the exposure.

Frequently asked questions

What is an AI safety case and why does my board need one?

A safety case is a structured argument, backed by documented evidence, that a system is acceptably safe to deploy. UL 4600 made it the central method for autonomous products. Your board needs one because it's the artefact a regulator, court, insurer, or acquirer will ask to see. Without it, you're asserting safety. With it, you can demonstrate it.

Isn't passing our test benchmarks enough to deploy safely?

No. Benchmarks measure average performance on known cases. Physical safety failures live in the rare, high-consequence tail and in the reality gap between simulation and the world. A benchmark is one input to a safety case, not a substitute for it. The Cruise dragging incident followed extensive testing.

We buy our AI from a vendor. Are we still on the hook?

Yes. Under ISO/IEC 42001 and the EU AI Act, the organisation deploying a system carries oversight and impact-assessment duties regardless of who built it. Vendor due diligence, contractual evidence requirements, and your own monitoring are part of the safety case. "The supplier handles it" is not a defence that survives an investigation. This is exactly where independent assessment earns its keep.

Which standard should we start with?

Start with the NIST AI RMF for shared language and ISO/IEC 42001 for a certifiable management system. Then add the domain standard that fits your hardware: UL 4600 for autonomous products, ISO/PAS 8800 for road vehicles, ISO 10218 for robots near people.

The bottom line

Here's the opinion. The organisations that win with physical AI won't be the ones with the cleverest models. They'll be the ones that can prove their systems are safe before a regulator, a court, or a customer asks. Assurance is becoming the licence to operate, and the gap between firms that treat it as core governance and firms that bolt it on after an incident is going to be brutal.

Physical-world failures often combine with digital exploitation techniques into multi-domain attack paths, and assessing them needs domain-specific expertise in robotics, sensors, and control systems. As agentic AI starts taking real-world actions, the same discipline applies to autonomous agent risk and to building trust in autonomous deployment. The board's job is to make sure the safety case exists, traces to a standard, and holds up before launch, not after.

This is the kind of work our AI red teaming and adversarial testing handles.