HEALTHCARE
Responsible AI Governance for Healthcare
Clinical and operational AI is already inside your organisation. The question your board now has to answer is whether it is governed: deployed within the medical-device, patient-data, and AI-governance rules, and safe for patients.
For CISOs, Data Protection Officers, Medical Directors, and Chief Medical Information Officers at health systems, medtech, and digital health businesses that need to deploy AI without crossing the rules.
THE CHALLENGE
AI Is Deployed Faster Than It Is Governed
Healthcare has the most to gain from AI and the least room to get it wrong. The same model that speeds a diagnosis can carry a safety risk, a data breach, or a regulatory breach that reaches the board. The rules are arriving in force, and the deployment is already ahead of them.
67%
Of organisations deploying AI hit significant unintended outcomes in year one
In a clinical setting an unintended outcome is not a marketing miss. It reaches patient safety, liability, and regulatory standing. The governance has to exist before the model goes live, not after the incident.
VerityAI Responsible AI buyer research, 2026
35M euro
Top EU AI Act penalty tier, or 7% of global turnover
Prohibited AI practices carry the 35M euro / 7% ceiling. High-risk breaches, the tier most clinical AI falls under, run to 15M euro or 3% of global turnover. The figures are large enough to land on the board agenda.
EU AI Act, penalty provisions (Articles 99 to 101)
1,000+
AI-enabled medical devices authorised by the FDA
The FDA has cleared over a thousand AI-enabled device software functions and now reviews them across the total product lifecycle, per its January 2025 draft guidance. The deployment wave is already here; the governance is racing to keep up.
FDA AI-Enabled Medical Device guidance and device list, 2025
2 Aug 2027
EU AI Act high-risk obligations apply to AI medical devices
High-risk duties for AI systems regulated as products under Annex I, which captures most AI medical devices, apply from this date. Conformity work, technical documentation, and human-oversight design take time. The runway is shorter than it looks.
EU AI Act, application dates for Annex I high-risk systems
THE OPPORTUNITY
Governance Is What Lets You Deploy, Not What Stops You
Treated as a checklist, compliance slows everything. Treated as a system, it is the thing that lets your clinicians use AI with confidence and lets your board approve it without holding its breath.
One framework, not three contradictory ones
The EU AI Act, the MDR, the FDA lifecycle approach, HIPAA, and GDPR overlap more than they conflict. We build a single governance system that answers to all of them, so your teams stop maintaining duplicate, drifting checklists.
Risk classification ends the guesswork
Once every AI system is mapped to its risk class, the board can see exactly what is high-risk, what needs conformity work, and what does not. Decisions get faster because the uncertainty is gone.
Documented governance is a competitive asset
Buyers, partners, and regulators increasingly ask how you govern AI before they engage. An audit-ready answer wins trust at the procurement stage, where it counts.
AI visibility, earned not gamed
When ChatGPT or Perplexity explains a condition or names a provider, the cited source gains patient trust at scale. We engineer that citation honestly, so your authority compounds without the dark patterns now drawing penalties across the AEO industry.
OUR APPROACH
Systems. Strategy. Execution.
The same three-level framework, recast for AI governance and compliance in a clinical-safety setting. The foundation is Responsible AI. Everything else is downstream of it.
SYSTEMS
AI Governance That Holds Up Clinically
We build the governance system that lets your organisation deploy clinical and operational AI inside the rules. One framework that answers to the EU AI Act, the MDR and IVDR, the FDA, HIPAA and GDPR at once, rather than a stack of contradictory checklists.
- -AI system inventory and risk classification (EU AI Act Article 6, Annex I and III)
- -Governance framework spanning MDR / IVDR, FDA lifecycle, HIPAA and GDPR Article 9
- -Human-oversight, transparency and record-keeping controls by design
- -Board and clinical-governance reporting a regulator can audit
STRATEGY
Clinical AI Risk and Compliance Strategy
We map where AI touches patients, patient data, and clinical decisions, then set the order of work by risk. The output is a decision a CISO, a DPO, a Medical Director, or a Chief Medical Information Officer can take to the board with confidence.
- -Clinical and operational AI risk review against current and incoming rules
- -Patient-data protection assessment (GDPR Article 9, HIPAA, special-category handling)
- -Vendor and model assessment for procured clinical AI
- -Board-ready Responsible AI roadmap with phased remediation
EXECUTION
Controls, Evidence, and AI Visibility Done Safely
When execution is needed, we engineer the controls and the evidence trail, and we engineer your visibility in AI answer engines without the dark patterns the rest of the AEO industry is already getting penalised for.
- -Conformity and technical documentation support for AI medical devices
- -Post-market monitoring and AI incident-response design
- -MedicalCondition, MedicalProcedure and Physician schema, implemented honestly
- -Answer Engine Optimisation that earns AI citation without manipulation
WHERE WE CREATE VALUE
Typical Healthcare Engagements
Illustrative scenarios reflecting the types of healthcare organisations we advise on Responsible AI.
HOSPITAL SYSTEM
Health System Rolling Out Clinical AI
Multiple AI tools in use across departments. Some procured, some built. No single inventory, no agreed risk classification, no clear owner for AI governance. The CISO and Medical Director cannot answer the board on exposure.
Systems-level engagement: a full AI inventory, risk classification against the EU AI Act and MDR, a governance framework, and clinical-governance reporting. We give the board one defensible view of where AI sits and how it is controlled.
MEDTECH
MedTech Building AI Into a Regulated Device
An AI feature inside a device that needs Notified Body assessment under the MDR. Strong clinical evidence, but the AI Act high-risk duties layered on top of MDR conformity are new and the August 2027 date is closing in.
Strategy-level engagement: risk classification under Article 6(1) and Annex I, a governance design that satisfies the MDR and the AI Act together, and technical documentation that holds up to both. Compliance engineered in, not bolted on at audit.
DIGITAL HEALTH
Digital Health Platform Training on Patient Data
A platform improving models on real patient records across the EU and US. Lawful basis under GDPR Article 9 is unclear, HIPAA obligations are partly met, and re-identification risk from model outputs has never been tested.
Patient-data protection review end to end, from ingestion to output, against GDPR Article 9 and HIPAA. We set the lawful basis, the minimisation controls, and the testing your DPO can sign.
BOARD ADVISORY
Board Needing an Independent AI Risk View
A board that has approved AI investment but has no independent read on the governance, the regulatory exposure, or the patient-safety controls behind it. Internal teams are close to the work and cannot mark their own homework.
Independent Responsible AI risk review, dated and documented, with a clear remediation order. The receipts a board, a regulator, or the MHRA expects, written before the conversation, not after the incident.
WHY US
Responsible AI Practitioners Who Know How AI Systems Work
We are a Responsible AI advisory, not a software vendor and not a platform. We have 27 years of experience governing technology for regulated businesses, and we work as the author of Ethical AI, AI Moats, and TRANSFORM. We understand clinical AI risk as a systems problem, where clinical governance, regulatory compliance, and patient-data protection must work as one. We build that system, and we document it so a board or a regulator can stand behind it.
Governance integrated, not bolted on
We design AI governance into the workflow, so clinical review and compliance run with deployment instead of stopping it. Safe and fast. Not one or the other.
Cross-jurisdiction by default
We work across the EU AI Act, the MDR and IVDR, the FDA lifecycle approach, HIPAA, GDPR Article 9, and the MHRA framework as it lands. One coherent system, not five siloed ones.
Receipts from day one
Every diagnosis and recommendation is dated and documented before it is discussed. The evidence trail a CISO, a DPO, or a board expects, written ahead of the incident, not after it.
FROM THE PUBLIC RECORD
What Ungoverned Clinical AI Actually Costs
The named cases below are matters of public record, cited to their sources. The third is a composite lesson from advisory work, with no client named.
Public record
A risk model that read cost as health
A widely used US population-health algorithm predicted care costs and treated them as a stand-in for need. Because less was historically spent on Black patients at the same level of illness, the model under-flagged them for extra care. The researchers found that correcting it would raise the share of Black patients referred for additional help from 17.7% to 46.5%.
The takeaway: a proxy that looks reasonable can encode a bias no one chose. We test the target variable, not just the accuracy score, before a clinical model goes near a patient.
Obermeyer, Powers, Vogeli and Mullainathan, Science, 2019, vol 366, pp 447 to 453.
Public record
A cancer AI trained on cases that weren't real
Internal documents obtained by STAT showed a high-profile oncology AI producing treatment recommendations described as unsafe and incorrect. The system had been trained on a small set of synthetic, hypothetical cases rather than real patient records, so its advice diverged from clinical guidelines.
The takeaway: training data is a governance question, not just a data-science one. We ask where the ground truth came from, and whether a clinician would recognise it, before a tool reaches the ward.
Ross and Swetlitz, STAT, 25 July 2018.
Composite lesson
The tool nobody owned
In work with a UK health-tech firm, without naming them, an AI triage feature had shipped with no single owner for its clinical risk. Product held the roadmap, engineering held the model, and no one held the question of what happened when it was wrong. The gap sat unnoticed until an audit surfaced it.
The takeaway: most clinical AI risk isn't a broken model, it's an unassigned one. We name the owner and the escalation path before deployment, so accountability exists before the incident does.
Composite of VerityAI advisory engagements. No client identified.
START HERE
Wherever You Are in the Decision
Three ways in, depending on where you are. Learn the rules, weigh up your options, or move straight to action.
Learn the rules
The Rules That Bind Healthcare AI
Start with the EU AI Act compliance checklist by industry, then read how GDPR rules apply when AI handles young people's data in youth-facing and paediatric services.
EU AI Act checklist →Compare your options
Weigh How You Audit and Protect Patient Data
Work through the AI compliance audit guide, see whether synthetic patient data lets you train within GDPR, and read where AI on patient records via new MCP architectures opens fresh risk.
AI compliance audit guide →Ready to act
Inventory Your AI, Then Talk to Us
Grab the free AI risk register template to map your clinical and operational AI, then bring it to a conversation about closing the gaps.
Risk register template →GO DEEPER
Responsible AI Knowledge Base
Practical guidance on governing, securing, and deploying AI in regulated settings, written for the people accountable for it.
AI Governance Knowledge Base
How to govern AI at board level: frameworks, controls, and accountability.
AI Privacy Knowledge Base
Patient and personal-data protection when AI sits on top of sensitive records.
AI Safety Knowledge Base
Designing AI that fails safely in high-stakes settings, including clinical care.
AI Risks Knowledge Base
The unintended outcomes that hit organisations deploying AI, and how to pre-empt them.
AI and Child Data Protection
GDPR compliance for AI handling young people's data, a sharp edge for paediatric and youth-facing health services.
AI Compliance Audit: Complete Guide
A step-by-step method for auditing AI systems against the rules that bind them.
GDPR, Synthetic Data and Privacy Rights
Whether synthetic patient data lets you train AI without breaching GDPR.
AI Agent Risk Assessment Framework
An executive framework for assessing the risk of agentic AI before deployment.
ASSESS YOUR READINESS
Start with a Self-Assessment
Free diagnostic tools for healthcare leaders. No signup required.
EU AI Act Readiness
Assess your clinical and operational AI against EU AI Act requirements. Find which systems are high-risk and what conformity work the August 2027 date demands.
Use Tool →AI Search Readiness Grader
Score how well AI answer engines can find, understand, and cite your organisation, and whether that visibility is built on signals that hold up.
Use Tool →Citation Readiness Checker
Analyse your clinical content against 10 AI citation signals. Identify what to improve so AI engines reference your expertise accurately, without manipulation.
Use Tool →BY JURISDICTION
UK, US and EU: The Rules Are Not the Same
The same clinical AI meets different rules depending on where it ships. Here's how the three regimes diverge, and where they overlap.
United Kingdom
MHRA-led, device-first
The MHRA regulates software and AI as a medical device, and sets the route to market for clinical AI. The ICO and UK GDPR govern patient data. Where it applies, the NHS Data Security and Protection Toolkit and the DTAC set the assurance an NHS buyer expects before deployment. There's no single UK AI statute, so the existing regulators apply their own rules to clinical AI.
United States
FDA and HIPAA, sector-led
The FDA reviews AI-enabled device software functions, including Software as a Medical Device, across the total product lifecycle. HIPAA governs protected health information held by covered entities and their business associates. The ONC sets health-IT certification and interoperability rules. There's no single federal AI statute, so these regimes apply alongside each other.
European Union
Codified and prescriptive
Under the EU AI Act, AI that is a medical device or its safety component is high-risk through the product-safety route in Article 6(1) and Annex I, sitting alongside the MDR and IVDR. GDPR Article 9 governs special-category health data. The obligations are written down, which means the audit trail has to match them.
VerityAI advises UK-first, and serves US and EU clients in English. We map each AI system to the regime that binds it, so you don't carry EU obligations on a UK-only product, or miss US rules on a launch into the States.
FAQ
Healthcare AI Compliance: Questions Boards and CISOs Ask
Direct answers on the rules that bind clinical and operational AI, written for the people accountable for getting it right.
When does clinical AI count as high-risk under the EU AI Act?
Under the EU AI Act, an AI system that is a safety component of a medical device, or that is itself a device requiring Notified Body assessment under the MDR (Regulation 2017/745) or IVDR, is classified high-risk through the product-safety route in Article 6(1) and Annex I. Most clinical decision-support and diagnostic AI lands here. High-risk obligations for these products apply from 2 August 2027. Penalties run to 35M euro or 7% of global turnover for prohibited practices, and 15M euro or 3% for high-risk breaches. We map each of your AI systems to its risk class before deployment.
How do EU AI Act duties sit alongside our existing MDR and FDA obligations?
They stack. A high-risk AI medical device must meet the MDR or IVDR and the AI Act, so risk management, data governance, record-keeping, transparency, human oversight and post-market surveillance now answer to both. In the US, the FDA reviews AI-enabled device software functions across the total product lifecycle, set out in its January 2025 draft guidance, and has authorised over 1,000 AI-enabled devices for marketing. We design one governance system that satisfies the overlapping regimes rather than three that contradict each other.
Is clinical AI a medical device under the EU AI Act?
It depends on what the AI does, not on the AI Act alone. If the software has a medical purpose set out in the MDR (Regulation 2017/745) or IVDR (Regulation 2017/746), it is already a medical device and needs the matching conformity assessment, usually with a Notified Body. The AI Act then classes that same system as high-risk through the product-safety route in Article 6(1) and Annex I, so the device rules and the AI rules apply together. Pure administrative or back-office AI with no medical purpose normally sits outside the device regime. We classify each system against the MDR or IVDR definition first, then map the AI Act duties that follow.
What changes for patient data when we put AI on top of it?
Health data is special-category data under GDPR Article 9, so processing is prohibited unless a specific condition applies. Training or running AI on patient records raises the bar on lawful basis, minimisation, and the risk of re-identification from model outputs. In the US, HIPAA governs protected health information held by covered entities and their business associates, and HIPAA compliance does not by itself satisfy GDPR. We audit the data path end to end, from ingestion to model output, against both.
Does HIPAA cover AI vendors?
It can, through the business associate route. If an AI vendor creates, receives, maintains, or transmits protected health information on behalf of a covered entity, it is a business associate and a HIPAA business associate agreement is required before that data moves. The agreement binds the vendor to the Privacy and Security Rules and to breach-notification duties. HIPAA does not reach a vendor that never touches protected health information, and it does not stand in for GDPR if EU patient data is involved. We check each vendor against the business associate test and the contract terms before any patient data is shared.
Does VerityAI build AI, or advise on it?
We are a Responsible AI advisory, not a software vendor or a platform. We review the AI your teams and vendors deploy, govern the risk, and document the controls a board, a regulator, or the MHRA can stand behind. Where your organisation also needs to be found and cited correctly by AI answer engines, we engineer that visibility without the dark patterns the wider AEO industry is already being penalised for. The advisory leads. AI visibility is the downstream commercial expression of the same expertise.
START HERE
Let's Talk About Governing AI in Your Organisation
A conversation about the AI your teams and vendors are deploying, the rules that now bind it, and where Responsible AI advisory turns clinical risk into controlled, board-ready governance.
Request a Consultation