Healthcare AI Governance

FlowRidge

Industry Knowledge Base

Regulatory landscape, sector risk overlays, compliance checklist, and COMPEL methodology mappings for AI systems used in clinical and administrative healthcare operations. Current as of April 2026.

Regulatory Requirements

Risk Overlays

Compliance Items

COMPEL Stage Maps

Regulatory Landscape

The US and EU healthcare AI regulatory environment combines federal device law (FDA), privacy law (HIPAA), sector guidance (HHS), state-level AI statutes, and the EU AI Act. Each authority below sets a distinct set of obligations for AI systems that create, receive, maintain, or influence patient care.

FDA guidance Effective 2021-01-12

Artificial Intelligence and Machine Learning in Software as a Medical Device

FDA framework for AI/ML-based Software as a Medical Device (SaMD), establishing premarket review expectations for AI-enabled medical devices including locked and adaptive algorithms.

Key provisions, impacted AI types, and compliance actions

Key Provisions

- Good Machine Learning Practice (GMLP) principles for AI/ML device development
- Total Product Lifecycle (TPLC) approach for AI/ML-based SaMD
- Predetermined Change Control Plan (PCCP) framework for adaptive algorithms
- Clinical evaluation expectations for AI/ML performance
- Transparency requirements for AI-assisted clinical decisions

Impacted AI Types

- Diagnostic imaging AI
- Clinical decision support systems
- Patient monitoring algorithms
- Predictive analytics for patient outcomes
- Automated screening tools

Compliance Actions

- Classify AI system under FDA SaMD risk framework (Class I/II/III)
- Develop PCCP for any algorithm with adaptive capabilities
- Document GMLP adherence throughout development lifecycle
- Establish clinical validation protocols with representative patient populations
- Implement post-market surveillance for AI performance monitoring

FDA final rule Effective 2025-12-19

Predetermined Change Control Plans for AI/ML-Enabled Devices

Final guidance on Predetermined Change Control Plans (PCCPs) enabling manufacturers to describe planned modifications to AI/ML-enabled devices and the methodology for implementing those changes without requiring new premarket submissions.

Key provisions, impacted AI types, and compliance actions

Key Provisions

- Description of Modifications (SaMD Pre-Specifications): specific planned changes to algorithm, performance, or inputs
- Algorithm Change Protocol (ACP): methods for developing, validating, and implementing each modification
- Modification verification and validation plan requirements
- Labeling update requirements for PCCP-implemented changes
- Post-implementation performance monitoring expectations

Impacted AI Types

- Continuously learning AI/ML medical devices
- AI systems with periodic model retraining
- Adaptive clinical decision support
- AI devices with expanding indication scope

Compliance Actions

- Draft PCCP defining all anticipated algorithm modifications
- Establish Algorithm Change Protocol with validation methodology
- Create modification verification and validation test plans
- Design post-implementation monitoring dashboards
- Document labeling update procedures for each change category

HHS effective Effective 2026-04-01

HHS AI Strategy and Minimum Risk Management Practices

HHS five-pillar AI strategy establishing minimum risk management practices for AI use in healthcare. Requires covered entities to implement risk assessments, bias testing, human oversight, and transparency measures for AI systems used in clinical and administrative healthcare decisions.

Key provisions, impacted AI types, and compliance actions

Key Provisions

- Pillar 1: Safety — AI systems must demonstrate clinical safety through validated testing before deployment
- Pillar 2: Trustworthiness — Transparency requirements for AI-assisted decisions affecting patient care
- Pillar 3: Equity — Mandatory bias testing across demographic groups for clinical AI applications
- Pillar 4: Privacy — Enhanced data protection requirements for AI training data and inference outputs
- Pillar 5: Accountability — Human oversight mechanisms and incident reporting for AI-related adverse events
- Minimum risk management practices for all AI-enabled healthcare activities
- Annual AI inventory and risk assessment requirements for covered entities

Impacted AI Types

- All AI systems used in clinical decision-making
- Administrative AI (prior authorization, claims processing)
- Patient-facing AI (chatbots, symptom checkers)
- Population health analytics
- Resource allocation algorithms

Compliance Actions

- Conduct initial AI inventory across all departments
- Implement risk assessment framework aligned with HHS five pillars
- Establish bias testing protocols for demographic equity analysis
- Create human oversight procedures for AI-assisted clinical decisions
- Design incident reporting workflow for AI-related adverse events
- Develop annual AI governance review process

State Legislatures (47 states) effective

State Healthcare AI Legislation

As of early 2026, 47 states have introduced 250+ AI-related bills, with 33 laws enacted across 21 states. Healthcare-specific provisions address prior authorization automation, clinical decision support transparency, patient notification requirements, and algorithmic bias in health equity.

Key provisions, impacted AI types, and compliance actions

Key Provisions

- Prior authorization AI: Multiple states require human review of AI-generated denials (e.g., California SB 1120, New York AB 1309)
- Clinical decision support: Transparency requirements for AI recommendations to clinicians
- Patient notification: Requirements to inform patients when AI is used in their care (e.g., Colorado SB 24-205)
- Health equity: Bias auditing requirements for AI used in coverage and treatment decisions
- AI inventory: State-level requirements for healthcare entities to maintain AI system registries
- Consent requirements: Patient consent for AI-assisted diagnosis in certain jurisdictions

Impacted AI Types

- Prior authorization automation systems
- Clinical decision support tools
- Patient-facing diagnostic AI
- Coverage determination algorithms
- Health equity assessment tools

Compliance Actions

- Map operational footprint to applicable state-level AI requirements
- Implement patient notification mechanisms for AI-assisted care decisions
- Establish human review workflows for AI-generated prior authorization decisions
- Create jurisdiction-specific compliance matrices
- Monitor legislative developments across operating states

HHS OCR effective Effective 1996-08-21

HIPAA and AI Systems

Application of HIPAA Privacy Rule, Security Rule, and Breach Notification Rule to AI systems that create, receive, maintain, or transmit protected health information (PHI). AI-specific considerations include training data governance, model inference outputs as PHI, and minimum necessary principle for AI feature access.

Key provisions, impacted AI types, and compliance actions

Key Provisions

- PHI in AI training data requires Business Associate Agreements (BAAs) with AI vendors
- Model outputs containing individually identifiable health information constitute PHI
- Minimum necessary principle applies to AI feature engineering and data access
- De-identification standards (Safe Harbor / Expert Determination) for AI training datasets
- Security Rule technical safeguards must extend to AI model storage and inference endpoints
- Breach notification requirements cover AI system data exposure events

Impacted AI Types

- All AI systems processing PHI
- AI models trained on patient data
- Natural language processing on clinical notes
- AI inference endpoints in clinical workflows
- Cloud-hosted AI services with PHI access

Compliance Actions

- Audit all AI systems for PHI touchpoints (training, inference, storage)
- Execute BAAs with all AI vendors processing PHI
- Implement de-identification protocols for AI training data pipelines
- Apply minimum necessary principle to AI feature engineering
- Extend security risk assessments to AI infrastructure components
- Include AI systems in breach incident response procedures

FDA guidance Effective 2022-09-28

Clinical Decision Support Software Classification

FDA classification criteria distinguishing regulated Clinical Decision Support (CDS) software from non-regulated CDS under the 21st Century Cures Act. Software meeting all four Cures Act criteria is exempt from FDA device regulation; software that does not meet all four criteria (e.g., clinician cannot independently review the basis) is regulated as a medical device.

Key provisions, impacted AI types, and compliance actions

Key Provisions

- Four Cures Act exemption criteria: (1) not intended for acquiring/analyzing medical images or signals, (2) intended for clinician use, (3) intended to display/make available basis for recommendations, (4) clinician can independently review the basis
- AI systems that replace clinical judgment (rather than support it) do not qualify for exemption
- Autonomous AI diagnostic systems are regulated as medical devices regardless of CDS criteria
- Locked vs. adaptive algorithm distinction affects regulatory pathway
- Risk-based framework for CDS that falls outside Cures Act exemption

Impacted AI Types

- Clinical decision support software
- AI-assisted diagnostic tools
- Treatment recommendation engines
- Risk stratification algorithms
- Care pathway optimization systems

Compliance Actions

- Evaluate each AI system against four Cures Act CDS exemption criteria
- Classify non-exempt CDS under appropriate FDA device classification
- Document clinical workflow showing clinician independent review capability
- Establish regulatory strategy for each AI system classification outcome
- Maintain classification decision documentation for regulatory inspections

European Commission effective Effective 2025-08-02

EU AI Act Healthcare Provisions

EU AI Act (Regulation 2024/1689) high-risk classification for healthcare AI systems under Annex III, Section 5 (access to essential services) and alignment with Medical Device Regulation (MDR 2017/745). Healthcare AI systems classified as high-risk face mandatory conformity assessment, CE marking, and post-market monitoring.

Key provisions, impacted AI types, and compliance actions

Key Provisions

- Annex III Section 5(a): AI systems intended to evaluate eligibility for public healthcare services
- Annex III Section 5(b): AI systems used to evaluate creditworthiness for healthcare services
- Healthcare AI safety components within MDR scope automatically classified as high-risk
- Mandatory conformity assessment by notified bodies for high-risk healthcare AI
- Risk management system requirements aligned with Article 9
- Data governance requirements for healthcare AI training datasets (Article 10)
- Technical documentation and transparency obligations (Articles 11-13)
- Human oversight requirements for clinical AI deployment (Article 14)

Impacted AI Types

- AI systems for hospital admission decisions
- Diagnostic AI classified as medical devices
- Treatment recommendation algorithms
- Patient triage and prioritization systems
- Healthcare resource allocation AI

Compliance Actions

- Classify healthcare AI systems under EU AI Act risk categories
- Align with MDR requirements for AI-enabled medical devices
- Implement Article 9 risk management system for high-risk healthcare AI
- Establish data governance protocols per Article 10 requirements
- Create technical documentation packages for conformity assessment
- Design human oversight mechanisms per Article 14

Sector Risk Overlays

Healthcare-specific risk categories layered over general AI governance. Each overlay describes the class of use, its EU AI Act classification, example systems, required mitigations, and which COMPEL stages own the associated controls.

Patient Admission and Triage

high risk

AI systems that influence hospital admission decisions, emergency triage, or patient prioritization carry high risk due to direct impact on patient safety and health equity.

Example Systems

- Emergency department triage algorithms
- Hospital bed allocation optimization
- Patient acuity scoring systems
- Surgical scheduling prioritization

Mitigation Requirements

- Clinician override capability at every decision point
- Bias testing across demographic groups (race, age, sex, insurance status)
- Real-time performance monitoring with alert thresholds
- Patient outcome tracking linked to AI recommendations
- Transparent scoring criteria accessible to clinicians

COMPEL stages: Calibrate Model Evaluate

Diagnostic AI

high risk

AI systems that assist or automate clinical diagnosis, including medical imaging analysis, pathology interpretation, and differential diagnosis generation.

Example Systems

- Radiology image analysis (X-ray, CT, MRI)
- Pathology slide interpretation
- Dermatology lesion classification
- Retinal screening for diabetic retinopathy
- ECG interpretation algorithms

Mitigation Requirements

- Clinical validation with representative patient populations
- Performance benchmarking against board-certified specialists
- Subgroup analysis across demographics and comorbidities
- FDA clearance or approval (if SaMD classification applies)
- Continuous calibration monitoring with drift detection
- Clinician training on AI limitations and error modes

COMPEL stages: Calibrate Model Produce Evaluate

Treatment Recommendations

high risk

AI systems that recommend treatment plans, medication dosing, or therapeutic interventions where incorrect recommendations could lead to patient harm.

Example Systems

- Medication dosing optimization
- Chemotherapy regimen recommendation
- Antibiotic stewardship algorithms
- Surgical approach recommendation
- Rehabilitation protocol personalization

Mitigation Requirements

- Evidence-based validation against clinical guidelines
- Drug interaction and contraindication checking integration
- Clinician review and approval before patient-facing output
- Adverse event monitoring and reporting pipeline
- Regular model revalidation against updated clinical evidence

COMPEL stages: Model Produce Evaluate

Prior Authorization and Claims

high risk

AI systems automating prior authorization decisions, claims adjudication, or coverage determinations where errors can delay or deny patient care.

Example Systems

- Automated prior authorization review
- Claims denial prediction and processing
- Medical necessity determination
- Coverage eligibility assessment
- Appeal outcome prediction

Mitigation Requirements

- Human review of all AI-generated denials (required by multiple state laws)
- Transparent denial reasoning accessible to providers and patients
- Appeal process with expedited human review pathway
- Bias auditing for disparate impact on protected populations
- Turnaround time monitoring to prevent delays in care

COMPEL stages: Calibrate Model Evaluate

Population Health Analytics

limited risk

AI systems analyzing population-level health data for risk stratification, disease surveillance, or public health resource allocation.

Example Systems

- Population risk stratification
- Disease outbreak prediction
- Social determinants of health modeling
- Care gap identification
- Chronic disease management targeting

Mitigation Requirements

- De-identification verification for all population datasets
- Equity impact assessment for resource allocation recommendations
- Transparency in risk scoring methodology
- Regular recalibration against updated population data
- IRB oversight for research-oriented population analytics

COMPEL stages: Calibrate Evaluate

Patient-Facing AI

limited risk

AI systems that interact directly with patients, including chatbots, symptom checkers, virtual health assistants, and patient education tools.

Example Systems

- AI-powered symptom checkers
- Virtual health assistant chatbots
- Patient education content personalization
- Mental health screening tools
- Medication adherence reminders with AI personalization

Mitigation Requirements

- Clear disclosure that patient is interacting with AI
- Escalation pathways to human clinicians for urgent concerns
- Clinical safety guardrails preventing dangerous recommendations
- Accessibility compliance (ADA, WCAG 2.2)
- Patient consent and opt-out mechanisms
- Content accuracy review by licensed clinicians

COMPEL stages: Model Produce

Compliance Checklist

Prioritized obligations grouped by domain. Each item identifies the regulatory source, the COMPEL stage that owns the work, and the evidence artifacts a healthcare organization should maintain.

Inventory & Classification (2)

hcc-001 critical Calibrate

Maintain comprehensive AI system inventory

Catalog all AI systems used in clinical and administrative healthcare operations, including vendor-provided and internally developed systems.

Regulatory source: HHS AI Strategy Pillar 5; State AI inventory requirements

Evidence required: AI system registry with classification; Vendor AI disclosure forms; Annual inventory review documentation

hcc-002 critical Calibrate

Classify AI systems under FDA SaMD framework

Evaluate each AI system against FDA Software as Medical Device classification criteria and Cures Act CDS exemption criteria.

Regulatory source: FDA SaMD Guidance; 21st Century Cures Act

Evidence required: SaMD classification decisions; Cures Act exemption analysis; Regulatory pathway documentation

Risk Assessment (1)

hcc-003 critical Calibrate

Conduct AI-specific risk assessments

Perform risk assessments for all healthcare AI systems addressing clinical safety, bias, privacy, and operational risks.

Regulatory source: HHS AI Strategy Pillar 1; EU AI Act Article 9

Evidence required: Risk assessment reports per AI system; Risk mitigation plans; Residual risk acceptance documentation

Bias & Equity (1)

hcc-004 critical Evaluate

Implement demographic bias testing

Test all clinical AI systems for performance disparities across race, ethnicity, age, sex, socioeconomic status, and insurance type.

Regulatory source: HHS AI Strategy Pillar 3; State health equity laws

Evidence required: Bias testing methodology documentation; Subgroup performance analysis reports; Remediation plans for identified disparities

Human Oversight (1)

hcc-005 critical Produce

Establish clinician oversight mechanisms

Ensure clinicians can review, override, and challenge AI recommendations in clinical workflows.

Regulatory source: HHS AI Strategy Pillar 5; EU AI Act Article 14

Evidence required: Override workflow documentation; Clinician training records; Override rate monitoring reports

Privacy & Data Governance (1)

hcc-006 critical Organize

HIPAA compliance for AI data pipelines

Ensure all AI training data, inference inputs, and model outputs comply with HIPAA Privacy and Security Rules.

Regulatory source: HIPAA Privacy Rule; HIPAA Security Rule

Evidence required: PHI data flow maps for AI systems; Business Associate Agreements with AI vendors; De-identification verification reports

Transparency (2)

hcc-007 high Produce

Patient notification for AI-assisted care

Inform patients when AI systems are used in their clinical care decisions, as required by applicable state laws.

Regulatory source: State patient notification laws (CO, CA, NY)

Evidence required: Patient notification templates; Consent form updates; Notification delivery audit logs

hcc-008 high Produce

Clinician-facing AI transparency

Provide clinicians with accessible information about AI system methodology, limitations, and known failure modes.

Regulatory source: FDA CDS Guidance; HHS AI Strategy Pillar 2

Evidence required: AI system user guides for clinicians; Known limitations documentation; Training completion records

Validation & Monitoring (2)

hcc-009 critical Model

Clinical validation before deployment

Validate AI system performance against clinical gold standards with representative patient populations before production use.

Regulatory source: FDA GMLP Principles; HHS AI Strategy Pillar 1

Evidence required: Clinical validation study reports; Performance benchmark comparisons; Subpopulation analysis results

hcc-010 critical Evaluate

Post-deployment performance monitoring

Continuously monitor AI system performance in production, including accuracy drift, fairness metrics, and adverse event detection.

Regulatory source: FDA PCCP Guidance; HHS AI Strategy Pillar 1

Evidence required: Performance monitoring dashboards; Drift detection alerts; Adverse event reports

Prior Authorization (1)

hcc-011 high Produce

Human review of AI-generated denials

Ensure all AI-generated prior authorization denials receive human clinical review before finalization.

Regulatory source: State prior authorization laws (CA SB 1120, NY AB 1309)

Evidence required: Human review workflow documentation; Review completion audit trail; Denial overturn rate reports

Incident Response (1)

hcc-012 high Evaluate

AI adverse event reporting

Establish incident response procedures for AI-related adverse events, including FDA MedWatch reporting for device-classified AI.

Regulatory source: HHS AI Strategy Pillar 5; FDA MDR requirements

Evidence required: AI incident response procedures; MedWatch reporting templates; Incident log and follow-up documentation

Change Management (1)

hcc-013 high Learn

AI model update governance

Govern all AI model updates through formal change control processes, including PCCP compliance for FDA-regulated systems.

Regulatory source: FDA PCCP Guidance; HHS AI Strategy Pillar 5

Evidence required: Change control board records; PCCP documentation (if applicable); Validation results for model updates

Vendor Management (1)

hcc-014 high Organize

AI vendor due diligence

Conduct due diligence on all AI vendors, including data practices, model transparency, bias testing, and HIPAA compliance.

Regulatory source: HIPAA BAA requirements; HHS AI Strategy

Evidence required: Vendor AI assessment questionnaires; BAA execution records; Vendor performance review reports

Governance Structure (1)

hcc-015 high Organize

Healthcare AI governance committee

Establish a multidisciplinary AI governance committee with clinical, technical, legal, compliance, and patient safety representation.

Regulatory source: HHS AI Strategy Pillar 5; Best practice

Evidence required: Committee charter; Membership roster with roles; Meeting minutes and decision logs

Training & Competency (1)

hcc-016 medium Learn

Healthcare AI literacy program

Train clinical and administrative staff on AI capabilities, limitations, appropriate use, and governance requirements.

Regulatory source: HHS AI Strategy Pillar 2; JCAHO standards

Evidence required: Training curriculum documentation; Completion rates by department; Competency assessment results

Maturity Assessment Criteria

Five-level maturity scale for healthcare AI governance domains. Use this to benchmark current state and target a realistic next level for each capability.

Clinical Safety

AI clinical safety assurance

Degree to which AI systems are validated, monitored, and governed for patient safety.

Level 1

Initial

No formal clinical safety process for AI systems

Level 2

Developing

Ad hoc clinical testing before deployment; no ongoing monitoring

Level 3

Defined

Standardized clinical validation process; basic post-deployment monitoring

Level 4

Managed

Comprehensive clinical validation with subgroup analysis; automated performance monitoring

Level 5

Optimized

Continuous clinical safety assurance with real-time monitoring, automated drift detection, and proactive remediation

Regulatory Compliance

Healthcare AI regulatory readiness

Organizational capability to identify, interpret, and comply with applicable healthcare AI regulations.

Level 1

Initial

Unaware of healthcare-specific AI regulations

Level 2

Developing

Awareness of key regulations but no systematic compliance process

Level 3

Defined

Regulatory requirements mapped to AI systems; compliance gaps identified

Level 4

Managed

Active compliance program with monitoring, reporting, and remediation

Level 5

Optimized

Proactive regulatory intelligence; automated compliance verification; regulatory change management

Health Equity

AI bias and equity assurance

Systematic approach to identifying and mitigating AI bias across patient demographics.

Level 1

Initial

No bias testing for healthcare AI systems

Level 2

Developing

Ad hoc bias testing on select systems; limited demographic analysis

Level 3

Defined

Standardized bias testing protocol; demographic subgroup analysis for all clinical AI

Level 4

Managed

Continuous bias monitoring; automated disparity detection; remediation workflows

Level 5

Optimized

Proactive equity-by-design; AI development includes equity validation from inception; community engagement in AI governance

Data Governance

Healthcare AI data governance

Governance of data used in AI training, validation, and inference within healthcare contexts.

Level 1

Initial

No data governance specific to AI systems

Level 2

Developing

Basic HIPAA compliance for AI data; limited data lineage tracking

Level 3

Defined

Formal AI data governance policies; de-identification standards; vendor data agreements

Level 4

Managed

Automated data quality monitoring; complete data lineage; consent management for AI use

Level 5

Optimized

Data governance integrated into AI lifecycle; real-time data quality; federated learning capabilities

COMPEL Methodology Mappings

How healthcare AI governance activities map onto the six COMPEL stages — Calibrate, Organize, Model, Produce, Evaluate, and Learn — including the artifacts and regulatory alignment for each stage.

Calibrate

Healthcare AI Readiness Assessment

Assess organizational readiness for healthcare AI governance including regulatory landscape analysis, AI inventory, and risk appetite definition.

Artifacts

- AI system inventory
- SaMD classification register
- Risk appetite statement
- Regulatory requirements matrix

Regulatory Alignment

- HHS AI Strategy Pillar 5
- FDA SaMD classification
- State AI inventory requirements

Organize