Regulatory Obligations and Incident Response

AITB-LAG: LLM Risk & Governance Specialist — Body of Knowledge Article 6 of 6

On 30 March 2023 the Italian data protection authority (the Garante per la protezione dei dati personali) issued Provvedimento 112/2023, a temporary provisional measure ordering OpenAI to suspend processing of Italian users’ personal data through ChatGPT. The measure identified specific failings: absence of adequate lawful basis for training-data processing, inadequate transparency to users, and insufficient protection for minors who could access the service without effective age verification¹. The suspension lasted three weeks and ended after OpenAI implemented remediations. Twenty months later, on 30 December 2024, the Garante concluded its formal investigation and issued a fifteen-million-euro fine plus additional remediation orders, setting the European enforcement tone that several other data-protection authorities subsequently echoed². The Garante decisions are teaching cases because they localize regulatory obligation on LLMs in terms any practitioner can recognize: lawful-basis questions, transparency questions, incident-disclosure questions, and cross-border questions. This article extends that recognition across the three anchors that structure LLM governance (the EU AI Act, the NIST AI Risk Management Framework, and ISO/IEC 42001) and closes with what an LLM-aware incident response looks like when those anchors intersect.

The EU AI Act: what applies when

Regulation (EU) 2024/1689, commonly referred to as the EU AI Act, entered into force in August 2024 with a phased applicability schedule that extends through 2027. The Act does not regulate LLMs as a named category; it regulates AI systems and AI models according to a layered scheme, and the parts of that scheme that bite on LLM deployments are Articles 6 and Annex III for high-risk use cases, Articles 50 through 56 for general-purpose AI model obligations, Article 52 for transparency duties, and Article 26 for deployer duties³.

Article 6 and Annex III, high-risk classification. A system is high-risk when it is used in one of the Annex III use cases (employment decisions, credit scoring, essential services access, education, law enforcement, and several others) or when it is a safety component of a regulated product. A retrieval-augmented LLM that screens job applicants falls under Annex III point 4. An LLM that supports credit decisioning falls under point 5(b). A customer service chatbot that also handles account access may touch point 5(a) depending on design. The classification triggers conformity-assessment, risk-management, quality-management, logging, human-oversight, and transparency obligations.

Articles 50 through 56, general-purpose AI model providers. A provider of a GPAI (general-purpose AI model), defined in Article 3(63) as a model trained on broad data capable of serving many downstream tasks, has obligations regardless of the downstream use case. These include technical documentation (Article 53), copyright-compliance policies (Article 53(1)(c)), and training-data transparency summaries. A GPAI model with systemic risk, meeting the capability thresholds in Article 51, has additional obligations including model evaluations, adversarial testing, incident reporting, and cybersecurity. Organizations that self-host open-weight models or fine-tune them may in some configurations meet the “provider” definition and inherit these obligations; organizations that consume a managed API do not, but their vendor does.

Article 50, transparency on interaction with AI. Any natural person interacting with an AI system must be informed that they are doing so, unless the fact is obvious from context. Deepfake content and synthetic media have additional labelling duties in Article 50(4). The Garante’s transparency finding in the ChatGPT case is the Italian DPA analogue of the same obligation, grounded in the General Data Protection Regulation’s transparency requirements rather than the AI Act, but the practical disclosure is similar.

Article 26, deployer duties. Even when the provider is another party, the deployer has operational obligations: use the system per its intended purpose, assign human oversight to trained individuals, monitor operation, keep logs, inform workers when the system is used in employment contexts, and take corrective action when the system presents a non-conformity risk. The Air Canada case discussed in Article 3 is a common-law deployer-liability case that prefigures the Article 26 regime in a regulatory form.

NIST AI RMF and ISO 42001: the management-system anchors

The EU AI Act tells the organization what it must achieve. NIST AI RMF and ISO/IEC 42001 tell the organization how to structure the management system that achieves it.

NIST AI RMF. NIST AI 100-1 organizes AI risk management into four functions: Govern, Map, Measure, and Manage. Each function has categories and subcategories. The practitioner translates the abstract text into concrete LLM controls. MAP 1.1 (context established, documented, and understood) maps to the risk-surface brief from Article 1. MEASURE 2.7 (AI system is evaluated for risks and impacts) maps to the evaluation harness from Article 5. MANAGE 1.4 (planned responses to AI risks and incidents) maps to the incident runbook discussed below. The Generative AI Profile (NIST AI 600-1) layers twelve risk categories on top of the base RMF, each of which touches at least one of the six layers of the risk surface⁴.

ISO/IEC 42001. The AI management-system standard published in 2023 provides the auditable clause structure that many organizations use to demonstrate compliance with both the NIST RMF and the EU AI Act’s Article 9 (risk-management system) and Article 17 (quality-management system) requirements for high-risk systems. Clause 6.1.2 (risk assessment) and 6.1.3 (risk treatment) are the planning clauses; clause 8.1 (operational planning and control) is where the deployed controls live; clause 9.1 (monitoring, measurement, analysis, and evaluation) is the evidence clause⁵. The evidence pack the auditor requests under 9.1 is effectively a curated extract from the harness and the runbook.

Mapping obligation to control

The obligation-to-control mapping is where abstract regulation becomes operational. A condensed version, suitable for the practitioner’s reference:

The table is short deliberately. The purpose is not to catalog every clause (Article 6 of the Act alone implicates a further chain of conformity-assessment articles) but to demonstrate that the controls from Articles 1 through 5 of this credential are already the spine of the evidence an auditor or regulator will request.

LLM-aware incident response

An AI incident (for LLMs) is not a generic security incident. The NIST AI RMF MANAGE 1.4 subcategory requires a planned response, but the planned response needs to anticipate incident classes that classical incident-response runbooks were not written for. A credential-specific taxonomy names five classes:

Confabulation harm. The feature produced fluent output that was materially wrong and led to harm: user misinformed, contract miscommunicated, legal filing contaminated. Moffatt v. Air Canada and Mata v. Avianca are the teaching cases.

Safety bypass. The feature produced content that violated the organization’s or the jurisdiction’s safety policy, covering categories such as dangerous instructions, harassment, and targeted abuse. The DPD poetry incident is a mild example; the categories escalate.

Prompt injection or jailbreak realized. An adversary successfully redirected the feature’s behavior and obtained content or action the feature was supposed to prevent. The Chevrolet dealership case is a direct-injection realization; an indirect-injection case where tool actions followed would be more severe.

Data leak. The feature exposed information it should not have: training-data memorization surfacing sensitive content, system-prompt leakage revealing internal logic, or retrieval from a corpus that should have been segmented by tenant or role. The Samsung case discussed in Article 1 is an inbound variant; the OpenAI March 2023 Redis-bug incident in which chat titles leaked across users is an outbound variant⁶.

Policy-violating tool use. A tool action was taken that should not have been: an email sent, a record updated, a commitment made. This is the class with the steepest real-world consequences once agent-capable deployments become common.

The runbook has seven phases, parallel to classical security incident response but with LLM-specific content at each: detect (who caught it, whether user report, monitoring, or external), triage (which class, what severity), contain (disable the feature, the tool, the retrieval path, or the specific pattern), notify (legal, regulator, customer, public as appropriate), remediate (fix the specific instance and the root cause), learn (add to the evaluation suite, update the runbook), and disclose (post-incident report, regulator filing under EU AI Act Article 73 for serious incidents on high-risk systems, and any applicable sectoral notification).

Notification deadlines matter. Article 73 requires serious-incident reporting to the relevant market-surveillance authority within specific windows, as short as two days for certain categories. An incident response plan without a pre-mapped notification workflow will miss the window.

The evidence pack

An auditor’s request, in one form or another, will be: “show us the evidence that the controls you claim to have are running.” The practitioner’s preparation is an evidence pack that bundles, per feature: the risk-surface brief, the model and prompt registry snapshot, the guardrail configuration and test results, the harness run logs for the reporting period, the red-team reports and closure evidence, the production monitoring dashboards for the period, the incident log and post-incident reports, the human-oversight training records, and the transparency disclosures as implemented. ISO 42001 Clause 9.1 is the evidence clause that formalizes this bundle for audit.

An evidence pack assembled only at audit time is an evidence pack that will be incomplete. The practice is to assemble continuously and to snapshot at audit time.

Summary

Regulation on LLMs is operational, not prospective. The EU AI Act, NIST AI RMF, ISO 42001, and sectoral regulation already impose obligations a practitioner can enumerate. The Garante decisions and the Moffatt tribunal are the public cases that translate those obligations into the language of concrete harms. Controls from the previous five articles compose the evidence an audit will request. An LLM-aware incident runbook, spanning confabulation, safety bypass, prompt injection, data leak, and policy-violating tool use, is the final operational artifact of this credential, and the artifact that more than any other distinguishes an LLM feature the organization can govern from one it merely hosts.

Further reading in the Core Stream: Understanding the EU AI Act: Foundations for Governance, EU AI Act Risk Categories and Your Organization, EU AI Act Compliance for Practitioners, and Audit Preparedness and Compliance Operations.

© FlowRidge.io — COMPEL AI Transformation Methodology. All rights reserved.