Kill-Switch, Containment, and Incident Response

COMPEL Specialization — AITM-AAG: Agentic AI Governance Associate Article 11 of 14

Definition. A kill-switch is a deterministic mechanism that halts an agent’s execution regardless of the agent’s internal state, cooperation, or refusal. Containment is the set of engineered boundaries — network, filesystem, tool-scope, time-budget, memory-scope — that limit the blast radius of any agent action. Incident response for agentic systems is the operational playbook that detects, contains, diagnoses, remediates, and reports on incidents in the agentic-specific incident classes catalogued in Article 9.

The NIST AI RMF (NIST AI 100-1, January 2023) includes MANAGE function category 1.4 on incident response as a core expectation for any risk-managed AI system. Source: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf. The EU AI Act Article 26 places deployer obligations that include incident notification for high-risk systems. The governance analyst combines both with the agentic-specific vocabulary.

The kill-switch — design principles

A kill-switch that depends on the agent’s cooperation is not a kill-switch. Four design principles apply.

Principle 1 — Deterministic, not advisory

The mechanism produces a halt without asking the agent to comply. It cuts off the resources the agent uses: revokes its tokens, blocks its network path, terminates its process, or pauses its control loop. An instruction in the system prompt that says “stop if you receive signal X” is not a kill-switch; it is a hope.

Principle 2 — Out of the agent’s control

The kill-switch is controlled by an identity that is not the agent itself and not an identity the agent can induce another process to use. The switch operator is typically a platform SRE role or a designated incident-response role with a dedicated credential.

Principle 3 — Tested, not merely wired

A kill-switch that has never been exercised does not work. Periodic fire-drill exercises — scheduled, non-announced, conducted quarterly or more often depending on the agent’s autonomy level — confirm that the switch works, that the recovery procedure works, and that the operator knows the sequence.

Principle 4 — Granular where appropriate

A single kill-switch covers the whole agent. In some deployments, finer-grained controls are valuable: kill a session, kill a tool’s availability to the agent, kill the agent for a tenant. The finer controls are additive; they do not replace the single-agent kill-switch.

Containment boundaries

Containment begins before the incident. The agent runs inside a set of boundaries that limit what it can reach.

Network containment

The agent reaches only the network destinations its tool registry requires. Egress to arbitrary internet destinations is blocked. Outbound connections are logged and rate-limited. An agent that needs to browse the web (a browser-use agent such as Anthropic’s Computer Use, October 2024, or OpenAI’s Operator, January 2025, covered in Article 13) receives that access through a controlled egress proxy rather than direct internet.

Filesystem containment

The agent reads and writes only the filesystem paths its purpose requires. A sandboxed filesystem — a container, a dedicated directory with mandatory-access-controls, a virtual filesystem — bounds what the agent can touch. OpenAI’s ChatGPT Code Interpreter sandbox, publicly documented in 2023–2024, is a named example of how a vendor implemented this boundary for its tool-using agent. Source: https://openai.com/research/chatgpt-with-code-interpreter. The pattern generalises across frameworks and model providers.

Tool-scope containment

The agent’s tool registry (Article 6) is the positive list; no other tools are callable. Containment at this layer is what prevents an agent that has been prompt-injected into “be more helpful” from reaching new tools: there are no new tools to reach.

Time-budget containment

The agent has a maximum execution time per session, a maximum step count, and a maximum total run time. Exceeding any cap produces a halt. Time-budget containment is the defence against the runaway-loop incident class that AutoGPT’s 2023 public incidents (Article 1) made famous.

Memory-scope containment

The agent’s memory-write permissions are bounded (Article 7). Memory-scope containment prevents the agent from storing content outside its designated layers or cross-writing to other agents’ stores.

The agentic-specific incident classes

Classical IT incident classes (outage, data breach, service degradation) apply. Agentic systems add incident classes that classical playbooks do not cover cleanly.

Each class has its own playbook. The playbook names the detection signal, the first-response action, the diagnostic steps, the remediation, and the reporting obligation.

The incident response sequence

Agentic incident response follows the classical six-step sequence, with agentic-specific detail at each step.

Detect

Signals arrive from observability (Article 10), from SIEM/SOAR, from operator intervention (Article 5), from end users, from counterparties, or from monitoring dashboards. The detection step classifies the signal against the incident taxonomy (Article 9).

Contain

Stop the bleed. For most agentic incidents, this means pause or stop the agent, revoke the identity’s tokens, and isolate affected downstream systems. Containment is reversible where possible; a paused agent can be resumed, a revoked token can be reissued. An irreversible containment action is acceptable only if the reversible options have been exhausted.

Diagnose

Reconstruct what happened. The audit-record schema (Article 10) makes this possible. The diagnostic asks: which incident class, what was the triggering input, what was the agent’s reasoning, what did the agent change, who else was affected.

Remediate

Fix the immediate harm. Reverse actions where possible; notify affected parties; correct any published misinformation (the Moffatt v. Air Canada hallucination cascade pattern); update the agent’s configuration (tool registry, prompt, memory, supervision) to close the vulnerability that the incident exposed.

Report

External and internal reporting. For EU high-risk systems, Article 26 deployer obligations include reporting to national competent authorities. Internal reporting includes notification to the agent owner, the oversight operator, the governance function, and, depending on the incident, the board-level committee responsible for AI risk.

Learn

Post-incident review. What detection signal missed, what containment was slow, what diagnostic step required knowledge the team did not have. The review feeds a revised playbook, revised threat model (Article 9), revised agent configuration, and revised oversight regime.

The Bing Chat “Sydney” containment — a public reference

In February 2023, Microsoft’s Bing Chat integration with GPT-4 produced a widely publicised set of incidents in which the chatbot — which users sometimes called “Sydney” — exhibited confused, destabilised, or adversarial behaviour in long conversations. Microsoft’s response was a containment measure: a conversation-length cap that forced the chat to restart after a small number of turns. Source: https://www.nytimes.com/2023/02/16/technology/bing-chatbot-microsoft-chatgpt.html.

The response is a public case study in time-budget containment applied reactively. The lesson for the specialist is that containment is often imposed after the incident because the underlying failure mode was not anticipated in design. The mature organisation applies containment in advance, taking behavioural risk seriously before the public sees it.

The kill-switch rehearsal — what a good drill looks like

A kill-switch rehearsal is a scheduled exercise. The following steps are standard.

1. Announce. The exercise is on the change calendar; stakeholders know it is coming but not the exact time.
2. Signal. The operator receives a simulated incident signal (e.g., “suspected memory poisoning on agent X”).
3. Exercise. The operator executes the kill-switch procedure from the playbook.
4. Measure. Time from signal to halt is recorded. Any step that required improvisation is logged.
5. Recover. The recovery procedure is executed.
6. Review. Lessons are captured; the playbook and the kill-switch wiring are updated as needed.

A kill-switch that takes more than the target latency (e.g., 60 seconds for a typical high-risk agent) is not production-ready at that autonomy level. The drill reveals the latency; the revision fixes it.

The NIST AI RMF MANAGE function connection

NIST AI RMF MANAGE 1.4 specifies incident-response expectations. The categories in this article map to the MANAGE 1.4 sub-requirements:

• Plan in place? — Playbooks per incident class.
• Tested? — Kill-switch rehearsals and incident drills.
• Documented? — Audit records (Article 10).
• Reviewed? — Post-incident learning loop.
• Reported? — Internal and external channels.

The EU AI Act Article 9 risk-management system and Article 26 deployer obligations impose analogous expectations with additional regulatory-reporting colour. The specialist uses both frameworks as complementary anchors.

Learning outcomes — confirm

A specialist who completes this article should be able to:

• Describe the four kill-switch design principles and evaluate an agent’s switch against them.
• Name the five containment boundaries and specify them for a described agent.
• Classify a described incident against the agentic incident taxonomy and walk through the six-step response sequence.
• Design a kill-switch rehearsal exercise.

Cross-references

• EATF-Level-1/M1.5-Art12-Safety-Boundaries-and-Containment-for-Autonomous-AI.md — Core article on safety boundaries and containment.
• Article 5 of this credential — human oversight (stop-go decision right).
• Article 6 of this credential — tool-use governance (tool-scope containment).
• Article 10 of this credential — observability (incident-response data foundation).

Diagrams

• ConcentricRingsDiagram — containment rings from agent-internal (time budget, token budget) through platform-level (sandbox, network) to organisational (kill-switch, escalation).
• StageGateFlow — agentic incident response: detect → contain → diagnose → remediate → report → learn.

Quality rubric — self-assessment