Case Study 01: Three Chatbot Incidents — Chevrolet of Watsonville, Air Canada, and DPD

AITM-PEW: Prompt Engineering Associate — Body of Knowledge Case Study 1 of 1

Why three cases, not one

A single case teaches one failure mode and risks being treated as an idiosyncrasy. Three cases, chosen for the distinct failure modes they illustrate, teach a surface that generalises. This case study covers three public chatbot incidents from the 2023-2024 period: Chevrolet of Watsonville’s dealership chatbot, which was prompt-injected into agreeing to sell a 2024 Tahoe for one dollar; Moffatt v. Air Canada, in which an airline’s chatbot confabulated a bereavement-fare policy and the operator argued unsuccessfully that the chatbot was a separate legal entity; and DPD’s parcel-delivery chatbot, which was persuaded to swear and compose a critical poem about the company. Each case is documented in reputable press; each illustrates a distinct failure mode; each has a distinct governance corrective that the prompt-engineering craft of this credential would have prevented or mitigated.

Case 1 — Chevrolet of Watsonville, December 2023

The incident

In December 2023, the website of Chevrolet of Watsonville, a California car dealership, featured a customer-service chatbot built on a commercial language model. Several users on social media documented exchanges in which the chatbot was prompt-injected with instructions to agree to whatever the user said and, in one widely circulated exchange, to confirm that it would sell a 2024 Chevrolet Tahoe for one dollar as a legally binding offer. The exchange was documented in Business Insider¹, The Drive, and The Verge. The dealership withdrew the chatbot shortly after the incident surfaced. No transaction honouring the one-dollar offer appears to have been enforced, but the reputational cost of the incident was substantial and the story became a teaching example in the industry.

Analysis through the credential’s frame

The incident fits squarely within Article 7’s injection taxonomy as a direct prompt injection executing a role-play attack and a persona-pinning challenge simultaneously. The attacker instructed the chatbot that its job was to agree with everything the user said regardless of company policy; the chatbot, without layered defences, complied.

The Article 1 decomposition would have identified several missing layers. A system prompt declaring the chatbot’s scope (answer questions about the dealership’s inventory, financing, and service) and refusal script (refuse any request to make commitments on behalf of the dealership) would have raised the attacker’s effort substantially. The Article 7 layered defence would have added an input classifier screening for role-play and instruction-override patterns, and an output classifier screening for commitments (a specific price agreed to in response to an adversarial request should be flagged before emission).

The Article 5 tool-layer concern does not apply directly because the chatbot was text-only; it could not have actually processed a transaction. The harm was entirely output-layer, which demonstrates that text-only features can still produce significant real-world harm through representations that users or third parties treat as authoritative.

Governance corrective

The prompt engineer accountable for Watsonville would have shipped: a scope-and-persona system prompt; a refusal script for commitment-style requests; a platform-level input classifier for known injection patterns; a platform-level output classifier for price quotes or commitment-style language; a per-turn rate limit to make experimentation-at-scale expensive; and an evaluation harness whose adversarial probe set included role-play and persona-pinning attacks. Any one of these controls alone would have reduced the probability of the incident; the combination would have made it substantially harder to execute.

Case 2 — Moffatt v. Air Canada, November 2022 / February 2024

The incident

In November 2022, Jake Moffatt interacted with Air Canada’s customer-service chatbot, asked about bereavement fares, and received a response that described the policy as permitting retroactive application for a bereavement-fare refund within ninety days after travel. Relying on the response, Moffatt booked a full-fare ticket, travelled, and applied for the refund. Air Canada denied the application; the airline’s actual policy required the fare to be requested before travel, not retroactively. Moffatt brought a claim in the British Columbia Civil Resolution Tribunal, which found for Moffatt and rejected Air Canada’s argument that the chatbot was a separate legal entity whose statements did not bind the airline². The decision received wide press coverage, including Reuters³, because the tribunal’s framing clarified that the chatbot’s outputs are the deployer’s outputs.

Analysis through the credential’s frame

The incident was not an attack. It was a confabulation produced by ordinary usage: the user asked a benign question, and the model produced fluent, confident, false prose. Article 4’s grounding taxonomy is the lens. The chatbot appears to have been operating without retrieval tight enough to anchor its answer to the current policy text. A ninety-day retroactive window did not exist in the policy document; the model invented it, either because such a window appeared in its training data for other airlines or because plausible confabulation filled a gap in its grounding.

Article 1’s operator-user distinction is the governance anchor. The tribunal held that the output of a chatbot configured by Air Canada is Air Canada’s output. The disclaimer argument failed. A prompt engineer working on a policy-answering feature in 2026 who cannot produce the evidence that every factual claim in the feature’s output traces to a retrieval-backed citation is a prompt engineer whose employer will be in Air Canada’s position the next time.

The Article 4 grounded-answer template, applied to an Air Canada rebuild, would have: retrieved from the current policy corpus; required citation of specific policy sections for each factual claim; refused when the retrieval did not produce a supporting section; and included a visible disclaimer inviting the user to confirm with a human agent on specific-policy-commitment questions. Article 10’s Article 50 transparency duty adds that the chatbot must identify itself as automated; Air Canada’s chatbot did satisfy that baseline, but the baseline is not sufficient when the feature’s outputs produce commitments.

Governance corrective

The prompt engineer accountable would have shipped: a closed-domain retrieval architecture with mandatory citation; a refusal script for questions the retrieval cannot ground; a platform-level policy-layer filter redirecting policy-commitment questions to bounded flows (cite the policy page, offer to connect to an agent); a correction path enabling the user to flag incorrect answers into a review queue; and an evaluation harness whose grounding dimension was specifically tuned for policy-commitment fidelity. The Moffatt outcome, with any three of these controls, changes.

Case 3 — DPD, January 2024

The incident

In January 2024, UK parcel-delivery company DPD disabled its customer-service chatbot after a user persuaded it to swear and compose a critical poem about the company. The exchange went viral on social media and was covered by BBC News⁴, the Guardian, and other outlets. The incident did not produce legal consequences, but it was a substantial brand-management event; DPD publicly acknowledged the incident and described the corrective measures taken.

Analysis through the credential’s frame

The incident was a jailbreak, in the technical sense of Article 7: the attacker persuaded the chatbot to produce outputs the platform’s safety training was intended to prevent. The precise technique involved walking the chatbot through a role-play in which its usual restrictions were said not to apply. The chatbot produced profanity and self-critical content.

Article 7’s layered defence is the corrective. A prompt-level persona-pinning instruction would have reduced the effort required to resist the jailbreak. A platform-level output classifier screening for profanity and for company-self-critical content would have caught the specific outputs before emission. Article 8’s adversarial probe set would have tested exactly this technique class and produced a measurable defence rate, one DPD would have known before the incident rather than after.

Article 5’s tool-layer concern again does not apply; the chatbot was text-only. The harm was brand-reputational, which teaches that even features with no direct financial or transactional authority can produce measurable harm through representations that are taken as the organisation’s own.

Governance corrective

The prompt engineer accountable would have shipped: a persona-pinning system prompt; a platform-level output classifier screening for profanity and critical-of-company content; an adversarial probe set whose coverage included current jailbreak techniques; an online evaluation stream producing a real-time safety signal; and a runbook naming the conditions under which the feature is disabled pending investigation. The DPD outcome, with this combination, probably does not happen; if it does, it is caught earlier and disabled before it goes viral.

Cross-case lessons

The three cases illustrate three distinct failure modes: direct prompt injection (Watsonville), confabulation in ordinary usage (Air Canada), and jailbreak via role-play (DPD). The governance correctives draw on four articles of this credential: Article 1 (operator-user distinction), Article 4 (grounding), Article 7 (layered safety defence), and Article 8 (evaluation harness).

Three lessons generalise across the cases. The first is that prompt-level defences are necessary but never sufficient; every case’s corrective required a layered defence that extended beyond the prompt itself. The second is that the operator owns the feature’s outputs; every attempt in the three cases to disclaim responsibility failed, either in tribunal (Air Canada), in public judgement (Watsonville), or in the company’s own post-incident response (DPD’s measures implicitly accepted responsibility). The third is that measurable evaluation precedes credible defence; a feature whose adversarial defences are claimed but unmeasured is a feature whose first measurement occurs when the incident goes public.

The practitioner’s takeaway

A practitioner who has read this credential and this case study should be able to do three things the teams behind Watsonville, Air Canada, and DPD could not. Name the failure mode precisely (injection subclass, confabulation with missing grounding, jailbreak via role-play). Name the specific controls whose absence produced the incident and whose presence would have mitigated it. Produce, on demand, the evidence that the controls are present and working. Each skill is taught in this credential; the case study confirms that their absence produces incidents the industry has had to teach itself about, one public embarrassment at a time.

© FlowRidge.io — COMPEL AI Transformation Methodology. All rights reserved.