Hallucination, Grounding, and Output Integrity

AITB-LAG: LLM Risk & Governance Specialist — Body of Knowledge Article 3 of 6

In early 2023 a pair of attorneys filed a motion in the Southern District of New York that cited six federal cases to support their client’s position. The cases did not exist. Opposing counsel noticed first, the court noticed next, and the attorneys eventually admitted that they had asked a general-purpose language model to find supporting case law and had filed its output without verification. Judge Castel sanctioned them, and the opinion in Mata v. Avianca became the most cited LLM-risk teaching case of 2023 for a reason that still matters: the model had produced fluent, confident, legally-styled text that was not supported by any ground truth¹. Less than a year later, the British Columbia Civil Resolution Tribunal ruled in Moffatt v. Air Canada that the airline was liable when its customer-service chatbot told a passenger that a bereavement refund could be submitted after the flight; the tribunal rejected the argument that the chatbot was a separate legal entity whose errors the carrier could disclaim². Between them, the two cases define the governance stakes of the failure class the LLM research community calls hallucination and NIST calls confabulation.

Why the model makes things up

The name matters. NIST AI 600-1 adopts confabulation specifically to avoid anthropomorphizing the failure mode; the Profile describes it as the generation of fluent output unsupported by any ground truth³. “Hallucination” is the common term but carries connotations of sensory misfire that do not fit. Confabulation is the more precise framing: the model is completing a plausible sequence of tokens given its training and its prompt, and plausibility is not the same thing as truth.

Three mechanisms produce confabulation, and a practitioner should be able to distinguish them because they have different mitigations.

Gaps in training. The model was never exposed to the specific fact the user needs, but its training causes it to produce a fluent answer anyway. This is the category that vanishes when the system switches to a closed-domain retrieval pattern; the model no longer needs to guess if the correct passage is in the context window.

Conflicts in training. The model was exposed to contradictory information, and at generation time it blends. Blending is the most dangerous category because the output often looks coherent and contains correctly-remembered fragments glued to confabulated ones. Mata v. Avianca was largely this: real citation formats, real party names in circulation, plausible holdings, but combinations that were invented.

Drift from context. The correct material is in the context window, but the model’s attention mechanism weights its training more heavily than the retrieved content, and it produces an answer that the source does not actually support. This is the failure mode that defeats a naïve “just add RAG and the problem goes away” posture.

The three mechanisms point to the grounding question.

Five grounding architectures and what they do to the failure rate

Grounding is the practice of constraining model generation by retrieved or tool-provided evidence so that outputs can be checked against their sources. There are five practical architectures, and they sit at different places on a two-axis map: internal versus external knowledge source, and low versus high verification effort.

Closed-book. No retrieval. The model answers from its training alone. The oldest and still most common pattern for simple consumer assistants. Cheapest to deploy, highest confabulation rate, acceptable only where the cost of a wrong answer is low and the user is expected to verify anyway.

Closed-domain RAG. Retrieval-augmented generation (RAG) where the retrieval corpus is a controlled, curated set: the organization’s own policy documents, product documentation, or filed customer support cases. Confabulation drops sharply for questions the corpus can answer; for questions it cannot, the model may still drift or refuse. A closed-domain RAG over HR policies is the standard internal-knowledge-base copilot pattern. Vector stores such as Pinecone, Weaviate, Qdrant, pgvector, and Milvus all implement this with comparable risk profiles; the retrieval store’s choice is a performance and operations decision, not a safety decision⁴.

Open-domain RAG. Retrieval over a much broader index, most commonly a web search. The model can answer questions the organization did not anticipate, but it also inherits whatever the broader index contains, including adversarial pages planted to produce indirect prompt injection. Open-domain RAG feels safer than closed-book because citations are visible, but the citations can be to pages the attacker controls.

Tool-augmented. The model does not retrieve text; it calls a function that returns structured data (a database query, a calculator, a datetime lookup, an account balance API). Tool augmentation dramatically reduces confabulation for questions the tool can answer, because the answer is no longer generated by the model; it is passed through. The governance question moves from “did the model make up the answer” to “did the model call the right tool with the right arguments and did it relay the result accurately.”

Agent loop. The model decides when to retrieve, when to call tools, and when to answer, iterating over multiple turns. Agent loops amplify the power of the previous patterns and also amplify their failure modes, because a single confabulated intermediate step can propagate through the loop. Agent loops straddle the matrix: knowledge source depends on which tools the agent calls, and verification effort tends to be high because every intermediate step needs to be inspectable.

None of these architectures eliminates confabulation. What they do is change its shape and, critically, its detectability. A closed-domain RAG with visible source citations is easier to audit than a closed-book answer precisely because the evidence is in the response.

Designing disclosure and correction

The Moffatt decision is instructive not only for the liability finding but for what Air Canada could have done and did not. Three controls, in combination, would have changed the outcome.

Scoped disclaimers. A disclaimer that says “this assistant may produce errors; verify before relying on this information for binding decisions” is a weak control in isolation, but it is not zero. What matters is where it appears. A disclaimer hidden in a terms-of-service document does nothing. A disclaimer beside every answer that involves policy, pricing, or commitments signals to the user that the output is advisory. The EU AI Act Article 50(1) imposes an analogous transparency obligation: where natural persons interact with an AI system, they must be informed unless the interaction is obvious⁵. Practitioners should treat Article 50 as a floor, not a ceiling.

Citation surfaces. Every answer derived from retrieved content should cite the source in a way the user can click through. Citation is not decoration; it is the user’s primary defense against confabulation. An answer that matches its source wins trust. An answer that contradicts its source produces an immediate correctable moment.

Correction paths. When a user reports that an answer was wrong, what happens? A correction channel that disappears into a support queue and is never seen by the team running the feature is worthless. A correction channel that feeds into the evaluation harness (Article 5), updates the retrieval corpus where appropriate, and triggers a review for systemic patterns is a governance control.

The three controls are technology-neutral. They apply equally to a feature running on a closed-weight managed API and to one running on an open-weight self-hosted model. What changes between stacks is where the telemetry lives, not whether it is required.

Confabulation in tool-using and agentic contexts

Confabulation at the output layer becomes more dangerous the moment an LLM feature can take action. A model that confidently tells a user the wrong policy merely misleads. A model that confidently calls a tool with the wrong arguments executes the wrong action. A model that confabulates the existence of a tool, for example asking for a “refund_full_amount” function that was never registered, may error gracefully or, in the worst case on some agent implementations, pass the string to a permissive handler.

The NIST AI RMF GenAI Profile names this as a cross-cutting risk, linking it to both the “information integrity” and “value chain” risk categories⁶. A practitioner’s checklist for tool-using systems includes: validating arguments against strict schemas, confirming tool existence before dispatch, logging every dispatched call with its source reasoning, and requiring human confirmation for any action whose reversal is expensive. None of those is optional in a feature whose actions touch money, benefits, or public commitments.

What the practitioner takes away

The Air Canada tribunal’s most quoted paragraph is the one rejecting the idea that the chatbot was a separate legal entity. That rejection is not exotic; it is a straightforward application of organizational liability to a new delivery channel. The governance implication for every deployer is that the organization owns its LLM’s outputs in the same way it owns the statements of its employees, and that ownership does not depend on whether the underlying model was built in-house or procured from a vendor. Confabulation is the category most likely to generate those statements. Grounding reduces it, disclosure warns about it, correction paths learn from it, and evaluation (Article 5) measures the residual.

Summary

Confabulation is not a bug to be patched; it is a property of the system that must be designed around. The five grounding architectures (closed-book, closed-domain RAG, open-domain RAG, tool-augmented, agent-loop) sit on a map of knowledge source and verification effort, with different residual rates and different controls. Disclosure, citation, and correction paths are the user-facing controls that translate residual confabulation into managed risk. Mata v. Avianca and Moffatt v. Air Canada are the teaching cases that quantify what happens when these controls are absent.

Further reading in the Core Stream: Grounding, Retrieval, and Factual Integrity for AI Agents, Generative AI and Large Language Models, and Measuring AI Value.

© FlowRidge.io — COMPEL AI Transformation Methodology. All rights reserved.