Privacy, Sensitive Data Classes, and Data Minimization

AI systems create privacy-risk classes that classical privacy programs were not built to address. Large language models can memorize training-set content and reproduce it on query. Prediction models can enable attribute inference, identifying a protected attribute of an individual from non-protected inputs. Re-identification attacks can rejoin ostensibly anonymized data against auxiliary datasets. The readiness practitioner extends the classical privacy program to cover these risks.

The AI-specific privacy failure modes

Four failure modes recur in AI privacy incidents.

Memorization

A large language model trained on a corpus containing personal data can reproduce that data at inference time. Carlini et al. demonstrated the effect rigorously in a USENIX Security 2021 paper showing that specific personally-identifying strings from the GPT-2 training corpus could be recovered through crafted queries.¹ The finding generalizes: memorization probability increases with training-data repetition, with training-loss optimization, and with the rarity of the string within the distribution. Any training corpus that contains personal data is at risk of memorization unless affirmative controls are applied.

Attribute inference

A model can infer a protected attribute from features that do not directly contain it. A recommender can infer likely age, gender, or health conditions from behavioral patterns. A loan-approval model can infer protected-class membership from combinations of non-protected features. The readiness practitioner audits for inference risk during the bias-relevant variable analysis (Article 7) and extends the audit here to cover the privacy dimension: even where the inference is accurate, the act of inferring can itself be a processing-of-sensitive-data event under GDPR Article 9.

Re-identification

Data that has been de-identified (name removed, identifiers hashed) can often be re-identified by cross-referencing with auxiliary datasets. The classic Netflix Prize re-identification demonstration and the AOL query-log incident are the canonical examples; more recent research continues to push the boundary on what data can be re-identified from what auxiliary sources. The readiness practitioner treats de-identification as one layer of a minimization strategy, not as a guarantee.

Prompt-archive leakage

A newer failure mode specific to LLM-powered applications. User prompts and model completions are often retained for debugging, analytics, or fine-tuning. Prompts can contain personal data that the original consent scope did not cover. The readiness practitioner audits prompt-retention policies and requires explicit scope, retention window, and purging commitments.

[DIAGRAM: ConcentricRingsDiagram — sensitivity-rings — five concentric rings labeled outward from center: public, internal, confidential, restricted (PII), regulated (special-category / PHI / PCI); each ring annotated with handling controls (retention, access scope, pseudonymization, differential privacy, on-premise constraint, training-use restrictions) and the reference legal regime (GDPR, HIPAA, PCI DSS, national security / export control)]

The minimization duty

Minimization is a principle with regulatory force in multiple jurisdictions.

• GDPR Article 5(1)(c) — personal data shall be “adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed.”²
• EU AI Act Article 10(2)(c) and (3) — training, validation and testing datasets shall have appropriate statistical properties and be relevant; the provider shall justify the data used.³
• US state consumer privacy laws (California, Virginia, Colorado, Connecticut, Utah, and others) each include a minimization duty in varying forms.

For an AI readiness assessment, minimization has three operational tests:

1. Purpose specification. Has the specific AI use case been specified in enough detail that the data needed can be derived from it?
2. Necessity justification. For each field in the training set, is there a documented reason why it is required for the use case? Fields without justification are minimization failures.
3. Retention justification. For the training-data snapshot retained after training, is there a documented reason to retain it, tied to the period the model will be deployed and the period the provider is obligated to support auditability?

The practitioner produces a minimization memo that answers all three tests and lives in the decision log. A use case that cannot pass the memo is a readiness failure and either must be re-scoped or cannot proceed.

The Italian Garante ChatGPT enforcement

The Italian data protection authority (Garante per la protezione dei dati personali) issued a final decision against OpenAI on 30 December 2024, imposing a €15 million fine for violations including lack of lawful basis for training data processing, transparency failures, and insufficient age verification for minors.⁴ The decision followed an earlier Garante order (Provvedimento 112/2023) that had temporarily restricted ChatGPT in Italy in March 2023 pending remediation.

The readiness translation is concrete. The Garante identified three classes of failure that any readiness practitioner should test against:

• Lawful basis for training. Was there a documented legal ground (Article 6 of GDPR, and for any special-category data, an Article 9 exemption) for each dataset used to train the model?
• Transparency. Were data subjects informed, in clear language, that their data would be used to train AI, and what the consequences were?
• Rights-affected design. Were data subjects’ rights (access, rectification, erasure, objection) operationally honored for data that had been incorporated into training?

An AI readiness scorecard should include a section specifically addressing each of the three tests for every in-scope dataset. A gap in any of the three is a high-priority finding.

Privacy-preserving techniques

Four classes of technique can reduce privacy risk in AI training. The readiness practitioner should understand the applicability and limits of each.

Differential privacy

Differential privacy is a mathematical guarantee that bounds the influence of any single record on outputs. A training algorithm with differential privacy adds calibrated noise to gradients during training, producing a model whose behavior does not meaningfully change if any single training record is removed. The formal definition was established by Dwork and collaborators; the textbook reference is Dwork and Roth (2014).⁵

Applicability: differential privacy is strong where the use case can tolerate the accuracy cost of noise injection. For extremely rare-event prediction, the noise may swamp the signal. For aggregate-pattern learning on large datasets, the cost is manageable.

Limits: the privacy parameter ε (epsilon) must be small enough to be meaningful; published studies on deployed systems sometimes report ε values that are too large to provide practical privacy. The practitioner should require the ε to be named, justified against the threat model, and recorded in the readiness evidence.

Federated learning

Federated learning keeps training data on the devices or in the facilities where it originates; only model updates are exchanged. This reduces the data-centralization risk that underlies many enterprise privacy incidents.

Applicability: federated learning is strong where the data is inherently distributed (device-side signals, multi-hospital consortia) and where central aggregation is politically or legally difficult.

Limits: federated updates can leak information about the training data through gradient analysis. Secure aggregation, differential privacy, and careful update-clipping are typically combined to strengthen the guarantee.

Synthetic data

Synthetic data is artificially generated data that mimics the statistical properties of a source dataset without reproducing specific real records. It is used for augmentation (fill in underrepresented subgroups), for privacy-preserving distribution (share a synthetic version externally), and for edge-case coverage.

Applicability: synthetic data is strong for use cases that learn distributional properties and where the synthetic-generation process can be governed. It has its own readiness rubric — the synthetic dataset is itself a dataset with its own provenance, quality, and representativeness.

Limits: poorly-generated synthetic data can introduce artifacts that the model learns instead of the underlying signal. A synthetic dataset that preserves statistics but fails to preserve relationships can mislead downstream training. The practitioner should require synthetic-data validation against held-out real data before accepting synthetic-augmented corpora.

Selective redaction

For text and multimodal data, selective redaction removes direct identifiers (names, phone numbers, account numbers) before training. Modern redaction uses named-entity recognition and rule-based extraction; quality varies by domain.

Applicability: selective redaction is a cheap first layer that should almost always be applied; it does not substitute for the other techniques but reduces the surface area the other techniques must protect.

Limits: redaction is imperfect. Rare-entity patterns and context-dependent identifiers slip through. Redaction should be combined with ongoing audit and, for sensitive domains, with a human review layer.

[DIAGRAM: BridgeDiagram — raw-to-minimized — bridge from “Raw personal dataset” on the left to “Minimized, privacy-preserved training dataset” on the right; the bridge spans show the applied techniques in order (scope specification, necessity filter, selective redaction, pseudonymization, differential-privacy training or federated training, validation of synthetic substitutes), with the residual-risk assessment annotated on the right pillar]

The extracting-training-data failure class

Carlini et al.’s 2021 paper is the most-cited member of a broader research family that keeps producing demonstrations of extraction attacks against deployed LLMs.¹ Subsequent work has extended the attack classes to code models, vision models, and retrieval-augmented systems. The readiness practitioner treats extraction-attack vulnerability as a first-order concern for any LLM use case trained on or fine-tuned with non-public data.

The mitigations combine: strict data minimization (do not train on what you cannot afford to leak), duplication detection (repetition drives memorization), differential-privacy training for the sensitive fraction of the corpus, and ongoing red-team probing (Article 10 covers the incident-response dimension).

Sensitive-data-class handling map

A practical readiness scorecard organizes privacy evidence by sensitive-data class. Each class has a distinct regulatory regime and a distinct set of expected controls. The practitioner should be able to walk a reviewer through the map row by row, answering “what data of this class is in scope” and “what controls are in place.”

• Personally identifiable information (PII). Direct identifiers (name, email, phone, government ID) and indirect identifiers (cookie ID, device fingerprint, user agent plus timestamp) together. Controls: minimization memo, access-scope enforcement, pseudonymization where feasible, data-subject rights process, retention limit.
• Special-category personal data (GDPR Article 9). Health, biometrics, ethnicity, sexual orientation, political opinion, religious belief, trade-union membership. Controls: Article 9 exemption basis documented, explicit consent where applicable, heightened access restrictions, audit log of all access, retention minimized.
• Protected health information (HIPAA). A narrower US-centric class with its own rule set. Controls: covered-entity or business-associate relationship established, minimum-necessary standard applied, audit log of access, de-identification per Safe Harbor or Expert Determination rules before training.
• Payment-card information (PCI DSS). Cardholder data with specific handling rules. Controls: PAN not retained in training sets, tokenization applied, cardholder environment segregated from modeling environment.
• Children’s data. US COPPA and many other jurisdictions carry specific rules for data of children under defined age thresholds. Controls: parental-consent documentation, age gating enforced, elevated minimization, separate retention rules.
• Employee data. Subject to employment law in addition to general privacy rules. Controls: works-council or union consultation where required, purpose limitation enforced, monitoring-transparency obligations met, retention tied to employment lifecycle.
• Client-confidential data. Not always personal data, but subject to contractual confidentiality. Controls: client-consent or contract basis documented, access restricted to engagement team, training-use restricted to the specific client engagement or aggregated forms, retention tied to engagement close-out.

A readiness scorecard’s privacy section walks every class that is in scope for the use case, presents the evidence for each, and surfaces gaps. A use case that touches four classes (for example, a customer-service assistant that sees PII, HIPAA-adjacent health inquiries, payment data, and a sliver of children’s-data interactions) has four rows to walk, not one composite row.

Vendor and third-party exposure to privacy regimes

Privacy duties follow the data across organizational boundaries. Where a third-party service processes personal data on behalf of the organization, that service is a data processor under GDPR or an equivalent role under other regimes, and a data-processing agreement is required. For AI use cases, two patterns recur:

• Third-party LLM API. The API vendor processes prompts that contain personal data. The vendor may retain prompts for service improvement, fine-tuning, or security monitoring. The readiness practitioner reads the vendor’s data-processing terms, confirms the training-use posture (default-on training on customer data is a red flag for enterprise deployments), and documents the route by which personal data reaches the vendor.
• Third-party labeling service. The service’s workforce sees raw data during labeling. Personal data handling by the labeling operation must meet the same standards as internal handling, and the practitioner should verify the workforce’s training and the service’s data-handling certifications.

Section 9 of this credential covers the wider third-party readiness regime; here the practitioner records the privacy-specific terms and any privacy-specific residual risk.

Building the minimization memo

The minimization memo is a one- to two-page document the readiness practitioner produces per use case. Its purpose is to make the minimization analysis explicit and reviewable. The memo’s structure:

1. Purpose specification. The AI use case, in enough detail that data necessity can be tested against it.
2. Data inventory. Each field in scope, with its sensitive-data class(es).
3. Necessity justification. Per field, a one-sentence rationale for why the field is required. Fields without a rationale are minimization failures and should be dropped.
4. Retention justification. Per dataset, the retention window and the reason. “We might need it” is not a reason.
5. Residual risk. After minimization, the residual privacy risk, with named mitigations (pseudonymization, access control, differential-privacy training, or others).
6. Signoff. DPO or privacy lead, data steward, and ML engineer, each with a basis statement.

A well-authored memo is short and specific. A lengthy memo usually indicates the use case is trying to cover too much and should be re-scoped.

Cross-references

• COMPEL Core — Data governance for AI (EATF-Level-1/M1.5-Art07-Data-Governance-for-AI.md) — the governance frame within which privacy controls sit.
• COMPEL Core — Process pillar domains: use cases and data (EATF-Level-1/M1.3-Art04-Process-Pillar-Domains-Use-Cases-and-Data.md) — the 20-domain maturity model’s treatment of the data pillar, including privacy as a sub-domain.
• AITM-DR Article 4 (./Article-04-Data-Lineage-Provenance-and-Documentation.md) — the documentation discipline that produces the minimization memo and the retention justification.
• AITM-DR Article 7 (./Article-07-Bias-Relevant-Variables-and-Subgroup-Coverage.md) — the protected-attribute handling that privacy controls extend.
• AITM-DR Article 10 (./Article-10-Drift-Monitoring-Incident-Classification-and-Sustainment.md) — the incident-response discipline that covers data-privacy incidents.

Summary

AI extends the privacy threat model with memorization, attribute inference, re-identification, and prompt-archive leakage. The minimization duty under GDPR Article 5 and EU AI Act Article 10 becomes operational through purpose specification, per-field necessity justification, and retention justification — the three-test minimization memo. Four privacy-preserving techniques — differential privacy, federated learning, synthetic data, and selective redaction — each apply in specific contexts with named limits. The Italian Garante’s 2024 ChatGPT decision provides a three-test framework (lawful basis, transparency, rights-affected design) that every AI readiness scorecard should apply to every dataset in scope.