Bias-Relevant Variables and Subgroup Coverage

This article defines the classes of bias-relevant variables, describes the subgroup coverage audit, addresses the handling of proxy variables, and names the two recurring case anchors — the Amazon recruiting tool retirement and the ProPublica COMPAS analysis — that every readiness practitioner should be able to reason about. The article ends with the net-fairness rule: every curation decision must be evaluated for its net effect on the protected populations, not only for the harm it was meant to address.

Where bias enters the data

Bias can enter a training set through at least four vectors:

• Historical decision bias. The labels or target values reflect past decisions that were themselves biased. A recruiting dataset labeled by ten years of historical hires reflects the hiring preferences of those ten years, biased or not.
• Sampling bias. The dataset was collected in a way that oversamples some populations and undersamples others. A customer-support dataset collected from phone calls undersamples populations that prefer chat or email.
• Measurement bias. The measurement instrument performs differently across groups. A camera-based dataset may have higher image quality for some skin tones than others, producing measurable performance gaps downstream.
• Proxy leakage. Protected attributes are not in the dataset, but other features correlate with them tightly enough to serve as proxies. A zip-code feature can proxy for race in many metropolitan areas; a purchase-category feature can proxy for gender.

The readiness practitioner audits for all four vectors. An audit that only examines direct inclusion of protected attributes misses the most pernicious class — proxy leakage — which shows up in production as disparate impact even when the model ostensibly does not use race, gender, or age.

The two case anchors

Amazon recruiting tool (2018)

Reuters reported in October 2018 that Amazon had discontinued an experimental recruiting tool after engineers found the system was penalizing résumés that contained references to women’s activities and organizations.¹ The tool had been trained on approximately ten years of historical résumés, which skewed heavily male in the technical roles being targeted. The training-data bias produced a model that down-weighted resumes patterned after the underrepresented group’s signals — a textbook historical-decision-bias cascade into proxy-feature learning.

A readiness practitioner who had assessed that dataset before training would have flagged the subgroup-coverage gap, identified the proxy risk in school names and activity descriptions, and required either a representative-sampling plan or an acknowledgment that the dataset could not support the intended use case. The engagement’s outcome would have been the same as the eventual retirement — the dataset did not support the use — but the cost would have been contained.

ProPublica COMPAS analysis (2016)

ProPublica’s 2016 analysis of the COMPAS risk-assessment system used in US criminal justice documented disparities in false-positive rates across racial groups.² Black defendants were nearly twice as likely as white defendants to be labeled high-risk and not re-offend; white defendants were more likely to be labeled low-risk and re-offend. The analysis triggered a multi-year methodological debate about which fairness definition is appropriate and whether the dataset and model can simultaneously satisfy multiple definitions.

The debate is instructive because it surfaced a theorem the practitioner must understand: in most real datasets, multiple fairness definitions cannot be simultaneously satisfied. The practitioner cannot claim to have eliminated bias; the practitioner can identify which bias vectors were analyzed, which mitigation choices were made, and why. The scorecard records the analysis, the choice, and the rationale.

Classes of bias-relevant variables

The readiness practitioner classifies every variable in the dataset into one of four categories:

1. Protected attributes. Variables named in applicable law as protected categories: race, ethnicity, gender, age, religion, disability, sexual orientation, and others depending on jurisdiction. Their direct inclusion in training data is regulated and sometimes prohibited.
2. Proxies. Variables that correlate with protected attributes strongly enough to enable proxy discrimination. Zip code, name, school, language, device type, and many others can proxy for protected attributes depending on context.
3. Intersectional factors. Combinations of variables that identify protected subgroups at higher resolution than any single variable. A combination of pregnancy-related purchase signals and location can identify individuals at sensitive life stages.
4. Neutral variables. Variables with no material correlation to protected attributes in the use case context. These are the bulk of most datasets.

The classification is use-case-specific. A variable can be neutral in one use case and a proxy in another. The readiness practitioner documents the classification per use case and re-classifies when the use case changes.

[DIAGRAM: MatrixDiagram — protected-predictive-curation-regimes — 2x2 of “Is the variable protected (or a proxy)?” (yes / no) against “Is the variable predictive of the target?” (yes / no), mapping the four quadrants to curation strategies: no action (not-protected, not-predictive), include with monitoring (not-protected, predictive), exclude and audit for proxy leakage (protected, not-predictive), careful fairness analysis required (protected, predictive)]

The subgroup coverage audit

Coverage is measured before training. The audit has three layers.

Absolute coverage

For each protected group relevant to the use case, count the records and compare against the target population. A dataset with 50 records for a group that represents 10% of the target population cannot support reliable training on that group regardless of the statistical method used downstream. The readiness practitioner sets minimum-coverage thresholds per group tied to the use case risk and the planned model complexity.

Relative coverage

Compare the group’s proportion in the dataset to its proportion in the target population. Gaps above a documented threshold are findings. The practitioner should not assume that the target population is the general population; for a B2B product, the target population is the business-customer universe, which may differ substantially from census distributions.

Intersectional coverage

Audit for combinations. A dataset with adequate coverage of women and adequate coverage of a particular ethnic group may still have inadequate coverage of women in that ethnic group. Intersectional gaps drive the failure modes that single-dimension audits miss.

The audit output is a coverage table indexed by group (and intersection) with counts, proportions, gaps, and decisions. It becomes part of the readiness scorecard.

[DIAGRAM: ScoreboardDiagram — subgroup-coverage-dashboard — table with rows for each relevant subgroup (and named intersections), columns for dataset-count, dataset-proportion, target-population-proportion, gap, status (green/amber/red), and decision (proceed / augment / down-scope), with threshold legend and sign-off line]

Selection-rate and disparate-impact analysis

Where labels carry selection decisions (who was hired, who was approved, who was flagged for review), the readiness practitioner computes selection rates per group and the disparate-impact ratio. The disparate-impact ratio is the selection rate of the less-favored group divided by the selection rate of the most-favored group; a ratio below 0.8 is the long-standing US Equal Employment Opportunity Commission benchmark for employment contexts, and is a useful reference point even outside that specific regulatory context.

The analysis is a data-side precursor to the model-side fairness analysis. If the training-set selection rates are already disparate, the model will almost certainly learn the disparity unless mitigation is deliberate. The practitioner’s scorecard records the pre-training disparity, the mitigation plan, and the expected post-training disparity (verified after training, before deployment).

The proxy-audit rule

Proxy auditing is the hardest class. The practitioner’s rule is: for every variable in the dataset, compute a mutual-information or correlation statistic with each protected attribute (where the protected attribute is available for the purpose of the audit, even if not used in training). Variables above a documented threshold are flagged as proxies. The practitioner then decides for each proxy: exclude, include with monitoring, transform to reduce the proxy signal, or accept with explicit justification.

The proxy audit requires, for the duration of the audit, access to protected-attribute data. This creates a minor paradox — to demonstrate non-discrimination, the practitioner needs the protected attributes — and is one reason EU AI Act Article 10(5) specifically authorizes processing of special-category data for bias detection, subject to safeguards including pseudonymization and limited retention.³ The practitioner’s evidence package includes the authorization basis, the safeguards applied, and the disposal of the protected-attribute data after the audit.

The net-fairness rule

Every curation decision has a net-fairness question. Dropping a feature to reduce bias on group A may inadvertently worsen performance for group B. Oversampling a minority group may cause the model to memorize that group’s records and degrade generalization. Synthetic augmentation may introduce distributional artifacts that the model learns instead of genuine signal.

The practitioner’s rule is: no curation decision is approved without an explicit net-fairness analysis covering all groups of concern, not only the group the decision was meant to address. The analysis lives in the decision log (Article 4) and is part of the scorecard’s supporting evidence.

Data augmentation for underrepresented subgroups

Where a subgroup is underrepresented, the practitioner has several remediation options, each with tradeoffs. The readiness scorecard records the option chosen and the rationale.

• Oversampling. Duplicate records from the underrepresented subgroup in training. Simple; risks overfitting on the duplicated records and can cause the model to memorize individual cases.
• Stratified sampling. Rebalance the sampling strategy so each subgroup contributes proportionally (or at a minimum floor). Preserves data integrity; may require acquiring more data for the underrepresented subgroup.
• Synthetic generation. Generate synthetic records for the underrepresented subgroup. Useful where real data is impossible or ethically problematic to acquire; requires careful validation against held-out real data to confirm the synthetic records preserve the joint distribution.
• Reweighting. Keep the sampling as is but weight training loss so underrepresented subgroups contribute proportionally. Clean computationally; does not address the information deficit from small sample sizes.
• Targeted collection. Go acquire more real data for the underrepresented subgroup through consented, governed collection. Addresses the root cause; takes time and budget.

None of these are universally preferred. The practitioner chooses based on the use case, the risk tier, the cost budget, and the ethical acceptability of each option. The scorecard records the choice and defends it.

Bias detection with protected attributes unavailable

In many real deployments, protected attributes are not stored in the operational data — by design, for privacy reasons, or because the business process never captured them. This creates a methodological tension: the practitioner needs protected attributes to audit for bias, but the organization does not retain them.

Several techniques address the tension:

• Bayesian improved surname geocoding (BISG) and related imputation. Estimate protected-attribute probabilities from names, addresses, or other features, use the estimates for audit only, and disclose the estimation method and its uncertainty.
• Audit-window collection. Collect protected attributes from a sample of individuals specifically for the audit, under explicit consent and limited retention, and use the sample to measure disparate impact.
• External-benchmark comparison. Compare the organization’s outcomes to published benchmarks (government statistics, academic studies) that include protected-attribute breakdowns.

All three have limits and each must be documented in the scorecard. A practitioner should never claim a clean bias audit with no access to protected attributes; the claim is indefensible.

Fairness definitions and their tradeoffs

The practitioner should be able to name four common fairness definitions and explain when each applies:

• Demographic parity — selection rates are equal across groups. Appropriate when the positive outcome is a benefit the organization is choosing to distribute.
• Equal opportunity — true-positive rates are equal across groups. Appropriate when the positive outcome requires a correct prediction about the individual.
• Equalized odds — both true-positive and false-positive rates are equal across groups. A stronger form of equal opportunity.
• Predictive parity — precision is equal across groups. The definition that COMPAS’s defenders argued was satisfied.

The definitions trade off. A practitioner who claims “the model is fair” without naming the definition is not producing auditable evidence. The scorecard names the definition, justifies the choice, and records the measurement.

Cross-references

• COMPEL Core — Data governance for AI (EATF-Level-1/M1.5-Art07-Data-Governance-for-AI.md) — the governance umbrella in which fairness-aware data curation sits.
• COMPEL Core — Process pillar domains: use cases and data (EATF-Level-1/M1.3-Art04-Process-Pillar-Domains-Use-Cases-and-Data.md) — the 20-domain maturity model’s treatment of the use-case and data process domains.
• AITM-DR Article 2 (./Article-02-Data-Quality-Dimensions-Extended-for-AI.md) — representativeness as a classical quality dimension extended by the fairness audit.
• AITM-DR Article 8 (./Article-08-Privacy-Sensitive-Data-Classes-and-Data-Minimization.md) — the privacy discipline that constrains protected-attribute handling during the fairness audit.

Summary

Data is where fairness is won or lost. The readiness practitioner classifies every variable into protected, proxy, intersectional, or neutral categories per use case; audits subgroup coverage at absolute, relative, and intersectional layers; computes selection rates and disparate-impact ratios where labels carry selection decisions; audits for proxy leakage with documented thresholds; applies the net-fairness rule to every curation decision; and names the specific fairness definition the model is built to satisfy. The Amazon recruiting tool retirement and the ProPublica COMPAS analysis are the two anchors that illustrate, respectively, how training-data bias produces discriminatory models and how the practitioner must choose between incompatible fairness definitions with explicit rationale.