DiD’s dominance in AI evaluation reflects a specific enterprise reality: most enterprise-scale AI features are not suitable for A/B tests — they are rolled out by geography, tier, or wave, and the rollout plan produces exactly the cross-unit, over-time variation DiD exploits. This article teaches the DiD specification, the parallel-trends check, small-panel inference, and the rollout-design decisions an AI value lead should make to preserve DiD identification.
The DiD specification
The canonical two-period, two-group DiD compares outcome changes between treated and control. Let $Y_{it}$ be the outcome for unit $i$ in period $t$. The DiD estimate is:
$$\hat{\delta}{DiD} = (\bar{Y}{T, post} - \bar{Y}{T, pre}) - (\bar{Y}{C, post} - \bar{Y}_{C, pre})$$
where $T$ are treated units, $C$ control units, pre/post the periods before and after treatment.
For multi-period, multi-group deployments, the same identification is expressed in a two-way fixed-effects (TWFE) regression:
$$Y_{it} = \alpha_i + \gamma_t + \delta \cdot D_{it} + \epsilon_{it}$$
where $\alpha_i$ is a unit fixed effect, $\gamma_t$ a time fixed effect, and $D_{it}$ a treatment indicator. $\hat{\delta}$ is the DiD estimate. Recent methodological advances — Callaway and Sant’Anna, Sun and Abraham, de Chaisemartin and d’Haultfoeuille — have shown that TWFE can be biased when treatment effects vary across groups or over time; estimators that average group-time treatment effects are now standard practice.1 An AI value practitioner does not need to build those estimators but should know to ask the analytics team whether they are used.
The parallel-trends assumption
DiD stands or falls on parallel trends. Treated and control units must have been moving in parallel — not necessarily at the same level, but with the same slope — before treatment. When pre-trends diverge, the DiD estimate captures both the treatment effect and the pre-existing divergence, and the result is biased.
The standard check is to plot outcome trajectories for treated and control across the pre-treatment window. Visual inspection catches most pre-trend violations. A more formal check runs a regression of outcome on a linear time trend interacted with treatment status over the pre-treatment window; a significant coefficient indicates a pre-trend.
Placebo tests extend the check: pretend a false treatment date in the pre-treatment window, re-estimate DiD, and confirm the placebo effect is near zero. Placebo-test publication has become standard in peer-reviewed DiD economics papers and is spreading to enterprise AI evaluations.
When pre-trends fail, the DiD design is invalid. The analyst can pivot to an event-study design with linear-trend controls, or to a matched-DiD design that pre-matches treated and control units on their pre-treatment trajectories. Neither salvages a truly non-parallel situation; in some cases the honest answer is “the design does not work; we need a different counterfactual.”
Small-panel inference
Enterprise AI rollouts often have small numbers of treated units — five geographies, eight teams, twelve tenants. Classical DiD standard errors assume large panels and under-estimate uncertainty with few clusters. The practitioner must use small-panel-robust inference: cluster-robust standard errors with wild-cluster bootstrap, or permutation-inference approaches (Abadie et al., 2015). Ignoring the small-panel correction produces overconfident p-values that a peer-reviewer or auditor will correctly challenge.
A rule of thumb: with fewer than ten clusters on either side of the treatment/control split, switch from standard cluster-robust standard errors to wild-cluster bootstrap or permutation inference. The AI value lead does not execute the statistical code but verifies the team is using small-panel-appropriate methods.
Rollout design to preserve DiD
Unlike A/B tests — where the design choice is upstream of rollout — DiD design is often downstream. The rollout has already been planned; the analyst must work with the rollout shape that product and operations leadership have chosen. This creates an opportunity and a risk. The opportunity is that the AI value lead can influence rollout design to improve DiD identification. The risk is that uninformed rollout choices destroy identification before the analyst has a chance to estimate anything.
Three rollout patterns preserve DiD identification well.
Pattern 1 — Treatment-timing variation
The ideal: multiple treated units, receiving treatment at different times, with the ordering not driven by outcome expectations. A copilot that launches Germany in Q1, France in Q2, Italy in Q3 produces three pre-treatment windows and three post-treatment windows. The timing variation supports event-study analysis, which yields dynamic treatment-effect estimates over time since treatment.
The parallel Microsoft 365 Copilot geography rollouts published across 2023–2024 have the shape of this pattern. The Work Trend Index reports document the rollout sequence; subsequent independent academic analysis has used the timing variation for DiD identification.2
Pattern 2 — Ring deployment
Common in enterprise software: a pilot ring (1–2 units) receives the feature first, followed by an early-adopter ring, a general-availability ring, and a late-adopter ring. If ring assignment is not driven by outcome expectations, each ring transition provides a DiD opportunity. The early-adopter transition is the cleanest because pilot-ring units often self-select in ways that threaten identification for pilot-vs-early-adopter comparisons.
Pattern 3 — Geography splits
The AI feature deploys in some geographies before others. If the geography sequence is driven by operational factors (language support, data residency, regulatory readiness) rather than expected outcome, the geography split is valid for DiD. If the sequence is driven by expected outcome (“we’ll launch in the highest-performing markets first”), the design is compromised.
The geography-split pattern is especially powerful when treated and untreated geographies are otherwise similar — same product, same customer base, same pricing. Dutch, Belgian, and German markets for a Benelux retailer; California and Washington for a West-Coast US retailer; paired English-speaking and French-speaking Canadian provinces.
Rollout patterns that destroy DiD
Three patterns destroy identification.
Same-day global rollout. No untreated units ever exist post-launch. DiD is infeasible. Fall back to synthetic control (Article 22) if a plausible donor pool exists, or pre/post with explicit disclosure.
Outcome-driven sequencing. Treating “the markets we think will benefit most” first makes the treated sample non-comparable to untreated markets. Parallel trends fail. No statistical fix.
Rollout reversal. Rolling the feature back from some treated units partway through destroys the standard DiD model. Estimators that allow for treatment switching (Callaway–Sant’Anna with “unknown later-treated” controls) can sometimes salvage it, but identification is weaker.
The CFO objection
Every DiD result faces the same CFO question: “Could something else have happened at the same time that explains the result?” The analyst’s answer structure is four parts. First, show pre-treatment parallel trends. Second, show the placebo test. Third, enumerate candidate alternative explanations — macroeconomic shift, competing product launch, seasonal pattern, regulatory change — and show robustness to each. Fourth, disclose the alternative explanations that cannot be ruled out.
The fourth part is where analyst discipline matters most. A DiD that cannot rule out a coincident macro shift is a weaker counterfactual than a DiD that can; the VRR must say so in section 3. An Uber pricing DiD published in 2017, later re-analyzed in academic work, made this disclosure discipline part of the citation pattern enterprise AI practitioners now follow.3
Cross-reference to Core Stream
EATP-Level-2/M2.5-Art02-Designing-the-Measurement-Framework.md#causal-design— practitioner introduction to quasi-experimental designs.EATP-Level-2/M2.5-Art10-From-Measurement-to-Decision.md— DiD results into decision architecture.
Self-check
- A pre-treatment parallel-trends plot shows treated units rising and control units flat. What does this mean for the DiD estimate, and what are the options?
- A rollout is sequenced “highest-expected-benefit markets first.” Why is DiD compromised, and what could salvage identification?
- Eight clusters total (3 treated, 5 control). What inference method is appropriate, and why?
- CFO asks: “Could a macroeconomic recession explain the DiD result?” Structure your four-part answer.
Further reading
- Callaway and Sant’Anna, Difference-in-Differences with Multiple Time Periods, Journal of Econometrics (2021).
- Roth et al., What’s Trending in Difference-in-Differences? A Synthesis of the Recent Econometrics Literature, Journal of Econometrics (2023).
- Angrist and Pischke, Mostly Harmless Econometrics (Princeton, 2009), Ch. 5.
Footnotes
-
Jonathan Roth, Pedro H. C. Sant’Anna, Alyssa Bilinski, and John Poe, What’s Trending in Difference-in-Differences? A Synthesis of the Recent Econometrics Literature, Journal of Econometrics (2023). https://doi.org/10.1016/j.jeconom.2023.03.008 ↩
-
Microsoft Corporation, Work Trend Index Special Report: What Can Copilot’s Earliest Users Teach Us About Generative AI at Work? (2023). https://www.microsoft.com/en-us/worklab/work-trend-index ↩
-
Peter Cohen, Robert Hahn, Jonathan Hall, Steven Levitt, and Robert Metcalfe, Using Big Data to Estimate Consumer Surplus: The Case of Uber, NBER Working Paper 22627 (2016). https://www.nber.org/papers/w22627 ↩