Course Architecture
The course builds a complete pipeline from defining causal effects to estimating them from data:
Final Exam: What's on It
Per the Review 2 deck (Adam Kelleher, May 6, 2026): the test focuses on the second half, but you also need graphs and d-separation. Application-focused, like the midterm.
Methods tested:
- Instrumental Variables (L7) — Wald/2SLS, exclusion restriction, weak instruments, LATE
- Front-Door Criterion (L8) — three graph conditions, FDC adjustment, per-mechanism shortcut
- Regression Discontinuity (L9) — sharp & fuzzy, local linear regression, density check
- Difference-in-Differences (L9) — parallel trends, ATT, time-varying controls
- Interrupted Time Series (L9) — no-confounding assumption, counterfactual model
- Panel & pooled cross-sectional designs (L10–11) — ever-treated, MCNNM
- Selection bias adjustment (L10–11) — post-stratification, IPSW, time-series s-bias
For each method, know:
- The graph assumption required to use it.
- How to recognize when a system satisfies it.
- Each estimator's additional assumptions.
- How to apply it (the code). Open-book won't help if you haven't practiced.
- Qualitative questions describing DGPs, which you'll need to connect to causal assumptions (similar to the last homework).
- A True/False section testing your understanding of inference methods (especially when their assumptions are justified).
Key Assumptions Across All Methods
L1: Foundations of Causal Inference
The individual causal effect $\delta_i$ is the difference between what would happen under treatment vs. control for the same unit.
$\text{ATT} = E[Y^{(D=1)} - Y^{(D=0)} \mid D=1]$
$\text{ATC} = E[Y^{(D=1)} - Y^{(D=0)} \mid D=0]$
$\text{ATE} = P(D=1)\cdot\text{ATT} + P(D=0)\cdot\text{ATC}$
Randomization check: Verify treatment/control balance using binomial probability bounds.
Poisson: Assign Poisson(1) weight to each row. Equivalent to standard bootstrap for large $n$.
Dependent-samples: Assign Poisson(1) weights at the cluster level (e.g., per user). Correctly accounts for within-cluster correlation. SE can be 2-3x larger than naive.
The do() Operator
$P(Y \mid do(X=x))$ is the interventional distribution — the distribution of $Y$ if we physically set $X=x$ for everyone. Graphically: delete all edges into $X$ and set $X=x$.
$P(Y \mid X=x)$ is the observational conditional — the distribution of $Y$ among units that happen to have $X=x$. This includes confounding.
Concrete Example: Job Training (NSW / LaLonde 1986)
The National Supported Work program offered job training to disadvantaged workers. For a single trainee Maria, $Y_i^{(D=1)}$ is her earnings 18 months after training; $Y_i^{(D=0)}$ is what she would have earned without training. We see one but never both — the fundamental problem of causal inference.
The naive estimator $E[Y \mid D{=}1] - E[Y \mid D{=}0]$ compares trainees to all non-trainees and reports a negative effect. But trainees were unemployed and disadvantaged when they enrolled — they weren't comparable to the general workforce. The selection-bias term $E[Y^{(0)} \mid D{=}1] - E[Y^{(0)} \mid D{=}0]$ is large and negative: trainees would have earned less anyway. Once we control for baseline earnings and demographics, the ATT is positive and economically meaningful.
What this illustrates: we never see both potential outcomes for the same person; the naive estimator includes selection bias; ATT $\neq$ ATE when treatment is self-selected.
L2: Data Generating Processes & Causal Graphs
Functional (Structural Equations): $X_i = f(\text{Parents}(X_i), \epsilon_i)$ where $\epsilon_i$ are independent noise terms.
Both are statistically equivalent — they produce samples from the same joint distribution. The functional approach makes interventions easier to reason about (just replace an equation).
Graph: $Q \to X$, $Q \to Y$ (no edge $X \to Y$).
Naive: $E[Y|X=x]$ shows a peak at 13 words. Stratified: $E[Y|X=x, Q=q]$ shows NO relationship — the correlation was entirely spurious.
Three Building Blocks of Causal Graphs
Conditioning on Z: $X \perp Y \mid Z$ (path is BLOCKED).
Light switch analogy: If you know the current is flowing ($Z$), knowing the switch position ($X$) tells you nothing new about the light ($Y$).
Caution: If $Z$ is a mediator of the causal effect of $X$ on $Y$, conditioning on $Z$ will make $X$ appear to have no effect — even though it does!
Numerical Example: Smoking → Tar → Cancer
$X$ = smoking, $Z$ = tar deposits, $Y$ = lung cancer.
| cor(X, Y) | cor(X, Y | Z) | |
|---|---|---|
| Full population | 0.40 (X and Y correlated) | 0.01 (independent given Z) |
Smoking affects cancer only through tar. Fixing tar, smoking tells you nothing new about cancer. But this also means: conditioning on a mediator kills the causal signal!
Conditioning on Z: $X \perp Y \mid Z$ (path is BLOCKED).
Lemonade & crime: Temperature ($Z$) causes both lemonade sales ($X$) and crime ($Y$). They're correlated but neither causes the other.
Numerical Example: Loyalty → Discount & Purchases
$Z$ = loyalty tier, $X$ = received discount, $Y$ = purchase rate. From the BDA example:
| Population | P(Discount) | Purchase Rate | Naive cor(X, Y) |
|---|---|---|---|
| All customers | 50% | — | 0.44 (inflated!) |
| Low loyalty only | 20% | 0.20 / 0.40 | 0.20 (true effect) |
| High loyalty only | 80% | 0.60 / 0.80 | 0.20 (true effect) |
Conditioning on loyalty (Z) removes the confounding. Within each stratum, the true causal effect of 0.20 emerges.
Conditioning on Z: $X \not\perp Y \mid Z$ (path is OPENED!).
University admission: Math ($X$) and writing ($Y$) skills are independent. But among admitted students ($Z=1$), they become negatively correlated.
Numerical Example: Math & Writing → Admission
$X$ = math score, $Y$ = writing score, $Z$ = admitted (Z=1 if $X + Y > $ threshold).
| Population | cor(Math, Writing) | N |
|---|---|---|
| All applicants | 0.012 (independent) | 10,000 |
| Admitted only (Z=1) | -0.290 (negative!) | 2,400 |
Among admitted students, low math implies high writing. Conditioning on the collider created a spurious negative association that doesn't exist in the full population.
Key rule: Conditioning on a collider (or its descendants) OPENS a previously blocked path.
Interactive: Three Building Blocks
Click a structure to see how information flows. Toggle conditioning on Z to see the path open or close.
Summary Table
| Structure | Unconditional | Conditioning on middle node |
|---|---|---|
| Chain: $X \to Z \to Y$ | $X \not\perp Y$ | $X \perp Y \mid Z$ (BLOCKS) |
| Fork: $X \leftarrow Z \to Y$ | $X \not\perp Y$ | $X \perp Y \mid Z$ (BLOCKS) |
| Collider: $X \to Z \leftarrow Y$ | $X \perp Y$ | $X \not\perp Y \mid Z$ (OPENS) |
Concrete Example: Berkeley Graduate Admissions (1973)
Berkeley was sued because women were admitted at a lower aggregate rate than men (35% vs 44%). Looking within department, women were admitted at equal or higher rates than men. Same data, opposite conclusion.
The graph: Gender → Department → Admission. Women applied disproportionately to humanities; men to engineering. Humanities had lower admit rates than engineering for everyone. Department is a fork on the back-door path Gender ← Department → Admission. Without conditioning on Department, the spurious gender→admission association is exposed; conditioning closes the back-door and the discriminatory signal disappears.
Bonus collider: in hospital data, smoking and car accidents are independent in the population, but among hospitalized patients they correlate negatively — Smoking → Hospital ← Accidents is a collider, and selecting on hospitalization opens the path. This is selection bias dressed as a graph.
What this illustrates: the fork building block creates Simpson's paradox; the collider building block creates selection-induced association.
L3: d-Separation & Back-Door Criterion
- A chain or fork node that is IN $Z$ (blocking), OR
- A collider node that is NOT in $Z$ and has no descendant in $Z$
- No descendant of $X$ is in $Z$
- $Z$ blocks all back-door paths from $X$ to $Y$ (paths that start with an arrow INTO $X$)
A "back-door path" is any path from $X$ to $Y$ that has an arrow pointing into $X$ (i.e., $\leftarrow X$). These are non-causal paths that create confounding.
This converts the interventional distribution into observational quantities we can estimate from data.
Finding Back-Door Sets: Step-by-Step
Interactive: d-Separation Exercise
Click on nodes to add/remove them from the conditioning set Z. The visualization shows which paths are blocked or open, and whether X and Y are d-separated.
Stratification & Coarsening
Stratification = Direct BDA Implementation
For each stratum $z$, compute $E[Y|X=x, Z=z]$, then weight by $P(Z=z)$.
Coarsening trade-off: Fine strata = less bias, more variance (sparse data per cell). Coarse strata = more bias (residual confounding), less variance (more data per cell).
Voter outreach example: County-level stratification (3000+ strata) was too fine with $N=2000$. State-level (50 strata) gave good estimates despite some residual confounding.
Concrete Example: Elite College & Earnings (Dale–Krueger 2002)
Does attending an elite college cause higher lifetime earnings? Confounders include high-school GPA, SAT, family income, parents' education, and unobserved motivation/ambition. The back-door criterion says: find a $Z$ that blocks all paths $X \leftarrow \cdots \to Y$.
A naive controlled regression on observed covariates leaves an "elite premium" of ~7% — but motivation is still flowing through an unblocked back-door. Dale & Krueger's clever fix: condition on the set of colleges a student applied to and was admitted to. Two students with the same admission set were similarly motivated and prepared, so the admission set acts as a near-perfect proxy for the unobserved confounder. Within admission-set strata, the elite-vs-non-elite earnings gap shrinks to near zero.
What this illustrates: the back-door criterion is a graph claim, not a data claim — domain knowledge supplies the graph; there's no statistical test for it; creative researchers find proxies that block paths through unmeasured confounders.
L4: Conditioning-Based Methods
Problem: With 5 confounders and 5 bins each: $5^5 = 3125$ strata. Most will be empty with moderate $N$.
App example: Naive estimate was $-0.04$ (wrong sign!). Exact matching with coarse bins recovered $+0.06$ (true = $0.10$). Residual bias from coarsening.
Caliper: Reject matches where distance exceeds a threshold. Trades off coverage for match quality.
Balance assessment: After matching, check that the distribution of $Z$ is similar between treated and matched controls. Use standardized mean difference statistic.
Checking support: (1) Estimate propensity score $e(Z) = P(X{=}1|Z)$ via logistic regression. (2) Plot overlapping histograms for $X{=}1$ vs $X{=}0$. (3) Flag scores below 0.05 or above 0.95 — these create instability.
Example: In the fork $X_1 \leftarrow X_2 \to X_3$, conditioning on continuous $X_2$ gives $\beta_{X_1}=0$ (correct). Discretizing $X_2$ into 5 bins gives $\beta_{X_1}=0.087$ (biased!).
Voter example: County contact rate was NOT a confounder (it affected $X$ but not $Y$ directly). Stratifying by it amplified bias: estimate went from 0.068 to 0.072 (true = 0.05).
Rule: Only condition on variables that satisfy the back-door criterion. Blindly "controlling for everything" is dangerous.
Propensity Score & Support: Visual Guide
Interactive Propensity Score Overlap
Adjust the slider to see how treatment assignment strength affects overlap. When treatment is nearly deterministic (high separation), support fails and causal estimation breaks down.
What Propensity Score Regions Mean
| $e(Z)$ region | Treated unit ($X=1$) | Control unit ($X=0$) | Implication |
|---|---|---|---|
| Near 0 | Very unusual — almost nobody like them is treated | Typical — most similar people are untreated | High IPW weight for treated; informative but unstable |
| Near 0.5 | Even odds, well-represented | Even odds, well-represented | Ideal — best overlap, most stable estimates |
| Near 1 | Typical — most similar people are also treated | Very unusual — almost nobody like them is untreated | High IPW weight for control; informative but unstable |
How Each Method Breaks Down Without Support
| Method | What happens |
|---|---|
| Stratification | Empty cell — can't compute stratum effect |
| Matching | No match found — can't construct counterfactual |
| IPW | Weight $= 1/P(X|Z) \to \infty$ — one unit dominates the estimate |
| Regression / ML | Extrapolates — fills in the gap, but trusts model where there's no data to verify |
Three Overlap Scenarios
Both distributions cover the same propensity score range. Support holds everywhere. All methods work.
Overlap in the middle, not at tails. Options: trim to overlap region (changes estimand) or extrapolate (trust model).
Groups completely separated. Treatment is essentially deterministic given Z. No method can help.
Matching Estimator Formulas
After matching, construct potential outcomes for each unit:
For ATT: only match the treated units. For ATC: only match the control units.
How Propensity Score Matching Works
Click Step Through to walk through the matching process one treated unit at a time. Each treated unit (orange) gets paired with the nearest control unit (blue) by propensity score.
Key Observations
At $e(Z)=0.3$, there may be 10 controls but only 2 treated. That's OK — we match from treated to control, not group-average. Controls can be reused (with replacement).
A control unit with $e(Z) = 0.8$ had an 80% predicted chance of being treated — but wasn't. They're rare and valuable: they look like treated units but didn't get treatment.
Density is a smoothed histogram — it shows concentration of units, not raw count. A tall peak at 0.7 means many units cluster there. The area under the curve integrates to 1.
$e(Z)=0.95$ means 95% chance of treatment given covariates. So most units there are treated. Few controls exist — matching gets hard. This is where support problems live.
Why Matching on e(Z) Is Sufficient
The Rosenbaum-Rubin theorem says: if conditioning on all of Z removes confounding, then conditioning on the single scalar $e(Z) = P(X{=}1|Z)$ does too.
Click on any propensity score band to see the units inside it. Despite having very different covariate profiles, they share the same treatment probability — so within each band, treatment is as good as random.
The Compression Intuition
Many different covariate profiles map to the same propensity score. A young, high-income, urban person and an old, low-income, rural person can both have $e(Z) = 0.4$. What matters is not why they have that probability, but that they do. Within this group, treatment assignment carries no information about Z — confounding is gone.
Distant Propensity Scores → Systematically Different People
Drag the treated unit along the propensity score axis to see how the best available control match changes — and why matching across large PS distances produces biased comparisons.
Concrete Example: Right-Heart Catheterization (Connors et al. 1996)
5,735 ICU patients; some catheterized (RHC), some not. The naive comparison: catheterized patients died more (38% vs 31%). But sicker patients were catheterized — confounding by indication.
Adjustment set $Z$: ~50 covariates (age, APACHE severity, lab values, primary diagnosis). Direct stratification was infeasible. The team used propensity-score matching: estimate $\hat e(Z) = P(\text{RHC} \mid Z)$ via logistic regression, then match each catheterized patient to a non-catheterized patient with similar $\hat e(Z)$. Patients with $\hat e(Z) > 0.95$ had no comparable controls and were excluded — a hard positivity failure that the propensity score made visible.
After matching, RHC still increased mortality by ~7 percentage points. The result drove practice change in critical-care medicine.
What this illustrates: high-dim $Z$ requires PSM rather than direct stratification; propensity scores expose positivity violations as a side effect; clean back-door analysis can overturn naive correlations.
L5: Weighting, Regression & ML Methods
The ATE is computed as two separate sums, one per group, then subtracted:
Reading the notation
| Symbol | Meaning |
|---|---|
| $i: x_i = 1$ | Sum only over treated units (a filter on the loop) |
| $i: x_i = 0$ | Sum only over control units |
| $N$ | Total sample size (both groups), NOT group size |
| $e(Z_i)$ | $P(X{=}1 \mid Z_i)$ — propensity score of unit $i$ |
| $1 - e(Z_i)$ | $P(X{=}0 \mid Z_i)$ — probability of being untreated |
The general weight rule
The denominator is always the probability of what actually happened:
Units are "surprising" when their actual treatment status was unlikely given their covariates — the denominator is small, so the weight is large.
Concrete example: Control + High Loyalty
| Property | Value |
|---|---|
| Actual status | $x_i = 0$ (control — not treated) |
| Covariates | $Z_i = \text{High Loyalty}$ |
| Propensity | $e(Z_i) = P(X{=}1 \mid Z{=}\text{High}) = 0.80$ |
| Weight denom. | $P(X{=}0 \mid Z{=}\text{High}) = 1 - 0.80 = 0.20$ |
| Weight | $w_i = 1 / 0.20 = \mathbf{5.0}$ |
| Interpretation | 80% of similar people got treated, but this unit didn't — surprising, so highly informative. Represents 5 similar people. |
Works for any function: Replace $y_i$ with $f(y_i)$ to estimate $E[f(Y) \mid do(X=x)]$ — not just means, but variances, quantiles, etc.
IPW vs BDA: Two Paths to the Same Answer
BDA and IPW are algebraically equivalent — they just rearrange the same adjustment formula. Toggle between them to see how.
For linear model $Y = \beta_0 + \beta_X X + \beta_Z Z + \epsilon$, claim: $\beta_X$ is the ATE.
Step-by-step: connecting regression to BDA
Write out the conditional expectation for each treatment value at a fixed $Z = z$:
$E[Y|X{=}0, Z{=}z] = \beta_0 + \beta_X(0) + \beta_Z z$
Subtract (the effect of treatment at this specific $z$):
$\beta_0$ cancels. $\beta_Z z$ cancels. What's left is $\beta_X$ — and it's the same for every value of $z$. The linear model assumes the treatment effect doesn't depend on Z.
Plug into BDA: $\sum_z \beta_X \cdot P(Z{=}z) = \beta_X \cdot 1 = \beta_X$ ✓
The naive regression trap (omitting Z)
Regress Y on X without Z: $Y = \alpha_0 + \alpha_X X + \epsilon$
Bias = how much Z affects Y ($\beta_Z$) × how different treated/control groups are on Z.
"$\delta$ varies with Z" means the treatment works differently for different people:
| Z (loyalty) | Effect of discount |
|---|---|
| Low loyalty | $\delta = +0.30$ (big boost — they needed the nudge) |
| High loyalty | $\delta = +0.10$ (small boost — would have bought anyway) |
The subtraction doesn't cancel cleanly: $E[Y|X{=}1, Z{=}z] - E[Y|X{=}0, Z{=}z] = \delta(z) \neq \text{constant}$
Constant-effect OLS gives a weighted average of effects — but weighted by Z's distribution within the treated group, not the population. So $\beta_X \neq$ ATE.
Two fixes
Fix 1 — Interaction term: $Y = \beta_0 + \beta_X X + \beta_Z Z + \beta_{XZ} X \cdot Z + \epsilon$. Then $\delta(z) = \beta_X + \beta_{XZ} z$.
Fix 2 — BDA predict-and-average: handles any functional form automatically.
BDA Predict-and-Average: Interactive
BDA works with any model. Toggle the effect type and walk through the 4 steps. The visualization shows the model's predictions under intervention.
Advanced Regression Methods
The danger: imposing a functional form
All methods use data. The specific danger with regression is that it imposes a functional form (linear, polynomial, etc.) and extends it into regions with no data. Example:
"Y increases linearly with Z in [0.2, 0.8] where I have data, so it must continue linearly at Z = 0.95."
But maybe the relationship curves, saturates, or reverses there. There's no data to check.
Lack of common support
Where $P(X{=}1|Z) \approx 0$ or $\approx 1$, one group has no units. We have no empirical basis for the counterfactual. Regression manufactures one from assumptions; matching and IPW refuse.
Hierarchy of honesty
| Method | No-overlap region | Failure mode |
|---|---|---|
| Exact matching | Drops the unit | Honest — no estimate |
| PSM with calipers | Drops the unit | Honest — no estimate |
| IPW | Weight → ∞ | Breaks loudly |
| Regression / BDA | Extrapolates | Breaks silently |
The silent failure is uniquely dangerous — you get a confident estimate and a confidence interval, with no indication that the answer rests entirely on functional form assumptions rather than data.
The model learns $E[Y \mid do(X)]$ directly.
Doubly-robust: Include $Z$ AND use IPW weights. Consistent if EITHER model is correct.
Why ML Models Break Causal Inference
ML models are biased by design. Regularization deliberately introduces systematic error to reduce variance. This is fine for prediction — but fatal for causal inference. Toggle to see why.
Double ML: Residualization
Division of labor
| Part | What | Who handles it | Requirement |
|---|---|---|---|
| $\delta X$ | Treatment effect (constant) | OLS on residuals | Valid inference, CIs |
| $g(Z)$ | Confounding (can be complex) | ML (any model) | Only needs good prediction |
The model is "partially linear" — linear in X (one coefficient $\delta$), but $g(Z)$ can be arbitrarily nonlinear. ML handles the hard part; OLS handles the easy part.
The residualization procedure
- Model $\hat{E}[Y|Z]$ — predict Y from confounders alone (any ML model)
- Model $\hat{E}[X|Z]$ — predict treatment from confounders (this is the propensity score!)
- Compute residuals: $\tilde{Y}_i = Y_i - \hat{E}[Y|Z_i]$ and $\tilde{X}_i = X_i - \hat{E}[X|Z_i]$
- Regress: $\hat{\delta} = \sum_i \tilde{X}_i \tilde{Y}_i \;/\; \sum_i \tilde{X}_i^2$
Why the residual formula = OLS slope
The formula $\hat{\delta} = \sum \tilde{X}_i \tilde{Y}_i / \sum \tilde{X}_i^2$ is the OLS coefficient for regressing $\tilde{Y}$ on $\tilde{X}$ (no intercept). Since residuals are mean-zero by construction, this equals $\text{cov}(\tilde{X}, \tilde{Y}) / \text{var}(\tilde{X})$ — the slope. One unit of "unexplained-by-Z treatment variation" causes $\delta$ units of "unexplained-by-Z outcome variation."
Cross-fitting: why it's needed
ML regularization introduces bias. If you predict residuals on the same data you trained on, that bias leaks into $\hat{\delta}$. Cross-fitting (like cross-validation) trains on K-1 folds, predicts on the held-out fold. Held-out predictions are unbiased.
CATE: Conditional Average Treatment Effect
"The treatment effect for people with covariates $Z = z$." Double ML gives one number ($\delta$) for everyone. CATE gives a function — the effect varies with Z.
R-Learner procedure
- Compute residuals exactly as Double ML: $\tilde{Y}_i$, $\tilde{X}_i$
- Fit a model $\hat{\tau}(Z)$ that minimizes:
Intuition: Double ML asks "what's the single slope between $\tilde{X}$ and $\tilde{Y}$?" R-learner asks "what's the slope, and how does it change with Z?"
The model $\hat{\tau}(Z)$ can be a random forest, neural net, etc. — it learns which subgroups benefit most from treatment.
| Double ML | R-Learner | |
|---|---|---|
| Effect | $\delta$ (one number) | $\tau(Z)$ (function of Z) |
| Residuals | Same | Same |
| Final step | OLS: single slope | ML: slope varies with Z |
| Use case | "Does treatment work?" | "For whom does it work most?" |
Double ML: Interactive Residualization
Walk through the residualization steps. Watch how removing Z's influence from both Y and X isolates the causal effect.
Concrete Example: Targeted Email Lift Estimation
A retailer wants to know: did the personalized recommendation email increase purchases? Population: 2M users; 200K received the email. Treatment was assigned by an existing ML personalization model on rich features $Z$ — last-purchase recency, browsing behavior, demographics, lifetime value. Standard back-door conditioning on $Z$ is identified, but $Z$ is high-dimensional.
IPW route: fit $\hat e(Z) = P(\text{email} \mid Z)$ via gradient boosting. Reweight each treated user by $1/\hat e(Z_i)$ and each control by $1/(1-\hat e(Z_i))$. Trim weights at $\hat e \in (0.05, 0.95)$ to avoid blow-up. The weighted ATE estimates lift.
Double ML route: 5-fold split. On each held-out fold, predict $\hat g(Z) = E[Y \mid Z]$ with XGBoost and $\hat m(Z) = P(D \mid Z)$ with another model. Compute residuals $\tilde Y = Y - \hat g(Z)$, $\tilde D = D - \hat m(Z)$, then OLS $\tilde Y$ on $\tilde D$. Coefficient gives valid $\sqrt n$ inference even with ML nuisances — Neyman orthogonality protects against ML's regularization bias.
What this illustrates: IPW handles high-dim $Z$ that stratification can't; Double ML lets you use ML for nuisance functions while still getting standard CIs for the causal parameter.
L6: Practical Limitations
There is no statistical test for whether your graph is correct. Domain knowledge is essential.
- Start with a fully connected graph
- Remove edges where conditional independence is found
- Orient edges using collider detection and propagation rules
Independence tests:
| Test | Data Type | Limitation |
|---|---|---|
| Fisher-Z | Continuous (+ binary) | Assumes linear associations |
| Chi-square | Discrete | Requires discretizing continuous vars |
| KCI | Any (nonparametric) | $O(n^3)$, impractical at scale |
Key limitation: Discretizing continuous mediators causes information loss and spurious edges.
Sensitivity Analysis: Stress-Testing Your Estimate
Every method assumes the DAG is correct — that all confounders are measured and adjusted for. Sensitivity analysis asks: what if we're wrong?
Suppose you've estimated $\hat{\delta} = 0.20$ after adjusting for all observed $Z$. But an unmeasured $U$ exists with $U \to X$ and $U \to Y$, creating an unblocked back-door path. The bias follows the same structure as the naive estimator:
How much $U$ affects $Y$ ($\beta_U$) × how different the groups are on $U$ (imbalance). Your true effect is $\delta_{\text{true}} = \hat{\delta} - \text{Bias}$.
What sensitivity analysis is NOT
- Does not tell you whether an unmeasured confounder exists
- Does not fix the bias
- Quantifies your vulnerability — how much hidden confounding your conclusion can tolerate
Analogy: "My bridge holds 10 tons. Sensitivity analysis says it survives up to 25 tons. I don't know if a 25-ton truck will ever cross — but I know my safety margin."
The key output: the minimum confounder strength that would zero out the effect.
Set $\hat{\delta} - \alpha = 0$ and solve. The breakdown value $\alpha^*$ is the confounding magnitude needed to make the estimated ATE vanish.
The shifted-outcome procedure (from the course)
- Start with your estimated ATE (e.g., $\hat{\delta} = 0.20$)
- Simulate an unmeasured confounder by shifting outcomes: $Y^{(\alpha)} = Y - \alpha \cdot (2D - 1) \cdot P(D = 1{-}d \mid Z)$
- Re-estimate the treatment effect with the shifted $Y^{(\alpha)}$
- Increase $\alpha$ until $\hat{\delta}^{(\alpha)} = 0$
- That $\alpha^*$ is the breakdown point
Interpreting the breakdown point
| Breakdown value | Interpretation | Verdict |
|---|---|---|
| Very large | Need a confounder stronger than anything observed | Robust |
| Small | A mild unmeasured confounder would kill the effect | Fragile |
| Negative | Adding confounding strengthens the effect | Very robust |
Connection to the hierarchy of assumptions
All causal methods rest on: (1) Graph structure is correct, (2) No unmeasured confounders, (3) Support/overlap. Sensitivity analysis stress-tests assumption #2. There's no statistical test for it — only this "how bad would it have to be?" reasoning.
Breakdown Point: Interactive
Drag the slider to increase the strength of a hypothetical unmeasured confounder. Watch the estimated ATE shrink toward zero.
When to Use Which Method
→ Yes: Randomize. Naive estimator + bootstrap CIs. No DAG needed. Done.
→ No: Need IV, RDD, or diff-in-diff (not on this exam). Stop.
→ Discrete, low-dim (≤ 3 vars): Stratification (direct BDA sum)
→ Continuous, low-dim: Matching (NN/Mahalanobis or PSM). Check overlap first.
→ High-dim or mixed: go to step 4.
→ Trust linear $E[Y|X,Z]$, constant effect: OLS
→ Don't trust linearity, trust your outcome model: BDA predict-and-average
→ Don't trust outcome model, trust propensity model: IPW
→ Don't trust either alone: Doubly-robust (IPW + regression)
→ Complex $g(Z)$, need valid CIs: Double ML
→ Want treatment effect heterogeneity $\tau(Z)$: R-Learner
No matter which method: your result rests on the assumption that $Z$ is sufficient. Sensitivity analysis quantifies how much that assumption can be violated before your conclusion breaks.
One-Sentence Summary of Each Method
Concrete Example: Hormone Replacement Therapy (1990s vs WHI 2002)
Throughout the 1990s, the Nurses' Health Study and other observational studies showed HRT users had ~50% lower cardiovascular events than non-users. Doctors prescribed HRT widely for heart-disease prevention.
In 2002, the Women's Health Initiative — a large RCT — found HRT increased cardiovascular events. The observational result was wrong by sign. What confounder did the studies miss? The healthy-user effect: women who started HRT were more health-conscious in many other ways (diet, exercise, screening, SES) than non-users.
A retrospective sensitivity analysis on the original studies asks: how strong would the unmeasured confounder need to be to explain away the observed correlation? Estimates suggest a confounder $U$ with effect ~0.3 on selection and ~0.4 on outcome — about the magnitude of "is this person a regular exerciser?" — was sufficient. The breakdown point was small, and the result should have been flagged as fragile before the RCT contradicted it.
What this illustrates: graphs can be wrong; sensitivity analysis quantifies how wrong they're allowed to be; a small breakdown point is a warning sign even when the point estimate is large.
L7: Instrumental Variables
When the back-door criterion fails — because you have an unobserved confounder $U$ between treatment $X$ and outcome $Y$ — you can sometimes exploit an instrumental variable $Z$ that affects $Y$ only through $X$.
The graph: $Z \to X \to Y$ with a hidden confounder $U \to X$, $U \to Y$.
We cannot adjust for $U$ because it is unobserved. But $Z$ provides a "natural experiment" — variation in $X$ that is exogenous (not driven by $U$).
Hidden confounder $U$ = ability, motivation
- Relevance: $Z$ must affect $X$. $\text{Cov}(Z, X) \neq 0$. Testable from data.
- Exclusion restriction: $Z$ affects $Y$ only through $X$. No direct $Z \to Y$ edge. Not testable — domain knowledge only.
- Independence: $Z \perp U$. The instrument is not confounded with the outcome. Not testable.
Two of three conditions are untestable. IV validity ultimately rests on domain knowledge and argumentation, not on statistical tests.
When $Z$ is binary (0/1):
The numerator is NOT zero when exclusion holds. $Z$ still affects $Y$ — through $X$. The reduced form $E[Y|Z\!=\!1] - E[Y|Z\!=\!0] = \alpha_1 \cdot \delta$ captures the indirect path $Z \to X \to Y$. The denominator $\alpha_1$ rescales this to a per-unit-of-$X$ effect.
Key decomposition: When exclusion holds ($\gamma = 0$): $\hat{\delta}_{IV} = \alpha_1 \delta / \alpha_1 = \delta$. When violated: $\hat{\delta}_{IV} = \delta + \gamma/\alpha_1$.
Caution: $\gamma$ is the direct $Z \to Y$ effect, NOT $\text{Cov}(Z,Y)$. The total $\text{Cov}(Z,Y) \propto \alpha_1 \delta + \gamma$ includes the indirect path. Even a valid instrument has nonzero $\text{Cov}(Z,Y)$.
For continuous $Z$, the equivalent form (under the SEM $Y = \delta D + U + \epsilon_Y$):
Three additional assumptions on top of exclusion:
- $\delta \perp \{D, Z\}$ — the effect doesn't vary with $D$ or $Z$. Satisfied if $\delta$ is constant or independently random.
- The exclusion restriction holds.
- The $Z \to D$ relationship is not weak.
Why "not weak": the estimator decomposes as
The numerator is bias from any small violation of the exclusion restriction. As $\mathrm{Cov}(D,Z) \to 0$, that bias blows up. Weak instruments amplify any leakage.
Diagnostic: first-stage F-statistic. Rule of thumb: $F \geq 10$.
The course's formal definition:
$Z$ is independent of $Y$ once you delete the edges into $D$ in the graph. Qualitatively: $Z$ causes $Y$ only through $D$.
Contaminated version (still works for TSLS): $(Z \perp Y \mid W)_{G_{\overline{D}}}$ for some set $W$. Include $W$ in both stages of TSLS.
Worked example — non-compliance: users assigned to see an ad ($Z$). Only those who open the app see it ($D$). Latent budget $U$ confounds $D$ and purchases $Y$. In $G_{\overline D}$ both $Z \to D$ and $U \to D$ are removed; only $U \to Y$ remains, $Z$ is isolated, $(Z \perp Y)_{G_{\overline D}}$ holds. ✓
When effects are heterogeneous and $\delta \not\perp \{D, Z\}$, the same Wald estimator instead identifies the LATE on a subgroup $G$:
For binary $Z$ and $D$, the revised assumptions:
- The exclusion restriction (or its contaminated form).
- $\kappa \neq 0$ for some — the instrument isn't weak.
- Monotonicity: $\kappa \geq 0$ xor $\kappa \leq 0$ for all units. (Replaces $\delta \perp \{D, Z\}$.)
If monotonicity holds with $\kappa \geq 0$, $G$ is the compliers; if $\kappa \leq 0$, the defiers. The two groups are mutually exclusive once a direction is fixed. All TSLS code is unchanged — only the interpretation shifts.
Generalization of Wald to continuous/multi-valued instruments:
Stage 2: Regress $Y$ on $\hat{X}$: $Y = \beta_0 + \beta_1 \hat{X} + \epsilon$
$\hat{X}$ is the exogenous part of $X$ — the variation predicted by the instrument, purged of confounding. The Stage 2 coefficient $\beta_1$ is the causal effect.
Why two stages? Direct regression of $Y$ on $X$ is biased by $U$. By using only the $Z$-predicted part of $X$, we break the $U \to X$ path.
With heterogeneous effects and imperfect compliance, IV estimates the LATE — the causal effect for compliers only:
Four subgroups defined by how units respond to the instrument:
| Type | $Z=0$ | $Z=1$ | Response |
|---|---|---|---|
| Compliers | $X=0$ | $X=1$ | Follow the instrument |
| Always-takers | $X=1$ | $X=1$ | Always treated |
| Never-takers | $X=0$ | $X=0$ | Never treated |
| Defiers | $X=1$ | $X=0$ | Do the opposite |
Monotonicity assumption: No defiers exist. Required for LATE to be well-defined. Always-takers and never-takers contribute nothing — the IV effect is entirely driven by compliers.
What if the exclusion restriction is slightly violated? The IV estimate becomes:
The bias $\gamma/\alpha_1$ depends on the ratio of violation size to first-stage strength. With a weak instrument (small $\alpha_1$), even a tiny $\gamma$ gets amplified massively.
Comparison to naive OLS: OLS gives $\hat{\beta} = \text{Cov}(X,Y)/\text{Var}(X) = \delta + \text{bias from }U$. OLS picks up ALL $X$-$Y$ association (causal + confounding). IV isolates the $Z$-driven variation but leaks any direct $Z \to Y$ path. When $\gamma/\alpha_1 >$ confounding bias, IV is worse than doing nothing.
Interactive: The IV Logic — Path Decomposition
The Wald estimator divides the reduced-form effect (how Z moves Y) by the first-stage effect (how Z moves X). Drag the sliders to see how paths combine and how violations propagate.
Concrete Example: Quarter of Birth & Returns to Schooling (Angrist–Krueger 1991)
Effect of years of schooling ($D$) on log earnings ($Y$). Standard OLS gives 7–9% return per year, but ability and motivation $U$ confound: more capable people get more schooling and earn more independently.
The instrument: quarter of birth ($Z$). US compulsory-schooling laws kicked in at age 16. A student born in Q1 starts school at ~6.75 years old and reaches 16 with fewer years of schooling than a Q4-born student who starts at ~6. Q1-borns end up with about 0.1 fewer years of schooling. Quarter of birth is plausibly random with respect to ability — exactly the exclusion-restriction story we need.
Wald: $\hat\delta = (\bar Y_{Q4} - \bar Y_{Q1}) / (\bar D_{Q4} - \bar D_{Q1}) \approx 0.01 / 0.1 = 0.10$. About 10% return per year — slightly higher than OLS, opposite of what you'd expect if ability bias inflated OLS upward.
Caveats: Bound, Jaeger & Baker (1995) showed the first stage was weak — small $\mathrm{Cov}(D, Z)$ amplifies any leakage of $Z$ directly into $Y$ (e.g., season-of-birth health effects). And the LATE applies only to compliers: students whose schooling was actually pushed by compulsory-schooling laws — typically lower-income students near the dropout margin. Generalizing to all students requires extra assumptions.
What this illustrates: the exclusion restriction (untestable, debated for QoB); weak-instrument bias amplification; LATE on a specific complier subpopulation, not the full ATE.
L8: Front-Door Criterion & Experiments
Two parallel themes for L8. First: another escape from unblockable confounding — the Front-Door Criterion (FDC), which identifies $P(Y \mid \mathrm{do}(X))$ via a fully observed mediator. Second: experiments and effect decomposition. The two are linked: FDC asks "what is the total $X \to Y$ effect through the mechanism?"; mediation asks "how much of that effect goes through which mechanism?"
Front-Door Criterion: Three Conditions
A set $Z$ satisfies the FDC relative to ordered pair $(X, Y)$ if:
- $Z$ intercepts all directed paths from $X$ to $Y$. Every causal arrow from $X$ to $Y$ must pass through some node in $Z$.
- No unblocked back-door path from $X$ to $Z$. The mediator's value, given $X$, is "as-if random."
- All back-door paths from $Z$ to $Y$ are blocked by $X$. Conditioning on $X$ closes any open path from $Z$ to $Y$.
Worked example: two mediators
Treatment $X$ acts on outcome $Y$ through two parallel mediators $Z_1$ and $Z_2$. An unobserved confounder $U$ affects both $X$ and $Y$ but does not reach $Z$ except through $X$. Take $Z = \{Z_1, Z_2\}$ — does it satisfy the FDC?
Trace forward: every solid arrow from $X$ to $Y$ passes through $Z_1$ or $Z_2$ — namely $X \to Z_1 \to Y$ and $X \to Z_2 \to Y$.
Back-door paths from $X$ must start by going up the dashed $X \leftarrow U$ edge. The only routes that reach $Z$ are $X \leftarrow U \to Y \leftarrow Z_1$ and $X \leftarrow U \to Y \leftarrow Z_2$ — both blocked by the collider at $Y$. This works because there's no $U \to Z$ edge in the graph; if you added one, $X \leftarrow U \to Z$ would be an unblocked back-door and condition 2 would fail.
From $Z_1$, back-door paths are $Z_1 \leftarrow X \to Z_2 \to Y$ and $Z_1 \leftarrow X \leftarrow U \to Y$ (and symmetrically for $Z_2$). $X$ sits as a fork or chain node on every one of them — conditioning on $X$ blocks them all.
All three conditions hold $\Rightarrow$ FDC adjustment via $\{Z_1, Z_2\}$ identifies $P(Y \mid \mathrm{do}(X))$ even though $U$ is never measured.
FDC: Formula, Variants, Applicability
Read it as a two-stage chain rule:
- $P(Z \mid X)$ — how the mediator responds to the treatment (this is causal because of condition 2).
- $\sum_{x'} P(Y \mid X{=}x', Z{=}z)\, P(X{=}x')$ — the back-door-adjusted effect of $Z$ on $Y$, where $X$ blocks the back-door (condition 3).
No new assumptions beyond the graph + the usual modeling assumptions: sufficient capacity and the right loss function for whichever conditional distributions you fit.
If the structural equation is additive across mediators:
Then you can estimate the effect through each mediator independently — you don't need to carry the full $Z$. Useful when one mediator is observed but others aren't, or for attribution.
Use FDC when:
- Confounder $U$ between $X$ and $Y$ is unmeasured or unmeasurable.
- You can measure the mechanism (mediator) cleanly.
- The mediator captures all of the $X \to Y$ effect (condition 1).
Classic example: price → funnel metrics → purchases. Customer budget $U$ confounds price and purchases (via personalization), but the funnel is fully measured.
Experiments & Decomposition
Randomized experiments are the gold standard for causal inference. When you control assignment, confounding disappears by design. But real experiments introduce complications: non-compliance, time-varying effects, and the challenge of decomposing how a treatment works.
Random assignment ensures $D \perp (Y^{(D=0)}, Y^{(D=1)})$:
The naive estimator is the causal effect. No adjustment needed. No unobserved confounders to worry about — because the graph has no edges into $D$ except from the randomization mechanism you control.
Apply treatment at time $t$ to a test group ($D_{it}=1$) and not to control ($D_{it}=0$). Treatment stays on; control stays off.
Define $D_i^* = 1$ if ever treated, 0 otherwise. The effect of $D_i^*$ on aggregate outcome $Y$ is identified by the experiment.
But: The effect of being treated at a specific time $D_{it}$ is harder — because outcomes carry forward (cumulative effects).
$\delta_t$ can change over time in several important ways:
- Novelty effects: Users explore a new feature more at first, engagement decays to steady-state
- Hawthorne effect: Behavior changes because subjects know they're being observed (factory lighting study)
- Cold start: Recommendation systems need data to work well — early performance understates long-run effect
To detect these, measure the effect at each time point $t$ separately. Look for stabilization. Due to smaller per-period samples, error analysis is especially important.
Once we know the ATE, we may want to understand how the treatment works — through which mechanisms.
Linear decomposition: If $Y = \beta_X X + \beta_{M} M + \epsilon$ and $M = \alpha X + \eta$, the total effect decomposes into:
- Direct effect: $\beta_X$ (treatment $\to$ outcome, not through mediators)
- Indirect effect: $\beta_M \cdot \alpha$ (treatment $\to$ mediator $\to$ outcome)
Neural decomposition: Uses neural networks to model non-linear mechanisms. Allows latent mediators and complex interactions. Identifies structural components without imposing linearity.
M (Mediator) vs Z (Instrument) — Mirror Images
| Instrument $Z$ | Mediator $M$ | |
|---|---|---|
| Graph | $Z \to X \to Y$ | $X \to M \to Y$ |
| Position | Upstream of treatment | Downstream of treatment |
| Goal | Estimate the total effect of $X$ on $Y$ | Decompose the total effect into direct vs. indirect |
| Why needed | Back-door fails (unobserved $U$) | Back-door works, but we want how the effect operates |
| Key danger | Conditioning on $Z$? No — $Z$ is exogenous | Conditioning on $M$ opens collider bias if $M$ has other causes |
The graphs are structurally similar ($A \to B \to C$) but the question is different. IV asks "what is the total $X \to Y$ effect?" Decomposition asks "how much of $X \to Y$ goes through $M$?"
When subjects don't comply with their assignment:
| Analysis | What it compares | What it estimates |
|---|---|---|
| Intent-to-Treat (ITT) | Groups as assigned | Effect of assignment — always unbiased |
| Per-Protocol | Groups by actual compliance | Effect of treatment — potentially biased (compliers ≠ non-compliers) |
| IV / LATE | Assignment as instrument for treatment | Effect for compliers — unbiased if monotonicity holds |
Connection to L7: The assignment $Z$ is a natural instrument for actual treatment $X$. The Wald estimator (ITT effect ÷ compliance rate) = LATE for compliers. ITT is the safe analysis; IV/LATE recovers the per-unit treatment effect without the selection bias of per-protocol analysis.
Interactive: A/B Test with Time-Varying Effects
See how novelty effects, cold start, and Hawthorne effects distort A/B test results over time. Toggle each phenomenon on/off.
Concrete Example: Smoking → Tar → Cancer (Pearl's Front-Door)
In the 1950s, tobacco companies argued the smoking–cancer correlation was confounded by a hypothetical "smoking gene" $U$: $U \to$ smoking and $U \to$ cancer. We can't measure $U$. But suppose we can measure tar deposits in lung tissue ($Z$).
Verify the FDC for $X = $ smoking, $Y = $ cancer, $Z = $ tar:
- $Z$ intercepts $X \to Y$: tar deposits sit on every causal path from smoking to lung cancer (assumed mechanism — biologically plausible). ✓
- No back-door $X \to Z$: the smoking gene doesn't cause tar directly — only via smoking. ✓
- Back-doors $Z \to Y$ blocked by $X$: smoking is the only common cause of tar and cancer; conditioning on $X$ closes the path. ✓
FDC adjustment then identifies $P(\text{cancer} \mid \mathrm{do}(\text{smoking}))$ from purely observational data — no need to measure $U$. This was Pearl's killing argument: even if the smoking gene exists, the front-door identifies the causal effect.
Companion: novelty effect in A/B testing
A streaming service tested a new recommendation algorithm. Day 1: +12% engagement (test vs control). Day 30: +3%. The decision depends on which number you trust. The 12% is novelty (exploration); the 3% is steady-state. The honest A/B report uses the post-stabilization number and flags the trajectory.
What this illustrates: FDC identifies $P(Y \mid \mathrm{do}(X))$ via a fully-observed mediator when the back-door fails; A/B tests must run long enough for time-varying effects to stabilize.
L9: Discontinuity Designs & Time Series
When experiments aren't possible and instruments aren't available, we can exploit natural cutoffs (regression discontinuity) or temporal structure (ITS, DID) to identify causal effects. These methods trade generality for credibility — they estimate local effects near specific boundaries or time points.
Regression Discontinuity
Treatment is completely determined by a running variable $Z$ crossing a cutoff $z_0$:
Units just above $z_0$ are nearly identical to units just below — like a local experiment.
Example: Scholarship eligibility at GPA = 3.0. Compare students at 3.01 vs 2.99.
The effect is local: it applies only at the cutoff, not to the full population.
The cutoff increases the probability of treatment but doesn't determine it completely:
This is exactly an IV problem at the discontinuity. The cutoff indicator is the instrument:
The numerator is the jump in outcomes; the denominator is the jump in treatment take-up. This is the Wald estimator at the cutoff.
Interpretation: LATE for compliers at the cutoff — units who are treated because of the cutoff.
If units can manipulate the running variable to cross the cutoff, the "as-if random" logic breaks:
- Students retaking exams to cross the GPA threshold
- Firms manipulating revenue to cross regulatory cutoffs
Diagnostic: Check for bunching (pileup) in the density of $Z$ at $z_0$. A smooth density supports the design; a spike suggests manipulation.
McCrary test: Formal test for discontinuity in the running variable's density.
From the Review 2 deck — what to verify on the exam:
- Sharpness: $P(D \mid Z = z_0^+) - P(D \mid Z = z_0^-) = 1$. Treatment really flips at $z_0$.
- Continuity of potential outcomes in the running variable at $z_0$ — if you didn't flip $D$, $Y$ would be continuous; nothing else changes at $z_0$.
- $Z$ is a back-door set at the cutoff: $Y^{(D=d)} \perp D \mid Z = z_0$.
- No manipulation of the running variable (density check / McCrary).
Estimator best practices: use a loss consistent with conditional expectations; avoid global polynomials (sensitive to data far from the cutoff); prefer local linear regression within a bandwidth.
Time Series Designs
Generalize from cross-section to $N$ units observed over $T$ time periods:
Individual effect: $\delta_{it} = Y_{it}^{(D=1)} - Y_{it}^{(D=0)}$
Data types by structure:
| Type | Units | Time | Structure |
|---|---|---|---|
| Cross-sectional | $N$ fixed | $t=1$ | One snapshot |
| Time series | $N=1$ | $T$ periods | One unit over time |
| Panel | $N$ fixed | $T$ discrete | Dense matrix $N \times T$ |
| Pooled cross-section | Varying | $T$ periods | Different units each period |
| Rolling cross-section | Varying | Continuous | Units measured at arbitrary times |
A single unit, measured over time. Treatment turns on at $t^*$. Estimate $\delta_{it}$ by extrapolating the pre-treatment trend into the post-treatment period:
The counterfactual $\hat{Y}^{(D=0)}$ is what would have happened without treatment, estimated from pre-treatment data.
Key assumption: The pre-treatment trend would have continued. This can fail due to:
- Anticipation: Unit changes behavior before treatment (e.g., switch products before a feature is removed)
- Simultaneous changes: Other events happen at $t^*$ (e.g., policy bundled with others)
- Confounding: Outside factors drive both the treatment timing and the outcome
Strategies to build confidence that the ITS effect is real:
- Examine other outcomes that should be affected — if marketing drives visits, do sign-ups also increase?
- Examine outcomes that should NOT be affected — if only new visits change, baseline user behavior should be stable
- Assess subgroup effects — do effects vary predictably? (e.g., targeted users respond more)
- Adjust for concurrent trends — control for other variables that changed around $t^*$
- Compare with other units — similar websites/products that didn't receive treatment
- Is removing the cause impactful? — when the campaign ends, does the effect go away?
Multiple units, some treated at $t^*$, some not. The DID regression:
The causal effect is $\delta = \beta_{DT}$ — the interaction term.
DID formula:
What DID removes: Constant confounders $U$ (they cancel in the first difference). $\beta_D$ captures baseline group differences; $\beta_t$ captures time trends; $\beta_{DT}$ is what's left = the treatment effect.
DID assumes the change in the confounder is the same for treatment and control groups.
If this fails — e.g., the treatment group was already trending differently — DID is biased.
Time-varying confounders: If $U_{it}$ changes differently across groups, the DID estimator includes bias: $\gamma \big(E[\Delta X_i | D^*\!=\!1] - E[\Delta X_i | D^*\!=\!0]\big)$
Fix: Add time-varying covariates $X_{it}$ to the regression: $E[Y_{it}|D^*\!=\!d, T\!=\!t, X_{it}] = \beta_0 + \beta_t t + \beta_D d + \beta_{DT} td + \gamma X_{it}$
Valid as long as $X_{it}$ is not itself affected by the treatment (not a post-treatment variable).
Interactive: Regression Discontinuity
See the sharp RD design in action. Adjust the bandwidth to see how the local estimate changes near the cutoff.
Interactive: Difference-in-Differences
See how DID isolates the treatment effect by differencing away both group differences and time trends. Toggle violations to see how bias enters.
Concrete Examples: Card–Krueger DID, Thistlethwaite RDD, California Prop 13 ITS
DID — Card & Krueger (1994), NJ minimum wage
In April 1992, New Jersey raised its state minimum wage from $4.25 to $5.05; Pennsylvania kept $4.25. Card and Krueger surveyed fast-food restaurants in NJ and PA border counties before (Feb–Mar) and after (Nov–Dec) the change. DID:
Result: NJ employment increased by ~0.6 workers per restaurant relative to PA — contradicting the naive supply-demand prediction. Parallel trends was credible because NJ and PA border counties shared regional shocks.
RDD — Thistlethwaite & Campbell (1960), original RD paper
National Merit Scholarship awards were assigned by a sharp test-score cutoff. Students one point above received a Certificate of Merit; students one point below didn't. Compare college outcomes for students at $\text{score} = c+1$ vs $\text{score} = c-1$. Local linear regression on each side, evaluated at $c$, gives the CATE at the cutoff. The famous "as-if random" intuition: at the boundary, who got the certificate is essentially a coin flip.
ITS — California Proposition 13 (1978)
Prop 13 capped property taxes, slashing local school revenue. Pre-1978 trend in per-pupil spending: rising 4%/yr. Post-1978: flat for 3 years, then resumed at 2%/yr. ITS effect = observed post-period minus extrapolated pre-trend. Caveat: anything else statewide that changed in 1978 would confound this — and it's a single unit, no control group.
What this illustrates: DID isolates a treatment effect by differencing out time-invariant confounding (parallel trends); RDD exploits a deterministic threshold for "as-if random" assignment locally; ITS extrapolates the counterfactual from a single unit's history when no comparison group exists.
L10–11: Time Series, Panel Designs & Selection Bias
When we move from cross-sectional data to time series and panels, new phenomena arise: dependence over time, regime shifts, and latent factors. The fundamental problem evolves — now we need to account for when treatment happens, not just whether it happens.
Time-Series Causal Graphs
In time series, each variable at time $t$ is a separate node in the causal graph. Dependence over time creates edges like $S_{t-1}^a \to S_t^a$ (autocorrelation).
Stationarity assumption: The structural function is the same at every time step: $S_t = f(S_{t-1})$ rather than $S_t = f_t(S_{t-1})$.
This is the time-series analog of i.i.d. — it lets us pool data across time to estimate a single causal function.
Markov assumptions simplify the graph: dependence only goes back $k$ steps, like an ARMA model: $Y_t = \beta_1 Y_{t-1} + \beta_2 Y_{t-2} + \beta_3 Y_{t-3} + \epsilon_t$.
The graph: $X_t \to S_t \leftarrow S_{t-1} \to X_t$ (and more complex interconnections with $U_t$).
Past values of outcomes ($S_{t-1}$) lie along back-door paths. Past causal states ($X_{t-1}$) can confound future outcomes.
Good news: Under stationarity, the back-door adjustment set is the same at every $t$: $Z = \{S_{t-1}^a, U_t\}$. We can apply the familiar back-door criterion — we just need to include lagged variables.
Causal Forecasting
Pure prediction: estimate $P(Y_{t+1} | X_t)$ by fitting $\hat{Y}_{t+1} = \hat{\beta} X_t$.
With a confounder $U_{t-1}$:
$Y_{t+1} = \beta_X X_t + \beta_U U_{t-1} + \epsilon_{Y,t+1}$
The naive OLS slope has omitted variable bias:
The bias depends on $\sigma_{\epsilon_X}^2$ — the exogenous variance of $X$.
If the exogenous noise $\sigma_{\epsilon_X}^2$ changes at time $t^*$ (a regime shift):
- The causal mechanism ($\beta_X$, $\beta_U$, $\beta$) is unchanged
- But the OLS bias term changes because it depends on $\sigma_{\epsilon_X}^2$
- A model fit on pre-$t^*$ data gives wrong predictions after $t^*$
Matrix Completion (MC-NNM)
In panel data with treatment switching on/off, we observe two incomplete matrices:
$Y_{it}^{(D=1)}$: observed when $D_{it}=1$, missing when $D_{it}=0$
They are exactly complementary: every cell is observed in one matrix and missing in the other.
Causal inference = matrix completion: Fill in the missing entries to construct counterfactuals.
Assume the outcome matrix is low-rank (driven by a few latent factors). Recover it by nuclear norm minimization:
where $\|Y^c\|_*$ is the nuclear norm (sum of singular values) and $M$ is the observation mask.
Key assumptions:
- Low-rank structure: Outcomes driven by a small number of latent factors
- Missing at random: Conditional on observed potential outcomes, missingness pattern is independent of unobserved potential outcomes
Solved via: Proximal gradient descent with singular value thresholding (SVT).
Latent factors $F_1, F_2, \ldots$ jointly determine both treatment assignment and outcomes. This creates selection bias.
MC-NNM handles this by exploiting shared low-rank structure: if the same factors drive both treatment and outcomes, the matrix structure encodes the confounding — and completing the matrix implicitly adjusts for it.
When does it work?
- The latent factor must drive many observed variables — otherwise we can't infer it
- Works naturally with constant confounders (factor varies across units but not time)
- Flexible with staggered adoption — different units treated at different times
In dynamic settings, past treatment $X_{i0}$ can influence a confounder $Z_{it}$, which then affects future treatment $X_{it}$ and outcome $Y_{it}$.
This creates a feedback loop between treatment and confounders. Standard back-door adjustment fails because $Z_{it}$ is both a confounder (it affects $Y$) and a mediator (it's affected by past $X$).
Solution: Sequential adjustment methods — adjust for $Z_{it}$ at each time step separately, being careful about conditioning on post-treatment variables.
- Handles staggered adoption (different treatment times for different units)
- Admits uncertainty quantification via bootstrapping or Bayesian extensions
- Combines regression across units and across time in a single framework
- Can incorporate control variables into the loss function
- Connects matrix completion (ML/RecSys) with causal inference (econometrics)
Interactive: Matrix Completion for Causal Inference
See how the potential outcome matrices relate. Click cells to toggle treatment assignment and watch the missingness pattern update.
Interactive: Regime Shift — Statistical vs Causal Forecasting
Compare how statistical (OLS) and causal models respond when the noise regime changes at $t^*$. Drag the noise slider to induce a regime shift.
Selection Bias Adjustment (L11)
Your sample is drawn under selection $S=1$ that depends on observables. The joint $P(X, Y \mid S=1)$ in your data isn't the population joint $P(X, Y)$. Adjustment is possible iff there's a covariate set $C$ that mediates selection.
If $C$ is measured both in the biased sample (alongside $X$ and $Y$) and at the population level (alongside $X$), and if $Y \perp S \mid \{X, C\}$, then $P(Y \mid X)$ is s-recoverable:
Critical: the assumption $Y \perp S \mid \{X, C\}$ is untestable — you don't have population-level $Y$ data, otherwise you wouldn't need the adjustment. Justify it with domain knowledge about what drives selection.
Three Selection-Bias Estimators
When: $X = \emptyset$ (you want a population-level expectation, not a conditional). And you only need an average, not a distribution.
Per-row weights: $w_i = P(C=c_i)/P(C=c_i \mid S=1)$. Standard survey-research move: weight your sample to match population strata (Census).
When: you have a model for $P(S=1 \mid X, C)$ — or you can recover it from joint measurements:
Optimize a weighted loss to fit any model without selection bias:
When: the population composition is roughly constant in time (or varies slowly enough to ignore) — so $N$ and $P(C=c)$ can be treated as fixed. Fit OLS to estimate $NP(C=c)$:
Use when you observe $Y_t$ over time in a biased panel and want population-level $E[Y_t]$ trajectory.
Concrete Examples: Prop 99 Synthetic Control + 2016 Polling Bias
Synthetic control — California Proposition 99 (1988)
California's 1988 cigarette tax raised prices and bundled anti-smoking ads. Did per-capita cigarette sales fall because of the policy? The clever construction: build a "synthetic California" as a weighted combination of other states (Utah, Nevada, Montana, Connecticut, Colorado) chosen so the synthetic series matches California's pre-1988 sales trajectory. After 1988, real and synthetic California diverge — by 2000, real CA sells ~26 packs/person/year vs synthetic CA's ~60. Estimated effect: −34 packs/person.
MC-NNM generalizes this: instead of restricting weights to convex combinations of donor states, let nuclear-norm minimization fill in the missing $Y^{(D=0)}$ entries directly. This handles staggered adoption (different states implement different policies at different times) without specifying donor pools by hand.
Selection bias — 2016 election polling
Most major polls in October 2016 had Clinton up by 3–4 points. Trump won the Electoral College. Why did the polls miss? Online polls and even RDD telephone polls over-represented college-educated, urban respondents. Many pollsters did weight by age and region — but missed the college / non-college divide within white voters, which had become the dominant predictor of vote choice. The right method (post-stratification) was applied to the wrong covariate set $C$.
Formula reminder: $\hat E[\text{vote share}] = \sum_c E[\text{vote} \mid C=c, S=1]\, P(C=c)$, with marginals $P(C)$ from a voter file. The result is only as good as $C$ — if $S$ depends on a variable not in $C$, you can't fix the bias.
What this illustrates: matrix completion handles staggered, partially-observed potential outcomes with a structural assumption (low rank); s-bias adjustment fixes selection only if $C$ separates $S$ from $Y$ — picking the wrong $C$ gives confidently wrong answers.
Estimand Selection: Statistical vs Causal
The question determines the estimand, not the dataset. Pick the estimand first, then choose a method that identifies it from the data you have. Conflating "what we want to know" with "what we can compute" is the most common failure mode in applied work.
Two Definitions
A functional of the observational joint distribution:
Examples: population means, conditional probabilities, quantiles. Lives at Pearl's Level 1 (seeing).
A functional of an interventional distribution:
Examples: ATE, ATT, CATE, LATE, elasticities. Lives at Pearl's Level 2 (doing).
Three Red Flags: When Causal Is Required
You need a causal estimand (not a statistical one) when any of these hold:
- Feedback. The estimand will change how $X$ is chosen. Conditional $P(Y \mid X)$ depends on the old policy; $P(Y \mid \mathrm{do}(X))$ doesn't. Any rollout, pricing change, threshold shift, or recommendation policy update is here.
- Shift. Exogenous distributions drift but structural equations are stable. Structural parameters transfer across the shift; regression coefficients don't. (Forecasting under regime change. Google Flu Trends: media-variance spike in 2013 changed the fitted coefficient even though the causal effect was constant.)
- Causal target. The question is inherently about intervention — elasticity, ATE, lift, "what if we did X?"
The 2×2 Adjustment Grid
Two questions decide the cell:
- Is the target $P(\cdot)$ or $P(\cdot \mid \mathrm{do}(X))$? (rows)
- Is there an open path between $S$ (selection) and $Y$? (columns)
| No selection bias | Selection bias | |
|---|---|---|
| Statistical target | (none — clean random sample, no adjustment needed) | s-bias adjustment (post-stratification, IPSW, time-series s-bias) |
| Causal target | Back-door / front-door / IV / RDD / DID / ITS | s-backdoor (combine selection adjustment with causal identification) |
Exam discipline: name the cell first; the method follows. The empty cell (statistical target, no selection bias) is rare in business — a clean random sample of the population needs none of this.
Estimand Fallback: When You Can't Identify the One You Want
Often you start with one estimand, realize you can't identify it from the data you have, and back off to a related one:
| Want | Can't identify because… | Fall back to |
|---|---|---|
| ATE | Heterogeneous effects + IV | LATE on compliers |
| ATE | RDD only identifies at the cutoff | CATE at $z_0$ |
| ATE | DID only valid for treated group | ATT on the treated |
| ATT in observational data | Common support fails for some treated units | ATT on the overlap region |
Each fallback narrows the population the answer applies to. Be explicit about it when reporting — "the effect on compliers near the cutoff" is a much more honest claim than "the effect."
Concrete Example: Google Flu Trends (2008–2014)
Google built a model predicting CDC flu cases from search-query volume — type "flu symptoms" or "what to take for fever," and the model added you to its estimate. From 2008 to 2012 it tracked the CDC numbers within ~2%, faster than the CDC itself. Then in January 2013, GFT predicted 11% of the population had flu; the CDC measured 6%. GFT had nearly doubled the right answer.
What broke: the relationship between search volume and flu prevalence wasn't structural — it was confounded by media coverage. In 2013, heavy news coverage of a bad flu season inflated search queries far above what people's actual flu prevalence implied. The OLS coefficient absorbed both the structural signal (sick people search more) and the media-driven amplification.
A causal forecast would have separated $P(\text{search} \mid \text{flu}, \mathrm{do}(\text{media coverage}))$ from $P(\text{search} \mid \text{flu})$. The structural mechanism stayed stable; only the exogenous distribution of media changed. Statistical models are vulnerable to this kind of shift — structural models aren't.
What this illustrates: the second of the three red flags — when exogenous distributions drift but structural equations are stable, regression coefficients don't transfer; structural parameters do. Choose the causal estimand even for a forecasting problem if the regime can shift.
Situational Drills (Final Review)
Each drill is a real business situation. None of them name an estimand up front — that's the first thing you have to decide. Three slides per drill: business problem, inference problem (what are we estimating? statistical or causal? why?), identification & estimator. Three at Level 1 (statistical), four at Level 2 (causal).
L1.1: Credit-Card Nowcast
Your hedge fund team trades a major US retailer's stock around its quarterly earnings announcement. To position before the print, you've licensed a credit-card transaction panel from a card issuer, hoping to nowcast retailer revenue. Catch: the cardholder base over-represents high-income, urban consumers. Earnings drop in two weeks.
Estimand: Statistical: $E[\text{revenue}_m \mid q]$ at the US population level.
Why statistical? No counterfactual, no feedback, no shift — just a current-state quantity, observed under biased sampling. The trade rests on the retailer's actual reported revenue, not on changing any policy.
Identification: Selection bias from sampling. With demographics $C$ measured both in-panel and at the population level (Census), assume $\text{revenue} \perp S \mid C$. s-bias adjustment applies.
Estimator: Post-stratification weighted sum.
with weights $w_i = P(C=c_i)/P(C=c_i \mid S=1)$.
L1.2: Foot-Traffic for Lease Renewal
You work at a REIT. The lease on a flagship retail property comes up for renewal. You want monthly visit counts for the last 12 months. You subscribe to a geolocation panel — anonymized SDK pings. Two issues: (1) which devices are in the panel ($S_1$) depends on apps installed; (2) ping cadence ($S_2$) varies by app and location.
Estimand: Statistical: $E[V_{P,t}]$ — expected monthly visits at the population level.
Why statistical? Renewal needs an honest count of how busy the property has been. No decision changes how visits are generated; only how visits are valued.
Identification: Two layers of selection. Adjust on demographics $Z$ for $S_1$; on $(Z, L)$ for $S_2$.
Estimator: Stage-wise inverse-propensity weighting:
Calibrate to a known ground-truth count (e.g. a nearby stadium event with reported attendance) before trusting absolute levels.
L1.3: Banner-Ad Reporting
Your e-commerce growth team launched a banner-ad campaign last week. The VP wants to know: of users who saw the banner, what fraction purchased? Catch: analytics only records sessions with dwell-time $\geq 30$s (a long-standing bot/bounce filter).
Estimand: Statistical conditional: $P(\text{purchase} \mid \text{banner shown})$ over all banner viewers.
Why statistical? No intervention — just describing what happened. But verify the VP doesn't actually want a causal estimand (lift) — they often do without realizing it.
Identification fails. $S$ (dwell-time $\geq 30$s) is on a descendant of the outcome — engaged users dwell longer and buy more. No reweighting on observed covariates recovers $P(Y \mid X)$.
Estimator: None that fixes it. Right answer: remove the filter at the source. Otherwise, give the VP a bound:
L2.1: Ad Rollout (IV)
Marketing wants to know whether to spend $10M next quarter rolling out a video-ad campaign. Last month they ran a small randomized pilot: half of users were eligible to be served the ad, but only $\sim 30\%$ of the eligible group ever opened the app and saw it.
Estimand: Causal: $E[Y^{(D=1)} - Y^{(D=0)}]$. Under heterogeneity, retreat to LATE on compliers.
Why causal? Two red flags: (1) rollout will change how often $D$ happens (feedback), (2) the question is inherently counterfactual.
Identification: Instrumental variables. $Z =$ randomized eligibility, $D =$ ad view. Exclusion restriction holds by design.
Or 2SLS. Under monotonicity, the estimate is LATE on compliers (the $\sim 30\%$ who view iff eligible).
L2.2: Pricing Decision (FDC)
Pricing team is debating dropping a SKU price by 10% next quarter. They have a year of observational sales data: every transaction has a price plus per-session funnel metrics — items viewed ($Z_1$) and time-to-checkout ($Z_2$). Customer budget $U$ confounds: discount-sensitive shoppers buy more and get shown lower prices via personalization.
Estimand: Causal: $E[Y \mid \mathrm{do}(X)]$.
Why causal? The price drop is a policy change. The relationship under the current personalization regime won't transfer to a regime where everyone gets the lower price.
Identification: Back-door could work if recommendation scores are accessible. Otherwise, since price plausibly affects purchases only through the funnel, use the front-door criterion via $\{Z_1, Z_2\}$.
Or, if the funnel is additive in $Z_1, Z_2$, the per-mechanism estimator.
L2.3: Scholarship Program (RDD)
A provost is reviewing the merit-scholarship program. Currently, GPA $\geq 3.5$ gets the scholarship automatically. The provost is considering lowering the cutoff to 3.3. A density check at 3.5 shows no pileup.
Estimand: Ideally the ATE, but the identification strategy will only get us a CATE near the existing cutoff: $E[Y^{(D=1)} - Y^{(D=0)} \mid Z = 3.5]$.
Why causal? The decision is whether to extend treatment to a new group — counterfactual.
Identification: Sharp RDD. Assignment is deterministic at GPA $= 3.5$; potential outcomes continuous in GPA at the cutoff; density check rules out manipulation.
Estimator: Local linear regression on each side of 3.5; difference of fitted values at the cutoff.
Caveat: extrapolating from CATE at 3.5 down to 3.3 needs a smoothness assumption — flag this for the provost.
L2.4: Minimum-Wage Policy (DID)
A state legislature is considering a minimum-wage hike after several nearby states raised their minimums last summer. Complication: the same treated states ended the federal pandemic UI supplement ($300/week) six months before the wage hike. Control states kept it until federal expiration. The early expiration plausibly tightened the low-wage labor market in treated states ahead of $t^*$.
Estimand: Causal ATT on the treated states: $E[Y^{(D=1)} - Y^{(D=0)} \mid D^* = 1, t > t^*]$.
Why ATT, not ATE? Most defensible target with this design. We only have unbiased $Y^{(D=1)}$ for treated states; parallel trends identifies the counterfactual specifically for them.
Identification: DID with parallel trends. Naive DID is biased: the early UI-supplement expiration created a low-wage employment trend in treated states ahead of $t^*$. Restore identification by conditioning on the time-varying control $X_{it} =$ per-state UI-supplement status by month.
The treatment effect is $\beta_{DT}$.
Drill → Cell Mapping
| No selection bias | Selection bias | |
|---|---|---|
| Statistical | — | L1.1, L1.2 (s-bias); L1.3 (fails) |
| Causal | L2.1 (IV), L2.2 (FDC), L2.3 (RDD), L2.4 (DID) | (s-backdoor) |
Exam discipline: name the cell first, the method follows.
Master Method Comparison
What Does Each Method Model?
| Method | Models outcome $E[Y|X,Z]$? | Models treatment $P(X|Z)$? | Key formula |
|---|---|---|---|
| Stratification | No | No | $\sum_z E[Y|X,Z{=}z] \cdot P(Z{=}z)$ |
| Matching (Exact/NN/PSM) | No | PSM: Yes | Pair treated/control by $Z$ or $e(Z)$, compare outcomes |
| IPW | No | Yes | $\frac{1}{N}\sum_{i: x_i=x} \frac{y_i}{P(X{=}x_i|Z_i)}$ |
| OLS | Yes (linear) | No | $Y = \beta_0 + \beta_X X + \beta_Z Z + \epsilon$; read off $\beta_X$ |
| BDA predict-and-average | Yes (any model) | No | $\frac{1}{N}\sum_i [\hat{f}(1, Z_i) - \hat{f}(0, Z_i)]$ |
| Double ML | Yes (nuisance) | Yes (nuisance) | $\hat{\delta} = \sum \tilde{X}_i \tilde{Y}_i / \sum \tilde{X}_i^2$ |
| Doubly-robust (IPW + Reg) | Yes | Yes | IPW-weighted MSE: $\sum_i \frac{(y_i - \hat{f}(x_i))^2}{P(X{=}x_i|Z_i)}$ |
Capabilities & Limitations
| Stratification | Matching | IPW | OLS | BDA | Double ML | Doubly-robust | |
|---|---|---|---|---|---|---|---|
| Continuous $Z$ | No | Via PS | Yes | Yes | Yes | Yes | Yes |
| High-dim $Z$ | No | Via PS | Yes | Risky | Risky | Yes | Yes |
| Heterogeneous effects | Yes | Implicitly | Yes | No (basic) | Yes | R-learner | Yes |
| Extrapolates? | No | No | No | Yes (danger) | Yes (danger) | Partially | Yes (danger) |
| Valid CIs with ML? | N/A | Bootstrap | Bootstrap | No | No | Yes | Depends |
| Any $f(Y)$? | Means only | Means only | Yes | Means only | Means only | Means only | Means only |
| Failure mode | Empty cells | No match | Weights explode | Extrapolation | Extrapolation | Nuisance failure | Both wrong |
When to Choose Each
| Situation | Best choice | Why |
|---|---|---|
| $Z$ is binary/few levels | Stratification | Simple, transparent, no modeling assumptions |
| Low-dim $Z$, want non-parametric | PSM + calipers | No outcome model, drops bad matches honestly |
| Need $E[f(Y)|do(X)]$ for arbitrary $f$ | IPW | Only method that estimates beyond conditional means |
| Trust linearity, constant effect | OLS | Simplest, $\beta_X$ = ATE, standard CIs |
| Any model, heterogeneous effects | BDA predict-and-average | Works with RF, NN, etc. — handles any $\delta(Z)$ |
| Complex $g(Z)$, need valid CIs | Double ML | ML flexibility + debiased inference via cross-fitting |
| Unsure which model is correct | Doubly-robust | Insurance: consistent if EITHER model right |
| Want CATE $\tau(Z)$ with ML | R-Learner | Extends Double ML residualization to heterogeneous effects |
Estimating $\delta(Z)$ (CATE)
| Method | Approach | Assumption |
|---|---|---|
| Stratified matching | $\hat{\delta}(z) = E[Y|X{=}1,Z{=}z] - E[Y|X{=}0,Z{=}z]$ | Discrete $Z$, enough data per stratum |
| OLS + interaction | $\delta(z) = \beta_X + \beta_{XZ} z$ | Linear heterogeneity |
| BDA point-wise | $\hat{\delta}(z) = \hat{f}(1, z) - \hat{f}(0, z)$ | Correct outcome model |
| R-Learner | Fit $\hat{\tau}(Z)$ to minimize $\sum(\tilde{Y}_i - \tau(Z_i)\tilde{X}_i)^2$ | Good nuisance models + cross-fitting |
The Three Assumptions Everything Rests On
Practice Problems
Back-door paths from $X$ to $Y$: (1) $X \leftarrow Z_1 \to Y$ and (2) $X \leftarrow Z_2 \to Y$.
Valid back-door sets: $\{Z_1, Z_2\}$, $\{Z_1\}$ alone blocks path 1 but not path 2, so we need both. Actually, $\{Z_1\}$ only blocks path 1. $\{Z_2\}$ only blocks path 2. We need $\{Z_1, Z_2\}$.
Check: Neither $Z_1$ nor $Z_2$ is a descendant of $X$. Both back-door paths are blocked.
No! $M$ is a descendant of $X$, violating condition 1 of the back-door criterion. Additionally, conditioning on $M$ would block the causal path $X \to M \to Y$, which we don't want.
The valid back-door set would be $\{U\}$, but $U$ is unobserved. Without observing $U$, we cannot use back-door adjustment here.
$E[Y|X{=}1, Z{=}0] = 5$, $E[Y|X{=}0, Z{=}0] = 3$, $P(Z{=}0) = 0.6$
$E[Y|X{=}1, Z{=}1] = 8$, $E[Y|X{=}0, Z{=}1] = 7$, $P(Z{=}1) = 0.4$
What is the ATE using the back-door adjustment formula?
$E[Y|do(X{=}1)] = E[Y|X{=}1,Z{=}0] \cdot P(Z{=}0) + E[Y|X{=}1,Z{=}1] \cdot P(Z{=}1)$
$= 5 \times 0.6 + 8 \times 0.4 = 3.0 + 3.2 = 6.2$
$E[Y|do(X{=}0)] = 3 \times 0.6 + 7 \times 0.4 = 1.8 + 2.8 = 4.6$
ATE $= 6.2 - 4.6 = 1.6$
Stratum-level effects: $\delta_{Z=0} = 5 - 3 = 2$, $\delta_{Z=1} = 8 - 7 = 1$.
$\text{ATE} = 2 \times 0.6 + 1 \times 0.4 = 1.6$. Same answer.
A collider $X \to Z \leftarrow Y$ blocks the path between $X$ and $Y$ by default. But conditioning on $Z$ (or selecting on $Z$) opens the path.
Intuition: Knowing the collider's value constrains the relationship between its parents. If $Z = X + Y$ and you know $Z = 10$, then knowing $X = 8$ tells you $Y = 2$. The parents become dependent given the collider's value, even though they were independent unconditionally.
Real-world: Among admitted students (collider = admission), math and writing skills appear negatively correlated, even though they're independent in the general population.
For unit with $X=1$: $w_i = 1/P(X{=}1|Z) = 1/0.8 = 1.25$
For unit with $X=0$: $w_i = 1/P(X{=}0|Z) = 1/(1-0.8) = 1/0.2 = 5.0$
The control unit at $Z$ where treatment is likely gets a much higher weight (5x vs 1.25x). This makes sense: seeing a control unit where almost everyone is treated is very informative about the counterfactual.
Constant effect: The BDA formula gives $E[Y|do(X{=}x)] = \sum_z E[Y|X{=}x, Z{=}z]P(Z{=}z)$. Plugging in the linear model: $E[Y|X{=}1,Z{=}z] - E[Y|X{=}0,Z{=}z] = \beta_X$ for all $z$. So $\delta = \sum_z \beta_X P(Z{=}z) = \beta_X$.
Heterogeneous effect: If the true model is $Y = \delta(Z) \cdot X + g(Z) + \epsilon$, fitting a constant-effect model forces $\delta(Z) \to \beta_X$. OLS minimizes MSE and produces a variance-weighted average of $\delta(Z)$, which doesn't equal $E[\delta(Z)]$ in general. The weighting depends on the propensity score distribution, biasing toward values of $Z$ where treatment assignment is most variable.
Fix: Add interaction term ($\beta_{XZ} XZ$) or use BDA predict-and-average.
- Cross-fit: Split data into $K$ folds
- Residualize: On each held-out fold, compute $\tilde{Y}_i = Y_i - \hat{E}[Y|Z_i]$ and $\tilde{X}_i = X_i - \hat{E}[X|Z_i]$ using models trained on other folds
- Estimate: Regress $\tilde{Y}$ on $\tilde{X}$: $\hat{\delta} = \sum \tilde{X}_i \tilde{Y}_i / \sum \tilde{X}_i^2$
Why cross-fitting? ML models use regularization (early stopping, depth limits, L2 penalties), which introduces bias in predictions. If nuisance models predict on their own training data, this bias leaks into the causal estimate. Cross-fitting ensures nuisance models never predict on data they were trained on, preventing overfitting bias from contaminating $\hat{\delta}$.
The breakdown point is the minimum violation of unconfoundedness needed to explain away the entire treatment effect (reduce ATE to zero).
$\alpha^* \approx 1.0$ means: an unmeasured confounder would need to create a 1.0 unit difference in expected potential outcomes between treated and control groups (within each stratum of observed confounders) to fully explain the observed effect.
Whether this is a lot or a little depends on domain knowledge. If $Y$ is measured in standard deviations, a 1.0 unit violation is quite large (the unmeasured confounder would need to shift expected outcomes by a full standard deviation). If $Y$ is in dollars, 1.0 might be trivially small.
Key insight: Sensitivity analysis doesn't tell you whether unconfoundedness holds. It tells you how robust your conclusion is to violations.
Not exact matching: $5^5 = 3125$ strata (with 5 bins each) — too many for most sample sizes. The curse of dimensionality.
Options:
- Propensity score matching: Reduces 5D matching to 1D. Fit logistic regression for $P(\text{app}|Z)$, match on the score. Check balance post-matching.
- OLS with all confounders: If linear model is reasonable. $\beta_{\text{app}}$ = ATE if constant effect.
- IPW: Weight by inverse propensity. Good if you trust the propensity model but not an outcome model.
- Double ML: Best if relationships are nonlinear. Uses flexible ML for nuisance models while giving valid CIs for the causal effect.
Always: Check support (propensity score overlap between users and non-users), and run sensitivity analysis afterwards.