Average treatment effect explained

The average treatment effect (ATE) is a measure used to compare treatments (or interventions) in randomized experiments, evaluation of policy interventions, and medical trials. The ATE measures the difference in mean (average) outcomes between units assigned to the treatment and units assigned to the control. In a randomized trial (i.e., an experimental study), the average treatment effect can be estimated from a sample using a comparison in mean outcomes for treated and untreated units. However, the ATE is generally understood as a causal parameter (i.e., an estimate or property of a population) that a researcher desires to know, defined without reference to the study design or estimation procedure. Both observational studies and experimental study designs with random assignment may enable one to estimate an ATE in a variety of ways.

The average treatment effect is under some conditions directly related to the partial dependence plot^[1]

General definition

Originating from early statistical analysis in the fields of agriculture and medicine, the term "treatment" is now applied, more generally, to other fields of natural and social science, especially psychology, political science, and economics such as, for example, the evaluation of the impact of public policies. The nature of a treatment or outcome is relatively unimportant in the estimation of the ATE—that is to say, calculation of the ATE requires that a treatment be applied to some units and not others, but the nature of that treatment (e.g., a pharmaceutical, an incentive payment, a political advertisement) is irrelevant to the definition and estimation of the ATE.

The expression "treatment effect" refers to the causal effect of a given treatment or intervention (for example, the administering of a drug) on an outcome variable of interest (for example, the health of the patient). In the Neyman-Rubin "potential outcomes framework" of causality a treatment effect is defined for each individual unit in terms of two "potential outcomes." Each unit has one outcome that would manifest if the unit were exposed to the treatment and another outcome that would manifest if the unit were exposed to the control. The "treatment effect" is the difference between these two potential outcomes. However, this individual-level treatment effect is unobservable because individual units can only receive the treatment or the control, but not both. Random assignment to treatment ensures that units assigned to the treatment and units assigned to the control are identical (over a large number of iterations of the experiment). Indeed, units in both groups have identical distributions of covariates and potential outcomes. Thus the average outcome among the treatment units serves as a counterfactual for the average outcome among the control units. The differences between these two averages is the ATE, which is an estimate of the central tendency of the distribution of unobservable individual-level treatment effects.^[2] If a sample is randomly constituted from a population, the sample ATE (abbreviated SATE) is also an estimate of the population ATE (abbreviated PATE).^[3]

While an experiment ensures, in expectation, that potential outcomes (and all covariates) are equivalently distributed in the treatment and control groups, this is not the case in an observational study. In an observational study, units are not assigned to treatment and control randomly, so their assignment to treatment may depend on unobserved or unobservable factors. Observed factors can be statistically controlled (e.g., through regression or matching), but any estimate of the ATE could be confounded by unobservable factors that influenced which units received the treatment versus the control.

Formal definition

In order to define formally the ATE, we define two potential outcomes :

y _{0} (i) is the value of the outcome variable for individual i if they are not treated, y _{1} (i) is the value of the outcome variable for individual i if they are treated. For example, y _{0} (i) is the health status of the individual if they are not administered the drug under study and y _{1} (i) is the health status if they are administered the drug.

The treatment effect for individual

i is given by y _{1} (i) - y _{0} (i) = \beta(i) . In the general case, there is no reason to expect this effect to be constant across individuals. The average treatment effect is given byATE = E [y _{1} - y _{0} ]

and can be estimated (if a law of large numbers holds)

\widehat{ATE} = (1)/(N) \sum _{i} (y _{1} (i) - y _{0} (i))

where the summation occurs over all

N individuals in the population.

If we could observe, for each individual,

y _{1} (i) and y _{0} (i) among a large representative sample of the population, we could estimate the ATE simply by taking the average value of y _{1} (i) - y _{0} (i) across the sample. However, we can not observe both y _{1} (i) and y _{0} (i) for each individual since an individual cannot be both treated and not treated. For example, in the drug example, we can only observe y _{1} (i) for individuals who have received the drug and y _{0} (i) for those who did not receive it. This is the main problem faced by scientists in the evaluation of treatment effects and has triggered a large body of estimation techniques.

Estimation

Depending on the data and its underlying circumstances, many methods can be used to estimate the ATE. The most common ones are:

• Natural experiments
• Difference in differences
• Regression discontinuity designs
• Propensity score matching
• Instrumental variables estimation

An example

Consider an example where all units are unemployed individuals, and some experience a policy intervention (the treatment group), while others do not (the control group). The causal effect of interest is the impact a job search monitoring policy (the treatment) has on the length of an unemployment spell: On average, how much shorter would one's unemployment be if they experienced the intervention? The ATE, in this case, is the difference in expected values (means) of the treatment and control groups' length of unemployment.

A positive ATE, in this example, would suggest that the job policy increased the length of unemployment. A negative ATE would suggest that the job policy decreased the length of unemployment. An ATE estimate equal to zero would suggest that there was no advantage or disadvantage to providing the treatment in terms of the length of unemployment. Determining whether an ATE estimate is distinguishable from zero (either positively or negatively) requires statistical inference.

Because the ATE is an estimate of the average effect of the treatment, a positive or negative ATE does not indicate that any particular individual would benefit or be harmed by the treatment. Thus the average treatment effect neglects the distribution of the treatment effect. Some parts of the population might be worse off with the treatment even if the mean effect is positive.

Heterogenous treatment effects

Some researchers call a treatment effect "heterogenous" if it affects different individuals differently (heterogeneously). For example, perhaps the above treatment of a job search monitoring policy affected men and women differently, or people who live in different states differently. ATE requires a strong assumption known as the stable unit treatment value assumption (SUTVA) which requires the value of the potential outcome

y(i) be unaffected by the mechanism used to assign the treatment and the treatment exposure of all other individuals. Let d be the treatment, the treatment effect for individual i is given by y _{1} (i,d) - y _{0} (i,d) . The SUTVA assumption allows us to declare y _{1} (i,d) = y _{1} (i), y _{0} (i,d) = y _{0} (i) .

One way to look for heterogeneous treatment effects is to divide the study data into subgroups (e.g., men and women, or by state), and see if the average treatment effects are different by subgroup. If the average treatment effects are different, SUTVA is violated. A per-subgroup ATE is called a "conditional average treatment effect" (CATE), i.e. the ATE conditioned on membership in the subgroup. CATE can be used as an estimate if SUTVA does not hold.

A challenge with this approach is that each subgroup may have substantially less data than the study as a whole, so if the study has been powered to detect the main effects without subgroup analysis, there may not be enough data to properly judge the effects on subgroups.

There is some work on detecting heterogeneous treatment effects using random forests^{[4] [5]} as well as detecting heterogeneous subpopulations using cluster analysis.^{[6] [7]} Recently, metalearning approaches have been developed that use arbitrary regression frameworks as base learners to infer the CATE.^{[8] [9]} Representation learning can be used to further improve the performance of these methods.^{[10] [11]}

Further reading

• Book: Wooldridge, Jeffrey M. . Policy Analysis with Pooled Cross Sections . 438–443 . Introductory Econometrics: A Modern Approach . 2013 . Thomson South-Western . Mason, OH . 978-1-111-53104-1 .

Notes and References

1. Zhao, Q., & Hastie, T. (2019). Causal Interpretations of Black-Box Models. Journal of Business & Economic Statistics, 39(1), 272–281. https://doi.org/10.1080/07350015.2019.1624293 online ncbi/
2. Holland . Paul W. . 1986 . Statistics and Causal Inference . . 81 . 396 . 945–960 . 2289064 . 10.1080/01621459.1986.10478354.
3. Imai . Kosuke . Gary . King . Elizabeth A. . Stuart . 2008 . Misunderstandings Between Experimentalists and Observationalists About Causal Inference . . 171 . 2 . 481–502 . 10.1111/j.1467-985X.2007.00527.x . 17852724 .
4. 1510.04342. Wager. Stefan. Athey. Susan. Estimation and Inference of Heterogeneous Treatment Effects using Random Forests. 2015. stat.ME.
5. Web site: Explicitly Optimizing on Causal Effects via the Causal Random Forest: A Practical Introduction and Tutorial. 14 October 2018 .
6. Markham . Alex . Das . Richeek . Grosse-Wentrup . Moritz . A Distance Covariance-based Kernel for Nonlinear Causal Clustering in Heterogeneous Populations . Proc. CLeaR . 2022 . PMLR 177 . 542–558 .
7. Huang . Biwei . Zhang . Kun . Xie . Pengtao . Gong . Mingming . Xing . Eric P. . Glymour . Clark . Specific and Shared Causal Relation Modeling and Mechanism-Based Clustering . Advances in Neural Information Processing Systems (NeurIPS) . 2019 . 32 .
8. Nie . Xinkun . Wager . Stefan . Quasi-oracle estimation of heterogeneous treatment effects . Biometrika . 2021 . 108 . 2 . 299–319 . 10.1093/biomet/asaa076. 1712.04912 .
9. Künzel . Sören . Sekhon . Jasjeet . Bickel . Peter . Yu . Bin . Metalearners for estimating heterogeneous treatment effects using machine learning . Proceedings of the National Academy of Sciences . 2019 . 116 . 10 . 4156–4165 . 10.1073/pnas.1804597116. 30770453 . 6410831 . free .
10. Johansson . Fredrik . Shalit . Uri . Sontag . David . Learning Representations for Counterfactual Inference . Proc. ICML . 2016 . PMLR 48 . 3020–3029 .
11. Book: Burkhart . Michael C. . Ruiz . Gabriel . Computational Science – ICCS 2022 . Neuroevolutionary Feature Representations for Causal Inference . Lecture Notes in Computer Science . 2022 . 13351 . 3–10 . 10.1007/978-3-031-08754-7_1 . 2205.10541. 978-3-031-08753-0 . 248987304 .