Randomized Experiments

Amanda Coston, Mateo Dulce Rubio, Edward H. Kennedy

AI for Social Impact Book. Edited by Milind Tambe, Fei Fang & Bryan Wilder. https://ai4sibook.org/

Abstract

Once deployed in the real world, even the most carefully designed artificial intelligence (AI) systems may fail to achieve their intended goals or may have adverse unintended consequences. How should researchers assess whether the AI actually improved outcomes? Randomized experiments are the gold standard for evaluation. They enable one to isolate the effect of the AI from other potentially confounding factors. Examples of AI systems for social impact that have been deployed and evaluated in the real world abound: Wang et al. [2019] studies the effect on the adenoma detection rate of using real-time AI-assisted colonoscopies, Mohler et al. [2015] evaluates the effect of using a predictive policing model on crime rates in Los Angeles, Mate et al. [2021] assesses the effect on dropout of using a model to prioritize the follow-up of participants in a maternal and child care information program, etc. In this chapter we give an introduction and review of randomized experiments. More details and exposition can be found in Hernán and Robins [2010], Imbens and Rubin [2015], Rosenbaum [2002], Tsiatis [2006], van der Laan and Robins [2003], for example, among many others.

Reference

Randomized Experiments
Amanda Coston, Mateo Dulce Rubio, Edward H. Kennedy
AI for Social Impact Book (AI4SI Book). 2022.
PDF