Are RCT’s all they are cracked up to be? Challenging the gold standard for causal evidence in outreach work. Experiences from an evaluation perspective

Evaluation guidelines of the UK educational outreach sector are often tied to a typology that differentiates and prioritises causal inferences. Causality in these instances is often thought to be the “gold standard” of research, one that values evaluation methods such as randomized control trials (RCTs), and quasi-experimental designs (QEDs). Our research seeks to challenge the so-called “gold standard” and its application to educational research.
We start by investigating the origins of RCTs in scientific and medical research and question its emergence in educational research. For example, we analyse how historically, RCTs—a method that is deeply rooted in medical research and psychology—was transplanted into educational research through educational psychology and social policy research. Given these historical considerations, we then, weigh the criticisms and values of using RCTs in educational research. Firstly, we explore existing literature to understand the greatest challenges and merits of using RCTs and QEDs in educational settings. The main merit we highlight is the potential—in ideal conditions—to indicate the causal (or at least highly correlated) effect of an intervention on individuals, without allocation bias. The downsides are the ability to control in educational settings (since you are unable to fully “blind” participants), the complex environments in educational settings, the expense of RCTs and QEDs, the generalisation and universalisation of findings, and the descriptive (and thus non-theoretical) basis of RCTs.
Secondly, we supplement this information with our own investigation, by interviewing and surveying practitioners in the field. Finally, we explain our positionality within this research by tying these findings with our own experiences as educational researchers and evaluators. Accordingly, our research will highlight the key challenges indicated and explore ways in which they can—if they can—be mitigated.
Finally, we suggest how the typology in educational outreach can be thought of, considering our findings. As an example, we dive into the way in which evaluation is distinguished as Type 1: Narrative, Type 2: Empirical Inquiry, and Type 3: Causality in the UK educational outreach sector. We then explore the values of using other types of data and methods that could potentially disrupt the way we think of Gold Standards in educational research and evaluation.