Embrace it when cherished interventions are shown not to work

We must update our beliefs – even if we are professionally invested in them

Earlier this month, the Education Endowment Foundation (EEF) published the results of a randomised controlled trial evaluating “Project SUCCESS” – a project in which students at FE colleges and their nominated study supporters were sent text messages to try and encourage positive discussions about their maths and English GCSEs, which the students were resitting. We, together with colleagues, designed and implemented the intervention both in this trial, and in two that were published previously.

Research on boosting and activating the social support networks of struggling students formed a large part of Bibi’s PhD, which Michael supervised, at UCL’s School of Public Policy. The first two trials we conducted were of a reasonable scale, and found positive impacts on exam pass-rates. We also heard from dozens of students, teachers, and supporters in qualitative research, who found the experience of participating in the programme positive, and who felt that it had contributed towards their positive outcomes. We both devoted a large amount of time, energy and emotion to these projects, and developed plans for the large scale adoption of the intervention. We believed it offered a low cost means to support young people’s learning – alongside the support and tuition they receive from their college.

The new trial, which is the largest to date and took place across more than 30 further education colleges in England, finds no evidence of a positive impact on participating students’ grades in maths or English, or of their attendance at class. Although the treatment group saw a slightly higher level of passes overall, this was both small and far from statistically significant. Feedback on the project collected through the implementation and process evaluation was positive, with students generally finding the messages useful and appropriate, and some identifying the messages as having played a role in their exam results.

We accept the findings, and thank both the Education Endowment Foundation, and the independent evaluators in NatCen, for their funding and the work that went into running the trial. Not everything went perfectly in the trial, but it never does. The outcome measures – a binary indication of pass and fail at GCSE – may not be ideal, but they are, ultimately the standard of success that is set, not just for our intervention, but for the educational journey of the students we want to help. Experience has taught us that if you wait for ideal circumstances to test an idea, you’ll be waiting a long time.

It is easy to talk about evidence based policy when you’re “winning” – when the interventions you espouse and believe in are being shown to work – but it is quite another when things go the other way, and an intervention that you have loved and raised from an idea to a full-scale trial is shown not to work. There is a sense of loss, and a palpable sadness, that emerges from those few little words: “We find no evidence…”.

Yet evidence-based policy and practice requires us not just to celebrate the victories but to embrace these null results as well. If resources are scarce, it is important that we continue to do things that work, and that we stop or at least rethink those that do not.

This is what we will do. We have updated our beliefs about what works and what doesn’t in this context. We will seek, together, to learn the lessons of what has not been successful, and we will move on. We learned more from this research than a binary “pass/fail” grading, and much of that will be useful to us or other researchers in future. In the meantime, we should not mourn the passing of a cherished intervention, but instead celebrate its life.

Dr Michael Sanders and Dr Bibi Groot

Dr Michael Sanders is Chief Executive of What Works for Children’s Social Care, and a Reader in Public Policy at the Policy Institute, King’s College London.

Dr Bibi Groot is Director & Head of Educational Policy at CLOO Behavioural Insights Unit.