Practical advice on school-led research – in the second blog in a series on research in schools, Robin Hall looks at how to set up randomised controlled trials.
Practical guides to school-led R&D
2. Randomised controlled trials
Randomised controlled trials are research tools that can tell us whether one approach works better than another. They’re called randomised controlled trials because they involve randomly allocating participants to two or more groups. One group carries on doing what they would usually do – this is known as the control group, while the other group tries something new. All the participants are then tested to find out which group has improved the most.
Diagram 1 shows an example of a ‘between subject’ trial design where you’re finding out the difference between two groups of participants. There are several ways to organise how this happens during your trial.
If the effect of your intervention can ‘wear off’ then you can use a ‘within subject’ design (see diagram 2 below). In this case, you are finding out the change within each participant. Everyone is tested after doing what they would usually do and after trying the new intervention. This has the advantage that all participants try out the new intervention and so there is no variation between the intervention and control groups.
Pre-testing
Some interventions do not wear off and will affect any subsequent testing. In this case, it is common practice to carry out a pre-test. This is when all the participants are tested before they take part in the trial as well as after (see diagram 3 below). You can use statistics to take account of any differences between the two groups by taking account of the pre-test scores when you analyse the post-test scores.
The benefits of randomisation
The randomisation process helps to avoid introducing any accidental bias when choosing who is in the intervention group and who is in the control group. The control group then gives you something to compare against. Almost any teaching intervention will help pupils to improve – what is important is to find out whether any new intervention helps them improve more than what you already do.
Significance testing
Whenever you test groups of pupils, the results will vary. Significance testing can help to understand whether any improvement in pupils’ test results is down to the new intervention or just good luck. If the post-test results for your intervention group are significantly better than the control group, you can be fairly certain that the intervention works. If the difference is not statistically significant, that means that the intervention is probably about the same as your current practice overall. It is still fine to use the intervention for pupils where other approaches don’t seem to be helping as you won’t be doing them any harm and it might work for them.
Diagram 4 (above) shows a trial where the attainment of the intervention group was greater than the control group but not by a statistically significant amount (a difference is usually considered statistically significant if the probability of it happening by chance, p, is less than 0.05.)
Data analysis
It is important to use the correct statistical test for your data. This will depend on the type of trial you ran and the kind of data you collected. There are many online resources to help with data analysis, such as the Social Science Statistics website. You could also ask other researchers, such as educational psychologists or university researchers. Maths teachers or psychology teachers may be able to help too.
Further reading
In 2015 around 50 teachers took part in NCTL’s Closing the gap: test and learn event to share the results from their own research trialing new interventions.
The slides from the training event, along with the research template and research poster guidance, show how the teachers on the project designed their trials. You can see their results in the Closing the gap: test and learn case study reports.
You can also read Robin’s blog on the Closing the gap event.
If you have any questions or feedback, please comment below. To keep up to date with this blog, you can sign up for email updates or follow NCTL on Twitter.
For more information or opportunities to get involved with our work, visit our pages on GOV.UK.
Leave a comment