Firstly, thanks for all those who took part in #sciteachjc last Tuesday. It was a great start to the year and got quite a bit of attention building up to the meeting not least from one of the paper’s authors:
Secondly, Alom doesn’t think people read to the bottom of blog posts ;), so I’ll post up the date of the next meeting at the top. The next meeting will be held 12th February and @teachingofsci will be your host. The topic is yet to be confirmed, however we’re aiming to do something that is able to have an immediate impact on your teaching the next day in the classroom.
A summary of the discussion on RCTs: (we had 222 tweets during the hour we were discussing)
@DrRacheal opened with the elephant in the room:
Well, I’ve got to say, as an “ex” scientist, it beggars belief why this system isn’t used already. #sciteachjc
In reality, RCTs are used in educational research. However it was interesting to consider whether teachers could run them within school.
The responses to the first question: Would you consider running or taking part in a randomised control trial in your school?
@alomshaha raised the concerns of a few people involved in the discussion:
@ Some people think there might be ethical issues and also practical issues of *actually* carrying out RCTs? #SciTeachJC
@ryansecondarysc responded with a similar point to that made in the Radio 4 documentary:
@ @ #sciteachjc ethical issues are overblown though surely. Less ethical to subject all students to a failing method?
This was also backed up by @teachingofsci
#SciTeachJC I think idea in paper, that we do all these different interventions *anyway*, mean it’s *more* ethical to find out which work
@ @ @ @ it’s the teaching practice that’s being experimented on not the students #sciteachjc
Both @teachingofsci and @miss_mcinerney suggested that RCTs might not be appropriate to run within school due to practicalities.
#SciTeachJC one of my thoughts was that it would work best between schools, rather than between classes.
@ @ @ #sciteachjc I agree that ethical issues are overblown; practical issues are more problematic.
This was concurred by others in the discussion. The general feeling so far is that implementing an RCT within a school is problematic.
The idea raised by @bio_joe
Students are given different teachers based on the click of a timetable programme. within an institution can you see who’s good? #sciteachjc
suggested that the different classes assigned to teachers might be suitable as samples, but in general it was felt the sample size would be too small, and that the samples wouldn’t be randomly picked. Indeed @miss_mcinerney made this point:
@ @ @ #SciTeachJC Exactly. Most people place kids in classes carefully based on behaviour dynamics.
and DrRacheal this:
@ @ @ And of course classes are set based on pupil performance, too. #sciteachjc
What about between schools?
@ #SciTeachJC But different schools often wildly different on many variables. This is more of a quasi-experiment than an RCT.
There were some great ideas about the design of RCTs, mainly between @miss_mcinerney and @teachingofsci e.g.
@ so recruit, pair schools that are as similar as possible, then randomize which receives A/B intervention? #SciTeachJC
@ Why parallel? Why not ‘before’ and ‘after’? E.g. do 1 term w/out then 1 term with? Or compare to previous year? #sciteachjc
Could RCTs be done retrospectively? i.e. find settings that have implemented competing methods and study random students? #sciteachjc
but @ryansecondarysc and @miss_mcinerney pointed out that this would no longer be an RCT
@ #sciteachjc that would be more of a case study than an RCT, as without pre-agreeing measurable outcomes, open to bias etc. @ #sciteachjc That’s quasi-experiment. Can be done, and very usefully, you just need to do stats a little differently to account
and @tom_hartley added
@ + with big data from many schools: hierarchical stats to model effects of school/teacher separately, I think #SciTeachJC
We focused again on the ethics with question two What concerns do you have with the ethical considerations of using RCTs in educational research?
@oldandrewuk was concerned about whether teachers would be happy to take part in an RCT:
@ @ @ #sciteachjc There’s a difference between teachers trying what they think might work & a formal…
@ @ @ …experiment organised by outsiders over-riding teacher’s professional judgement. #sciteachjc @ #sciteachjc My concern is that teacher’s would be made to persist with methods they see failing, in order to test them.
#sciteachjc a teacher s professional judgement may be compromised and have a negative impact on student learning
@Bio_joe pointed out that :
@ @ in medical trials if the drug is doing really badly then they will stop the study, evalutaion is key #sciteachjc
and I added:
RCTs are usually done in medical trials after they’ve been shown not to have a negative effect. Find out which is best #sciteachjc
@miss_mcinerney made the point that
@ #sciteachjc There’s also evidence to suggest that if teachers don’t agree/believe in the intervention it’s less likely to work
to which I coined a new phrase (we think)
@ Are there any examples of good RCTs which have been done in education? @ #SciTeachJC
I highlighted some that I’d seen, but hadn’t had a chance to read fully:
@ @ @ Examples are work by Carole Torgerson, @ etc #sciteachjc
and @teachingofsci said:
I would like to hear from a teacher that has been involved in an education RCT. What actually happened. #sciteachjc
Final question: Which educational intervention would you like to see more evidence for, in the form of RCTs (or other methodology)?
@alomshaha suggested his favourite topic:
but we descended (no thanks to me) into a discussion on ensuring that teachers were aware of any research that had been done and the lack of open availability to research.
#sciteachjc Research has to be made easily available to teachers and policy makers (SLT or gov). We must know why?
#SciTeachJC *yes* — clearing house of ed research summaries, classroom ready, funded centrally for us all.
Here is a selection of the links posted by @teachingofsci during the discussion
There are loads of other valid points that were made, and @bio_joe, @oldandrewuk and @ryansecondarysc carried on the discussion afterwards. In all a good start to the year, and remember the next meeting on the 12th February. (Did you get to the bottom of the post?)
Here is a full Storify archive compiled by @eyebeams (thanks):