Summary of Meeting Sixteen

Firstly, thanks for all those who took part in #sciteachjc last Tuesday. It was a great start to the year and got quite a bit of attention building up to the meeting not least from one of the paper’s authors:

#sciteachjc discussing RCTs in education as discussed in our Cabinet Office paper http://t.co/2mh508IR via @
@bengoldacre
ben goldacre

Secondly, Alom doesn’t think people read to the bottom of blog posts ;), so I’ll post up the date of the next meeting at the top. The next meeting will be held 12th February and @teachingofsci will be your host. The topic is yet to be confirmed, however we’re aiming to do something that is able to have an immediate impact on your teaching the next day in the classroom.

A summary of the discussion on RCTs: (we had 222 tweets during the hour we were discussing)

@DrRacheal opened with the elephant in the room:

Well, I've got to say, as an "ex" scientist, it beggars belief why this system isn't used already. #sciteachjc
@DrRacheal
Dr Racheal

In reality, RCTs are used in educational research. However it was interesting to consider whether teachers could run them within school.

The responses to the first question: Would you con­sider run­ning or tak­ing part in a ran­domised con­trol trial in your school?

@alomshaha raised the concerns of a few people involved in the discussion:

@ Some people think there might be ethical issues and also practical issues of *actually* carrying out RCTs? #SciTeachJC
@alomshaha
Alom Shaha

@ryansecondarysc responded with a similar point to that made in the Radio 4 documentary:

@ @ #sciteachjc ethical issues are overblown though surely. Less ethical to subject all students to a failing method?

This was also backed up by @teachingofsci
#SciTeachJC I think idea in paper, that we do all these different interventions *anyway*, mean it's *more* ethical to find out which work

and @bio_joe
@ @ @ @ it's the teaching practice that's being experimented on not the students #sciteachjc
@Bio_Joe
Joe Wright

Both @teachingofsci and @miss_mcinerney suggested that RCTs might not be appropriate to run within school due to practicalities.

#SciTeachJC one of my thoughts was that it would work best between schools, rather than between classes.

@ @ @ #sciteachjc I agree that ethical issues are overblown; practical issues are more problematic.
@miss_mcinerney
Laura McInerney

This was concurred by others in the discussion. The general feeling so far is that implementing an RCT within a school is problematic.
The idea raised by @bio_joe

Students are given different teachers based on the click of a timetable programme. within an institution can you see who's good? #sciteachjc
@Bio_Joe
Joe Wright

suggested that the different classes assigned to teachers might be suitable as samples, but in general it was felt the sample size would be too small, and that the samples wouldn’t be randomly picked. Indeed @miss_mcinerney made this point:
@ @ @ #SciTeachJC Exactly. Most people place kids in classes carefully based on behaviour dynamics.
@miss_mcinerney
Laura McInerney

and DrRacheal this:
@ @ @ And of course classes are set based on pupil performance, too. #sciteachjc
@DrRacheal
Dr Racheal

What about between schools?

@ #SciTeachJC But different schools often wildly different on many variables. This is more of a quasi-experiment than an RCT.
@miss_mcinerney
Laura McInerney

There were some great ideas about the design of RCTs, mainly between @miss_mcinerney and @teachingofsci e.g.

@ so recruit, pair schools that are as similar as possible, then randomize which receives A/B intervention? #SciTeachJC

@ Why parallel? Why not 'before' and 'after'? E.g. do 1 term w/out then 1 term with? Or compare to previous year? #sciteachjc
@miss_mcinerney
Laura McInerney

I asked:

Could RCTs be done retrospectively? i.e. find settings that have implemented competing methods and study random students? #sciteachjc
@A_Weatherall
Alex Weatherall

but @ryansecondarysc and @miss_mcinerney pointed out that this would no longer be an RCT.
@ #sciteachjc that would be more of a case study than an RCT, as without pre-agreeing measurable outcomes, open to bias etc.

@ #sciteachjc That's quasi-experiment. Can be done, and very usefully, you just need to do stats a little differently to account
@miss_mcinerney
Laura McInerney

and @tom_hartley added
@ + with big data from many schools: hierarchical stats to model effects of school/teacher separately, I think #SciTeachJC
@tom_hartley
Tom Hartley

We focused again on the ethics with question two What con­cerns do you have with the eth­i­cal con­sid­er­a­tions of using RCTs in edu­ca­tional research?

@oldandrewuk was concerned about whether teachers would be happy to take part in an RCT:

@ @ @ #sciteachjc There's a difference between teachers trying what they think might work & a formal...
@oldandrewuk
Andrew Old

@ @ @ ...experiment organised by outsiders over-riding teacher's professional judgement. #sciteachjc
@oldandrewuk
Andrew Old

@ #sciteachjc My concern is that teacher's would be made to persist with methods they see failing, in order to test them.
@oldandrewuk
Andrew Old

@dfalc agreed:
#sciteachjc a teacher s professional judgement may be compromised and have a negative impact on student learning
@DFalc
DFalconer

@Bio_joe pointed out that :
@ @ in medical trials if the drug is doing really badly then they will stop the study, evalutaion is key #sciteachjc
@Bio_Joe
Joe Wright

and I added:
RCTs are usually done in medical trials after they've been shown not to have a negative effect. Find out which is best #sciteachjc
@A_Weatherall
Alex Weatherall

@miss_mcinerney made the point that

@ #sciteachjc There's also evidence to suggest that if teachers don't agree/believe in the intervention it's less likely to work
@miss_mcinerney
Laura McInerney

to which I coined a new phrase (we think)

@alomshaha asked

@ Are there any examples of good RCTs which have been done in education? @ #SciTeachJC
@alomshaha
Alom Shaha

I highlighted some that I’d seen, but hadn’t had a chance to read fully:

and @teachingofsci said:
#SciTeachJC a lot more is happening in the US it seems - see http://t.co/axzVqrBc for some discussion.

@bio_joe asked
I would like to hear from a teacher that has been involved in an education RCT. What actually happened. #sciteachjc
@Bio_Joe
Joe Wright

Final question: Which educational intervention would you like to see more evidence for, in the form of RCTs (or other methodology)?

@alomshaha suggested his favourite topic:

@ Practical work in science 😉 #SciTeachJC
@alomshaha
Alom Shaha

and @Bio_Joe

but we descended (no thanks to me) into a discussion on ensuring that teachers were aware of any research that had been done and the lack of open availability to research.

#sciteachjc Research has to be made easily available to teachers and policy makers (SLT or gov). We must know why?
@A_Weatherall
Alex Weatherall

Otherwise we are just going through the motions of implementation. http://t.co/eVOHwP2R #sciteachjc
@A_Weatherall
Alex Weatherall

#SciTeachJC *yes* - clearing house of ed research summaries, classroom ready, funded centrally for us all.

@ #sciteachjc I agree. (In fact, I wrote something similar in a blog for @ http://t.co/hM1SMM94)
@miss_mcinerney
Laura McInerney

Here is a selection of the links posted by @teachingofsci during the discussion

There are loads of other valid points that were made, and @bio_joe, @oldandrewuk and @ryansecondarysc carried on the discussion afterwards. In all a good start to the year, and remember the next meeting on the 12th February. (Did you get to the bottom of the post?)

Here is a full Storify archive compiled by @eyebeams (thanks):
http://storify.com/eyebeams/rct-discussion-on-sciteachjc

This entry was posted in Uncategorized. Bookmark the permalink.

2 Responses to Summary of Meeting Sixteen

  1. Pingback: The Evidence-Based-Teaching Bandwagon | Teaching Science

  2. Pingback: The evidence based teaching bandwagon

Leave a Reply

Your email address will not be published. Required fields are marked *