An updste from the RCT-course

Then we arrived in Stavanger, the oil capital of Norway, on a day with a big jump in oil prices. However, no one in the RCT course was talking about oil. Everyone’s focus was on the research.

Nearly 20 researchers from different fields participated in the course, mostly PhD students but also others. Even an emeritus was among the participants. (I hope I will be as curious and eager to learn as him when I am his age.) However, it is not an everyday occurrence to have a visitor who is ranked among the world’s leading education economists.

It was a real pleasure to participate in Eric Bettinger’s lecture today. Different aspects were touched upon. They related to the components of an RCT (units, treatments, and outcome), mechanisms and counterfactuals (e.g. different outcomes from different dosages), selection bias (the one fulfilling the intervention differed from the one not fulfilling the intervention), the balance between the treatment and the control group for different non-manipulated variables (such as age, sex, etc.), the level of detail in the research question/generalization from the results (for a certain group of children at a certain age versus for all children) and the importance of theoretical justifications for a treatment for the advancement of the science.

Bettinger showed a great deal of enthusiasm through his pleasure, interest and engagement. He knows what he is talking about and gives a lot of examples from a variety of fields. Interaction was encouraged, and the lecture shifted between traditional teaching with well-designed PowerPoint slides (including nice references), training and discussions, which ensured variation and gave a feeling that the lecturing did not lasted too long, even if it was seven hours. Questions and comments were encouraged, and we were also given the opportunity to ask questions during the (approximately) 10-minute break. The language was clear, and the important points were stressed from different points of view. So Bettinger is not only a great economist but also a really great teacher and should we believe some of the existing research this may not be a too frequent combination.  A RCT-study from Netherland by Palali, van Elk, Bolhaar and Rud (2017) looked at the relationship between research quality and teaching quality and investigated if good researchers also are good teachers. They found that master students thought by teachers with high quality publications got higher grades. However, the student’s evaluation of the teaches did not fully reflected this since they were not ranked higher.

After today’s lectures, we took the bus back to Stavanger and went for a short walk to find our Airbnb house. After Indian takeaway food and some shopping (it is really cold here in Stavanger so I  had to buy a new jumper), we started our second task for today, namely to revise our questionnaire to the speech and language therapists, preschool teachers and health nurses. Finally, before we went to bed we did our homework.

I am looking forward to another course day tomorrow!

--------

Reference:

Palali, A., van Elk, R., Bolhaar, J., & Rud, I. (2017). Are good researchers also good teachers? The relationship between research quality and teaching quality. CPB Discussion Paper 347. Maastricht: CPB Netherlands Bureau for Economic Policy Analysis.

The detailed design process of a high-quality randomized controlled stuttering trial has started

Hilde, Linn, Åse and I are about to arrive in Stavanger to participate in a one-week course about randomized controlled trials (RCT) in social science at the University of Stavanger. The course is led by Eric Perry Bettinger from Standford School of Education, and we will actually be students for five days from 9:00 a.m. to 4:00 p.m. daily and have a reading list of seven pages full of references.

I am still a bit old fashioned when it comes to reading; I like paper versions of articles best so that I can make notes (e.g., inferences, comments, associations, questions) with a pen. I am aware of the latest technology, such as the Apple pencil that can be used to make comments by hand on the iPad, but I still prefer jotting down notes the old way. I do not know if it is because of my bad eyesight or the fact that it gives me a better overview of the text’s spatial landmarks and thereby reduces the cognitive load required to orient myself to the text. Or maybe it’s the depth of reading that is possible (e.g. effectively revisiting parts). It may simply be due to habit (reading rigidity). Luckily, the three others in the EST-team also prefer paper, so I am comforted by the feeling that I am not so outdated (read ‘old’) and it make me optimistic about my potential for learning the course materials. A systematic review of the research on the effects of reading media on reading comprehension by Delgadoa, Vargasb, Ackermanc and Salmeróna (2018) actually concluded that paper-based reading yields better comprehension outcomes than digital-based reading. So at least I could say that my preferred strategy is based on results from international research.

My thanks go to Linn and Åse for printing all the articles on the reading list and sending them to the Repro Central at the University for binding as books. They became no less than five “large books”, so then I knew what to do in the weekend. On Saturday, I opened the first book of articles and started to read enthusiastically Understanding and Misunderstanding Randomized Controlled Trials, by Deaton and Cartwright (2017). The content reminded me very much of the views on RCTs of a colleague at the University of Oslo, Tone Kvernbekk. She underlined RCTs pitfalls and the need for a combination of methods to determine what works and why.

On the one hand, the Deaton and Cartwright (2017) article critically highlights the pitfalls of the design of RCTs, such as potential confounders; on the other hand, it stresses the fact that other designs also have the same challenges in addition to their own. Therefore, RCT is still broadly considered to be the gold standard of designs. Neveertheless, the authors highlight the importance of having a realistic understanding of the strengths of RCT design and of its limitations and pointed to three common misunderstandings of it’s quality:

1)    Randomization ensures that the treatment and control groups differ only with regards to the treatment variable

2)    Randomization provides precise and unbiased estimates of average treatment effects

3)    Standard significant tests of RCTs are reliable

In addition, the athors underline the need for replication studies and complementary study designs to gain insights into the results of different samples and contexts and why the target intervention is effective and for whom (individual variations).

An update of our course experiences will follow shortly.

All the best,

Kari-Anne

-----
References:

Delgadoa, P., Vargasb, C., Ackermanc, R., & Salmeróna, L. (2018). Don't throw away your printed books: A meta-analysis on the effects of reading media on reading comprehension. Educational Research Review, 25, 23-38.

Deaton, A. & Cartwright, N. (2017). Understanding and misunderstanding randomized controlled trials. NBER working paper 22595. Cambridge: National bureau of economic research.