Peer Effects in Education in South Africa

A great paper presented at the NEUDC conference by compatriot Rob Garlick on peer effects in education.

Children’s success in school depends, partly, on the actions of their peers. An important policy question is therefore the assignment of children in schools. Should we, for example, track students by separating the “weak” and “strong” students into different classes? Garlick finds that, at least when it comes to dormitories at the University of Cape Town, tracking leads to a reduction in average grades.

Garlick stumbled into a natural experiment at his old University when residential admissions changed from tracking to random assignment. So, he could take the change in average marks in the dormitories, before and after the policy, and compare this to the whole university. He also looked at how particular students (based on their high school grades) are affected by the policy. Turns out that the weaker students benefit massively from being in a dormitory with stronger students; however, the stronger students don’t seem to be badly affected by the reverse. Combing weak and strong students in a dormitory therefore leads to improved grades.  

What explains this peer effect? It seems like it has more to do with the friends you make, rather than the people you work or live with. The peer effects don’t operate across races. Since South African students still tend to socialise across racial lines, this suggests it only helps to live in a dorm with a strong student that you actually socialise with. Another paper at the same conference on peer effects found similar evidence. In a leading business school in India, students are randomly allocated to a dorm and a classroom. The paper finds no peer effects in the classroom, but strong peer effects in the dormitory.

So, what conclusions can be reached from this paper? This is strong evidence that tracking within a school is seriously detrimental to the weaker students. The more interesting question in South Africa, however, is the distribution of education outcomes between schools. Certainly, township schools suffer when their smartest students get scholarships and leave for good schools. But what about the flip side: will students in under-performing schools benefit if they are placed in well-performing schools? This paper suggests that positive peer effects might not operate if students don’t socialise across racial groups. 

External Validity and False Positives in Randomised Control Trials

Two presentations at the latest NEUDC grabbed my attention in providing some necessary caveats on the conclusions we can reach from RCT's. 

First, a paper looked at the scalability of a proven intervention. Researchers intentionally select able NGO’s to implement the projects. (They often also put a lot of individual effort in to ensure quality implementation). However, policy conclusions often involve large-scale government roll out. Can a government repeat on scale the success of a highly motivated and able NGO?

This paper looked specifically at the use of contract teachers in education. (a great summary of this on the CSAE blog). Duflo Dupas and Kremer (2009) show that the use of contract teachers can significant increase educational outcomes in Kenya, partly because they face stronger incentives to teach well. However, turns out that implementation relied on a food NGO. When the NGO scaled up the project it worked, but when the government scaled up the project it didn’t.

This places some caveats on the policy conclusions we can reach from many RCT’s.

The second paper applies the standards required from RCT’s in medical trials to economic papers and finds us severely lacking. We have stolen the method of RCT’s from medicine, but we ignored what they have learnt about the shortcomings of RCT’s. This is a glaring gap and I can’t believe that this paper is the first to do it.

Randomisation solves endogeneity problems; however, biases emerge in the way that we conduct and report our studies, which could lead to false positives.

One big source of bias is lack of “blinding”. Participants respond or act differently because they know they are being treated, a kind of “Hawthorne effect”. This change in behaviour could have nothing to do with the actual treatment. In medical trials this is solved by giving the control group a placebo, but this is far more difficult in social projects. Furthermore, data collectors might ask questions differently in treatment units, because of the perceived pressure from the researcher to get a positive result.

The big problem, of course, is that researchers are biased. Aspirational graduate students (like myself!) invest years in a project and future job prospects often depend on finding a positive result. So, the more discretion is left to the researcher (in sample selection or reporting of results, for example), the higher the bias.

The authors propose introducing standards for conducting RCT’s and reporting results, similar to that in medical trials. The more we can tie the hands of the researchers, the less chance that /she can bias results. This would be a massive contribution to the field.