Reporting bias: why researchers don’t publish when innovative teaching approaches don’t work

studies-coverThere’s a lot of research about ‘what works’ in education. But what about ‘what doesn’t work’? In a recently published paper in Studies in Higher Education, we investigate a phenomenon called reporting bias, which the Cochrane Collaboration’s handbook defines as:

Reporting biases arise when the dissemination of research findings is influenced by the nature and direction of results. Statistically significant, ‘positive’ results that indicate that an intervention works are more likely to be published, more likely to be published rapidly, more likely to be published in English, more likely to be published more than once, more likely to be published in high impact journals and, related to the last point, more likely to be cited by others.

We analyse the research literature through a number of approaches and find strong evidence for reporting bias in learning and teaching. In our field, researchers usually don’t publish when a new teaching approach doesn’t give the results we expect… we chalk it up to experience or discuss with close colleagues instead. In our paper, we discuss the problems this causes, and some potential solutions:

Sharing successes and hiding failures: ‘reporting bias’ in learning and teaching research. Studies in Higher Education, 2016. Dawson, P., & Dawson, S.

When researchers selectively report significant positive results, and omit non-significant or negative results, the published literature skews in a particular direction. This is called ‘reporting bias’, and it can cause both casual readers and meta-analysts to develop an inaccurate understanding of the efficacy of an intervention. This paper identifies potential reporting bias in a recent high-profile higher education meta-analysis. It then examines a range of potential factors that may make higher education learning and teaching research particularly susceptible to reporting bias. These include the fuzzy boundaries between learning and teaching research, scholarship and teaching; the positive agendas of ‘learning and teaching’ funding bodies; methodological issues; and para-academic researchers in roles without tenure or academic freedom. Recommendations are provided for how researchers, journals, funders, ethics committees and universities can reduce reporting bias.

We hope researchers find this a useful paper to cite when publishing negative or non-significant results in learning and teaching. Feel free to email to discuss, or to request a copy of the paper.

Leave a Reply