What is p-hacking? (3 hours)

↵ Back to module homepage

A common issue in data analysis is the use of questionable research practices (also referred to as "p-hacking", "researcher degrees of freedom", or "the garden of forking paths") to twist one's own data to support a prediction the researcher already had in mind. In theory we should let the results of our research tell us what to believe; in practice, however, it is tempting to let our beliefs tell us what the results should be, and to manipulate the results until they match our beliefs.

Rather than listening to me explain how p-hacking works, you can understand it better if you see a demonstration of p-hacking in action. Each of the papers, videos, or blog posts below demonstrates a situation in which a researcher gathered some data and then used problematic analysis techniques to make the data appear to prove something ridiculous or even impossible. Read (or watch) any one of these (you can choose whichever you think looks most interesting, most fun, most relevant to your research, or whatever) and figure out (a) what was the ridiculous conclusion they demonstrated with their data analysis; and (b) what problematic analysis technique (or techniques) did they use to try to prove the ridiculous thing.

• Video demo of p-hacking with country GDP data by Neuroskeptic (example begins around 14:48)
  • You can try doing these analyses yourself by using the sample Excel data file here
• Simmons, J., Nelson, L., & Simonsohn, U. (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366.
• Blog post by Simonsohn about horoscopes and odd numbers
• Roberts, S., & Winters, J. (2013). Linguistic diversity and traffic accidents: lessons from statistical studies of cultural traits. PLoS ONE, 8, e70902.
• Baker, C., Hutchison, T., & Kanwisher, N. (2007). Does the fusiform face area contain subregions highly selective for nonfaces? Nature Neuroscience, 10, 3-4.
• Bennett, C., Baird, A., Miller, M., & Wolford, G. (2009). Neural correlates of interspecies perspective taking in the post-mortem atlantic salmon: an argument for proper multiple comparisons correction. 15th Annual Meeting of the Organization for Human Brain Mapping, San Francisco. (A poster version is available here.)

When you have answered questions (a) and (b) above, continue to the next section of the module: "Reflection on p-hacking".

by Stephen Politzer-Ahles. Last modified on 2021-05-17. CC-BY-4.0.