Reflection on p-hacking

Reflection on p-hacking (1 hour)

Here is a list of the papers from the previous task, along with brief descriptions of what ridiculous things they demonstrated:

Music about old age makes you younger: Simmons, J., Nelson, L., & Simonsohn, U. (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366.
Countries with shorter names have higher GDPs: Video demo by Neuroskeptic (example begins around 14:48)
- Data (xlsx)
People whose subject ID is odd believe odd things: Blog post by Simonsohn
Countries with more languages have more traffic accidents: Roberts, S., & Winters, J. (2013). Linguistic diversity and traffic accidents: lessons from statistical studies of cultural traits. PLoS ONE, 8, e70902.
Brain activation in the nose: Baker, C., Hutchison, T., & Kanwisher, N. (2007). Does the fusiform face area contain subregions highly selective for nonfaces? Nature Neuroscience, 10, 3-4.
Brain activation in a dead fish: Bennett, C., Baird, A., Miller, M., & Wolford, G. (2009). Neural correlates of interspecies perspective taking in the post-mortem atlantic salmon: an argument for proper multiple comparisons correction. 15th Annual Meeting of the Organization for Human Brain Mapping, San Francisco.
General review about p-hacking: Gelman, A., & Loken, E. (2013 manuscript). The garden of forking paths: why multiple comparisons can be a problem, even when there is no "fishing expedition" or "p-hacking" and the research hypothesis was posited ahead of time.
The 9 Circles of Scientific Hell by Neuroskeptic

What all those papers have in common is that they demonstrate that a certain way of data analysis is bad because it might lead you to making crazy conclusions. For example, what the Bennett paper shows is: if you don't do proper correction for multiple comparisons when analyzing fMRI data, you might wrongly conclude that the brain of a dead fish shows different brain responses when it is shown happy faces vs. angry faces. (This is obviously insane.) What the Simmons paper shows is: if you perform lots of different kinds of data analyses without any analysis plan, you might wrongly conclude that participants who listen to a Beatles song literally become younger. etc. The Simmons paper and the Gelman paper included at the above website both give detailed and comprehensive explanations of what p-hacking is and how it can cause you to get junk results which lead to junk conclusions; if you don't feel like you understand the concept of p-hacking, I recommend reading first the Simmons paper and then the Gelman paper. Or, for a funny introduction, see the video below (this touches on a lot of things other than just p-hacking, but p-hacking is briefly mentioned).

Continue to the questions below to reflect on your understanding of p-hacking.

Are there any common research practices (including research practices that you often see in papers in your field, or research practices you've done yourself on previous research projects you did) that you thought were ok but that you now learned are a form of p-hacking? If so, what are they?

Think about your own research (including studies you have done before and/or studies you plan to carry out in the future). What is one way that you might accidentally p-hack—in other words, what is something you might be tempted to do, that would be p-hacking? (The reason I'm asking this is of course not to encourage you to p-hack, but to encourage you to identify potential research practices that you need to be wary of and need to be careful to avoid.)

When you have finished these activities, continue to the next section of the module: "Pre-registration".

by Stephen Politzer-Ahles. Last modified on 2021-05-17. CC-BY-4.0.