Researcher degrees of freedom explained

Researcher degrees of freedom is a concept referring to the inherent flexibility involved in the process of designing and conducting a scientific experiment, and in analyzing its results. The term reflects the fact that researchers can choose between multiple ways of collecting and analyzing data, and these decisions can be made either arbitrarily or because they, unlike other possible choices, produce a positive and statistically significant result. The researcher degrees of freedom has positives such as affording the ability to look at nature from different angles, allowing new discoveries and hypotheses to be generated.[1] [2] [3] However, researcher degrees of freedom can lead to data dredging and other questionable research practices where the different interpretations and analyses are taken for granted [4] [5] Their widespread use represents an inherent methodological limitation in scientific research, and contributes to an inflated rate of false-positive findings.[6] They can also lead to overestimated effect sizes.[7]

Though the concept of researcher degrees of freedom has mainly been discussed in the context of psychology, it can affect any scientific discipline.[6] [8] Like publication bias, the existence of researcher degrees of freedom has the potential to lead to an inflated degree of funnel plot asymmetry.[9] It is also a potential explanation for p-hacking, as researchers have so many degrees of freedom to draw on, especially in the social and behavioral sciences. Multiverse analysis is a method that helps bring these degrees of freedom to light. Studies with smaller sample sizes are more susceptible to the biasing influence of researcher degrees of freedom.[10]

Examples

Steegen et al. (2016) showed how, starting from a single raw data set, applying different reasonable data processing decisions can give rise to a multitude of processed data sets (called the data multiverse), often leading to different statistical results.[11] Wicherts et al. (2016) provided a list of 34 degrees of freedom (DFs) researchers have when conducting psychological research. The DFs listed span every stage of the research process, from formulating a hypothesis to the reporting of results. They include conducting exploratory, hypothesis-free research, which the authors note "...pervades many of the researcher DFs that we describe below in the later phases of the study." Other DFs listed in this paper include the creation of multiple manipulated independent variables and the measurement of additional variables that may be selected for analysis later on.[7]

See also

Notes and References

  1. Book: Tukey, J.W. . Exploratory data analysis Volume 2 . Pearson . 1977.
  2. Box . George E. P. . December 1976 . Science and Statistics . Journal of the American Statistical Association . 71 . 356 . 791–799 . 10.2307/2286841. 2286841 .
  3. Book: Groot . Adriaan D. de . Thought and choice in chess . Groot . Adrianus Dingeman de . 1978 . Mouton . 978-90-279-7914-8 . 2 . Psychological studies . The Hague.
  4. Wicherts . Jelte M. . Veldkamp . Coosje L. S. . Augusteijn . Hilde E. M. . Bakker . Marjan . van Aert . Robbie C. M. . van Assen . Marcel A. L. M. . 2016 . Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking . Frontiers in Psychology . 7 . 1832 . 10.3389/fpsyg.2016.01832 . 1664-1078 . 5122713 . 27933012 . free.
  5. Simmons . Joseph P. . Nelson . Leif D. . Simonsohn . Uri . 2011-10-17 . False-Positive Psychology . Psychological Science . 22 . 11 . 1359–1366 . 10.1177/0956797611417632 . 22006061 . 13802986 . 0956-7976.
  6. Simmons . Joseph P. . Nelson . Leif D. . Simonsohn . Uri . November 2011 . False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant . Psychological Science . en . 22 . 11 . 1359–1366 . 10.1177/0956797611417632 . 22006061 . 13802986 . 0956-7976.
  7. Wicherts . Jelte M. . Veldkamp . Coosje L. S. . Augusteijn . Hilde E. M. . Bakker . Marjan . van Aert . Robbie C. M. . van Assen . Marcel A. L. M. . 2016-11-25 . Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking . Frontiers in Psychology . 7 . 1832 . 10.3389/fpsyg.2016.01832 . 1664-1078 . 5122713 . 27933012. free .
  8. Coretta . Stefano . Casillas . Joseph V. . Roessig . Simon . Franke . Michael . Ahn . Byron . Al-Hoorie . Ali H. . Al-Tamimi . Jalal . Alotaibi . Najd E. . AlShakhori . Mohammed K. . Altmiller . Ruth M. . Arantes . Pablo . Athanasopoulou . Angeliki . Baese-Berk . Melissa M. . Bailey . George . Sangma . Cheman Baira A . July 2023 . Multidimensional Signals and Analytic Flexibility: Estimating Degrees of Freedom in Human-Speech Analyses . Advances in Methods and Practices in Psychological Science . en . 6 . 3 . 10.1177/25152459231162567 . 2515-2459. free . 2346/95600 . free .
  9. Carter . Evan C. . McCullough . Michael E. . 2014-07-30 . Publication bias and the limited strength model of self-control: has the evidence for ego depletion been overestimated? . Frontiers in Psychology . 5 . 823 . 10.3389/fpsyg.2014.00823 . 1664-1078 . 4115664 . 25126083. free .
  10. Schweizer . Geoffrey . Furley . Philip . March 2016 . Reproducible research in sport and exercise psychology: The role of sample sizes . Psychology of Sport and Exercise . en . 23 . 114–122 . 10.1016/j.psychsport.2015.11.005.
  11. Steegen. Sara. Tuerlinckx. Francis. Gelman. Andrew. Vanpaemel. Wolf. September 2016. Increasing Transparency Through a Multiverse Analysis. Perspectives on Psychological Science. en-US. 11. 5. 702–712. 10.1177/1745691616658637. 27694465. 1745-6916. free.

This article is licensed under the GNU Free Documentation License. It uses material from the Wikipedia article "Researcher degrees of freedom".

Except where otherwise indicated, Everything.Explained.Today is © Copyright 2009-2024, A B Cryer, All Rights Reserved. Cookie policy.