Multiverse analysis explained

Multiverse analysis is a scientific method that specifies and then runs a set of plausible alternative models or statistical tests for a single hypothesis.[1] It is a method to address the issue that the "scientific process confronts researchers with a multiplicity of seemingly minor, yet nontrivial, decision points, each of which may introduce variability in research outcomes".[2] A problem also known as Researcher degrees of freedom[3] or as the garden of forking paths. It is a method arising in response to the credibility and replication crisis taking place in science, because it can diagnose the fragility or robustness of a study's findings. Multiverse analyses have been used in the fields of psychology[4] and neuroscience.[5] It is also a form of meta-analysis allowing researchers to provide evidence on how different model specifications impact results for the same hypothesis, and thus can point scientists toward where they might need better theory or causal models.

Details

Multiverse analysis most often produces a large number of results that tend to go in all directions. This means that most studies do not offer consensus or specific rejection of an hypothesis. Its strongest utilities thus far are instead to provide evidence against conclusions based on findings from single studies or to provide evidence about which model specifications are more or less likely to cause larger or more robust effect sizes (or not).

Evidence against single studies or statistical models, is useful in identifying potential false positive results. For example, a now infamous study concluded that female gender named hurricanes are more deadly than male gender named hurricanes.[6] In a follow up study,[7] researchers ran thousands of models using the same hurricane data, but making various plausible adjustments to the regression model. By plotting a density curve of all regression coefficients, they showed that the coefficient of the original study was an extreme outlier.

In a study of birth order effects,[8] researchers visualized a multiverse of plausible models using a specification curve which allows researchers to visually inspect a plot of all model outcomes against various model specifications. They could show that their findings supported previous research of birth order on intellect, but provided evidence against an effect on life satisfaction and various personality traits.

Notes and References

  1. Steegen . Sara . Tuerlinckx . Francis . Gelman . Andrew . Vanpaemel . Wolf . Increasing Transparency Through a Multiverse Analysis . Perspectives on Psychological Science . September 2016 . 11 . 5 . 702–712 . 10.1177/1745691616658637. free .
  2. Breznau . Nate . Rinke . Eike Mark . Wuttke . Alexander . Nguyen . Hung H. V. . Adem . Muna . Adriaans . Jule . Alvarez-Benjumea . Amalia . Andersen . Henrik K. . Auer . Daniel . Azevedo . Flavio . Bahnsen . Oke . Balzer . Dave . Bauer . Gerrit . Bauer . Paul C. . Baumann . Markus . Baute . Sharon . Benoit . Verena . Bernauer . Julian . Berning . Carl . Berthold . Anna . Bethke . Felix S. . Biegert . Thomas . Blinzler . Katharina . Blumenberg . Johannes N. . Bobzien . Licia . Bohman . Andrea . Bol . Thijs . Bostic . Amie . Brzozowska . Zuzanna . Burgdorf . Katharina . Burger . Kaspar . Busch . Kathrin B. . Carlos-Castillo . Juan . Chan . Nathan . Christmann . Pablo . Connelly . Roxanne . Czymara . Christian S. . Damian . Elena . Ecker . Alejandro . Edelmann . Achim . Eger . Maureen A. . Ellerbrock . Simon . Forke . Anna . Forster . Andrea . Gaasendam . Chris . Gavras . Konstantin . Gayle . Vernon . Gessler . Theresa . Gnambs . Timo . Godefroidt . Amélie . Grömping . Max . Groß . Martin . Gruber . Stefan . Gummer . Tobias . Hadjar . Andreas . Heisig . Jan Paul . Hellmeier . Sebastian . Heyne . Stefanie . Hirsch . Magdalena . Hjerm . Mikael . Hochman . Oshrat . Hövermann . Andreas . Hunger . Sophia . Hunkler . Christian . Huth . Nora . Ignácz . Zsófia S. . Jacobs . Laura . Jacobsen . Jannes . Jaeger . Bastian . Jungkunz . Sebastian . Jungmann . Nils . Kauff . Mathias . Kleinert . Manuel . Klinger . Julia . Kolb . Jan-Philipp . Kołczyńska . Marta . Kuk . John . Kunißen . Katharina . Kurti Sinatra . Dafina . Langenkamp . Alexander . Lersch . Philipp M. . Löbel . Lea-Maria . Lutscher . Philipp . Mader . Matthias . Madia . Joan E. . Malancu . Natalia . Maldonado . Luis . Marahrens . Helge . Martin . Nicole . Martinez . Paul . Mayerl . Jochen . Mayorga . Oscar J. . McManus . Patricia . McWagner . Kyle . Meeusen . Cecil . Meierrieks . Daniel . Mellon . Jonathan . Merhout . Friedolin . Merk . Samuel . 1 . Meyer . Daniel . Micheli . Leticia . Mijs . Jonathan . Moya . Cristóbal . Neunhoeffer . Marcel . Nüst . Daniel . Nygård . Olav . Ochsenfeld . Fabian . Otte . Gunnar . Pechenkina . Anna O. . Prosser . Christopher . Raes . Louis . Ralston . Kevin . Ramos . Miguel R. . Roets . Arne . Rogers . Jonathan . Ropers . Guido . Samuel . Robin . Sand . Gregor . Schachter . Ariela . Schaeffer . Merlin . Schieferdecker . David . Schlueter . Elmar . Schmidt . Regine . Schmidt . Katja M. . Schmidt-Catran . Alexander . Schmiedeberg . Claudia . Schneider . Jürgen . Schoonvelde . Martijn . Schulte-Cloos . Julia . Schumann . Sandy . Schunck . Reinhard . Schupp . Jürgen . Seuring . Julian . Silber . Henning . Sleegers . Willem . Sonntag . Nico . Staudt . Alexander . Steiber . Nadia . Steiner . Nils . Sternberg . Sebastian . Stiers . Dieter . Stojmenovska . Dragana . Storz . Nora . Striessnig . Erich . Stroppe . Anne-Kathrin . Teltemann . Janna . Tibajev . Andrey . Tung . Brian . Vagni . Giacomo . Van Assche . Jasper . van der Linden . Meta . van der Noll . Jolanda . Van Hootegem . Arno . Vogtenhuber . Stefan . Voicu . Bogdan . Wagemans . Fieke . Wehl . Nadja . Werner . Hannah . Wiernik . Brenton M. . Winter . Fabian . Wolf . Christof . Yamada . Yuki . Zhang . Nan . Ziller . Conrad . Zins . Stefan . Żółtak . Tomasz . Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty . Proceedings of the National Academy of Sciences . 28 October 2022 . 119 . 44 . 10.1073/pnas.2203150119. free . 2066/285367 . free .
  3. Wicherts . Jelte M. . Veldkamp . Coosje L. S. . Augusteijn . Hilde E. M. . Bakker . Marjan . van Aert . Robbie C. M. . van Assen . Marcel A. L. M. . Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking . Frontiers in Psychology . 2016 . 7 . 1832 . 10.3389/fpsyg.2016.01832. 27933012 . 5122713 . free .
  4. Harder . Jenna A. . 2020 . The Multiverse of Methods: Extending the Multiverse Analysis to Address Data-Collection Decisions . Perspectives on Psychological Science . en . 15 . 5 . 1158–1177 . 10.1177/1745691620917678 . 1745-6916.
  5. Clayson . Peter E. . 2024-03-01 . Beyond single paradigms, pipelines, and outcomes: Embracing multiverse analyses in psychophysiology . International Journal of Psychophysiology . 197 . 112311 . 10.1016/j.ijpsycho.2024.112311 . 0167-8760. free .
  6. Jung . Kiju . Shavitt . Sharon . Viswanathan . Madhu . Hilbe . Joseph M. . Female hurricanes are deadlier than male hurricanes . Proceedings of the National Academy of Sciences . 17 June 2014 . 111 . 24 . 8782–8787 . 10.1073/pnas.1402786111. 24889620 . 4066510 . free .
  7. Muñoz . John . Young . Cristobal . We Ran 9 Billion Regressions: Eliminating False Positives through Computational Model Robustness . Sociological Methodology . August 2018 . 48 . 1 . 1–33 . 10.1177/0081175018777988. free .
  8. Rohrer . Julia M. . Egloff . Boris . Schmukle . Stefan C. . Probing Birth-Order Effects on Narrow Traits Using Specification-Curve Analysis . Psychological Science . December 2017 . 28 . 12 . 1821–1832 . 10.1177/0956797617723726. 21.11116/0000-0000-3826-7 . free .

This article is licensed under the GNU Free Documentation License. It uses material from the Wikipedia article "Multiverse analysis".

Except where otherwise indicated, Everything.Explained.Today is © Copyright 2009-2024, A B Cryer, All Rights Reserved. Cookie policy.