Show simple item record

dc.contributor.authorSchweinsberg, M.
dc.contributor.authorFeldman, M.
dc.contributor.authorStaub, N.
dc.contributor.authorvan den Akker, O.R.
dc.contributor.authorvan Aert, R.C.M.
dc.contributor.authorvan Assen, M.A.L.M.
dc.contributor.authorLiu, Y.
dc.contributor.authorAlthoff, T.
dc.contributor.authorHeer, J.
dc.contributor.authorKale, A.
dc.contributor.authorMohamed, Z.
dc.contributor.authorAmireh, H.
dc.contributor.authorVenkatesh Prasad, V.
dc.contributor.authorBernstein, A.
dc.contributor.authorRobinson, E.
dc.contributor.authorSnellman, K.
dc.contributor.authorAmy Sommer, S.
dc.contributor.authorOtner, S.M.G.
dc.contributor.authorRobinson, D.
dc.contributor.authorMadan, N.
dc.contributor.authorSilberzahn, R.
dc.contributor.authorGoldstein, P.
dc.contributor.authorTierney, W.
dc.contributor.authorMurase, T.
dc.contributor.authorMandl, B.
dc.contributor.authorViganola, D.
dc.contributor.authorStrobl, C.
dc.contributor.authorSchaumans, C.B.C.
dc.contributor.authorKelchtermans, S.
dc.contributor.authorNaseeb, C.
dc.contributor.authorMason Garrison, S.
dc.contributor.authorYarkoni, T.
dc.contributor.authorRichard Chan, C.S.
dc.contributor.authorAdie, P.
dc.contributor.authorAlaburda, P.
dc.contributor.authorAlbers, C.
dc.contributor.authorAlspaugh, S.
dc.contributor.authorAlstott, J.
dc.contributor.authorNelson, A.A.
dc.contributor.authorAriño de la Rubia, E.
dc.contributor.authorArzi, A.
dc.contributor.authorBahník, Š.
dc.contributor.authorBaik, J.
dc.contributor.authorWinther Balling, L.
dc.contributor.authorBanker, S.
dc.contributor.authorAA Baranger, D.
dc.contributor.authorBarr, D.J.
dc.contributor.authorBarros-Rivera, B.
dc.contributor.authorBauer, M.
dc.contributor.authorBlaise, E.
dc.contributor.authorBoelen, L.
dc.contributor.authorBohle Carbonell, K.
dc.contributor.authorBriers, R.A.
dc.contributor.authorBurkhard, O.
dc.contributor.authorCanela, M.-A.
dc.contributor.authorCastrillo, L.
dc.contributor.authorCatlett, T.
dc.contributor.authorChen, O.
dc.contributor.authorClark, M.
dc.contributor.authorCohn, B.
dc.contributor.authorCoppock, A.
dc.contributor.authorCugueró-Escofet, N.
dc.contributor.authorCurran, P.G.
dc.contributor.authorCyrus-Lai, W.
dc.contributor.authorDai, D.
dc.contributor.authorValentino Dalla Riva, G.
dc.contributor.authorDanielsson, H.
dc.contributor.authorRusso, R.D.F.S.M.
dc.contributor.authorde Silva, N.
dc.contributor.authorDerungs, C.
dc.contributor.authorDondelinger, F.
dc.contributor.authorDuarte de Souza, C.
dc.contributor.authorTyson Dube, B.
dc.contributor.authorDubova, M.
dc.contributor.authorMark Dunn, B.
dc.contributor.authorAdriaan Edelsbrunner, P.
dc.contributor.authorFinley, S.
dc.contributor.authorFox, N.
dc.contributor.authorGnambs, T.
dc.contributor.authorGong, Y.
dc.contributor.authorGrand, E.
dc.contributor.authorGreenawalt, B.
dc.contributor.authorHan, D.
dc.contributor.authorHanel, P.H.P.
dc.contributor.authorHong, A.B.
dc.contributor.authorHood, D.
dc.contributor.authorHsueh, J.
dc.contributor.authorHuang, L.
dc.contributor.authorHui, K.N.
dc.contributor.authorHultman, K.A.
dc.contributor.authorJavaid, A.
dc.contributor.authorJi Jiang, L.
dc.contributor.authorJong, J.
dc.contributor.authorKamdar, J.
dc.contributor.authorKane, D.
dc.contributor.authorKappler, G.
dc.contributor.authorKaszubowski, E.
dc.contributor.authorKavanagh, C.M.
dc.contributor.authorKhabsa, M.
dc.contributor.authorKleinberg, B.
dc.contributor.authorKouros, J.
dc.contributor.authorKrause, H.
dc.contributor.authorKrypotos, A.-M.
dc.contributor.authorLavbič, D.
dc.contributor.authorLing Lee, R.
dc.contributor.authorLeffel, T.
dc.contributor.authorYang Lim, W.
dc.contributor.authorLiverani, S.
dc.contributor.authorLoh, B.
dc.contributor.authorLønsmann, D.
dc.contributor.authorWei Low, J.
dc.contributor.authorLu, A.
dc.contributor.authorMacDonald, K.
dc.contributor.authorMadan, C.R.
dc.contributor.authorHjorth Madsen, L.
dc.contributor.authorMaimone, C.
dc.contributor.authorMangold, A.
dc.contributor.authorMarshall, A.
dc.contributor.authorEster Matskewich, H.
dc.contributor.authorMavon, K.
dc.contributor.authorMcLain, K.L.
dc.contributor.authorMcNamara, A.A.
dc.contributor.authorMcNeill, M.
dc.contributor.authorMertens, U.
dc.contributor.authorMiller, D.
dc.contributor.authorMoore, B.
dc.contributor.authorMoore, A.
dc.contributor.authorNantz, E.
dc.contributor.authorNasrullah, Z.
dc.contributor.authorNejkovic, V.
dc.contributor.authorNell, C.S.
dc.contributor.authorArthur Nelson, A.
dc.contributor.authorNilsonne, G.
dc.contributor.authorNolan, R.
dc.contributor.authorO Brien, C.E.
dc.contributor.authorO Neill, P.
dc.contributor.authorO Shea, K.
dc.contributor.authorOlita, Toto
dc.contributor.authorOtterbacher, J.
dc.contributor.authorPalsetia, D.
dc.contributor.authorPereira, B.
dc.contributor.authorPozdniakov, I.
dc.contributor.authorProtzko, J.
dc.contributor.authorReyt, J.-N.
dc.contributor.authorRiddle, T.
dc.contributor.authorAkmal Ridhwan Omar Ali, A.
dc.contributor.authorRopovik, I.
dc.contributor.authorRosenberg, J.M.
dc.contributor.authorRothen, S.
dc.contributor.authorSchulte-Mecklenbeck, M.
dc.contributor.authorSharma, N.
dc.contributor.authorShotwell, G.
dc.contributor.authorSkarzynski, M.
dc.contributor.authorStedden, W.
dc.contributor.authorStodden, V.
dc.contributor.authorStoffel, M.A.
dc.contributor.authorStoltzman, S.
dc.contributor.authorSubbaiah, S.
dc.contributor.authorTatman, R.
dc.contributor.authorThibodeau, P.H.
dc.contributor.authorTomkins, S.
dc.contributor.authorValdivia, A.
dc.contributor.authorDruijff-van de Woestijne, G.B.
dc.contributor.authorViana, L.
dc.contributor.authorVillesèche, F.
dc.contributor.authorDuncan Wadsworth, W.
dc.contributor.authorWanders, F.
dc.contributor.authorWatts, K.
dc.contributor.authorWells, J.D.
dc.contributor.authorWhelpley, C.E.
dc.contributor.authorWon, A.
dc.contributor.authorWu, L.
dc.contributor.authorYip, A.
dc.contributor.authorYoungflesh, C.
dc.contributor.authorYu, J.-C.
dc.contributor.authorZandian, A.
dc.contributor.authorZhang, L.
dc.contributor.authorZibman, C.
dc.contributor.authorLuis Uhlmann, E.
dc.date.accessioned2021-07-27T08:08:51Z
dc.date.available2021-07-27T08:08:51Z
dc.date.issued2021
dc.identifier.citationSchweinsberg, M. and Feldman, M. and Staub, N. and van den Akker, O.R. and van Aert, R.C.M. and van Assen, M.A.L.M. and Liu, Y. et al. 2021. Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis. Organizational Behavior and Human Decision Processes. 165: pp. 228-249.
dc.identifier.urihttp://hdl.handle.net/20.500.11937/84806
dc.identifier.doi10.1016/j.obhdp.2021.02.003
dc.description.abstract

In this crowdsourced initiative, independent analysts used the same dataset to test two hypotheses regarding the effects of scientists’ gender and professional status on verbosity during group meetings. Not only the analytic approach but also the operationalizations of key variables were left unconstrained and up to individual analysts. For instance, analysts could choose to operationalize status as job title, institutional ranking, citation counts, or some combination. To maximize transparency regarding the process by which analytic choices are made, the analysts used a platform we developed called DataExplained to justify both preferred and rejected analytic paths in real time. Analyses lacking sufficient detail, reproducible code, or with statistical errors were excluded, resulting in 29 analyses in the final sample. Researchers reported radically different analyses and dispersed empirical outcomes, in a number of cases obtaining significant effects in opposite directions for the same research question. A Boba multiverse analysis demonstrates that decisions about how to operationalize variables explain variability in outcomes above and beyond statistical choices (e.g., covariates). Subjective researcher decisions play a critical role in driving the reported empirical results, underscoring the need for open data, systematic robustness checks, and transparency regarding both analytic paths taken and not taken. Implications for organizations and leaders, whose decision making relies in part on scientific findings, consulting reports, and internal analyses by data scientists, are discussed.

dc.rights.urihttp://creativecommons.org/licenses/by/4.0/
dc.titleSame data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis
dc.typeJournal Article
dcterms.source.volume165
dcterms.source.startPage228
dcterms.source.endPage249
dcterms.source.issn0749-5978
dcterms.source.titleOrganizational Behavior and Human Decision Processes
dc.date.updated2021-07-27T08:08:51Z
curtin.departmentSchool of Molecular and Life Sciences (MLS)
curtin.accessStatusOpen access
curtin.facultyFaculty of Science and Engineering
curtin.contributor.orcidOlita, Toto [0000-0002-3247-3756]
curtin.contributor.orcidOlita, Toto [0000-0002-3247-3756]


Files in this item

Thumbnail

This item appears in the following Collection(s)

Show simple item record

http://creativecommons.org/licenses/by/4.0/
Except where otherwise noted, this item's license is described as http://creativecommons.org/licenses/by/4.0/