Estimating the Reproducibility of Experimental Philosophy

Florian Cova; Brent Strickland; Angela Abatista; Aurélien Allard; James Andow; Mario Attie; Renatas Berniūnas; Jordane Boudesseul; James Beebe; Matteo Colombo; Fiery Cushman; Rodrigo Diaz; Noah N’Djaye Nikolai van Dongen; Vilius Dranseika; Brian D. Earp; Antonio Gaitán Torres; Ivar Hannikainen; José V. Hernández-Conde; Wenjia Hu; François Jaquet; Kareem Khalifa; Hanna Kim; Markus Kneer; Joshua Knobe; Miklos Kurthy; Anthony Lantian; Shen yi Liao; Edouard Machery; Tania Moerenhout; Christian Mott; Mark Phelan; Jonathan Phillips; Navin Rambharose; Kevin Reuter; Felipe Romero; Paulo Sousa; Jan Sprenger; Emile Thalabard; Kevin Tobia; Hugo Viciana; Daniel Wilkenfeld; Xiang Zhou

doi:10.1007/s13164-018-0400-9

Estimating the Reproducibility of Experimental Philosophy

Florian Cova, Brent Strickland, Angela Abatista, Aurélien Allard, James Andow, Mario Attie, Renatas Berniūnas, Jordane Boudesseul, James Beebe, Matteo Colombo, Fiery Cushman, Rodrigo Diaz, Noah N’Djaye Nikolai van Dongen, Vilius Dranseika, Brian D. Earp, Antonio Gaitán Torres, Ivar Hannikainen, José V. Hernández-Conde, Wenjia Hu, François JaquetKareem Khalifa, Hanna Kim, Markus Kneer, Joshua Knobe, Miklos Kurthy, Anthony Lantian, Shen yi Liao, Edouard Machery, Tania Moerenhout, Christian Mott, Mark Phelan, Jonathan Phillips, Navin Rambharose, Kevin Reuter, Felipe Romero, Paulo Sousa, Jan Sprenger, Emile Thalabard, Kevin Tobia, Hugo Viciana, Daniel Wilkenfeld, Xiang Zhou

Research output: Contribution to journal › Article (Contribution to Journal) › peer-review

87 Scopus citations

Abstract

Responding to recent concerns about the reliability of the published literature in psychology and other disciplines, we formed the X-Phi Replicability Project (XRP) to estimate the reproducibility of experimental philosophy (osf.io/dvkpr). Drawing on a representative sample of 40 x-phi studies published between 2003 and 2015, we enlisted 20 research teams across 8 countries to conduct a high-quality replication of each study in order to compare the results to the original published findings. We found that x-phi studies – as represented in our sample – successfully replicated about 70% of the time. We discuss possible reasons for this relatively high replication rate in the field of experimental philosophy and offer suggestions for best research practices going forward.

Original language	English
Pages (from-to)	9-44
Number of pages	36
Journal	Review of Philosophy and Psychology
Volume	12
Issue number	1
DOIs	https://doi.org/10.1007/s13164-018-0400-9
State	Published - Mar 2021

Access to Document

10.1007/s13164-018-0400-9

Cite this

Cova, F., Strickland, B., Abatista, A., Allard, A., Andow, J., Attie, M., Berniūnas, R., Boudesseul, J., Beebe, J., Colombo, M., Cushman, F., Diaz, R., N’Djaye Nikolai van Dongen, N., Dranseika, V., Earp, B. D., Torres, A. G., Hannikainen, I., Hernández-Conde, J. V., Hu, W., ... Zhou, X. (2021). Estimating the Reproducibility of Experimental Philosophy. Review of Philosophy and Psychology, 12(1), 9-44. https://doi.org/10.1007/s13164-018-0400-9

@article{77c9810969984519b4f894d064df92c9,

title = "Estimating the Reproducibility of Experimental Philosophy",

abstract = "Responding to recent concerns about the reliability of the published literature in psychology and other disciplines, we formed the X-Phi Replicability Project (XRP) to estimate the reproducibility of experimental philosophy (osf.io/dvkpr). Drawing on a representative sample of 40 x-phi studies published between 2003 and 2015, we enlisted 20 research teams across 8 countries to conduct a high-quality replication of each study in order to compare the results to the original published findings. We found that x-phi studies – as represented in our sample – successfully replicated about 70% of the time. We discuss possible reasons for this relatively high replication rate in the field of experimental philosophy and offer suggestions for best research practices going forward.",

author = "Florian Cova and Brent Strickland and Angela Abatista and Aur{\'e}lien Allard and James Andow and Mario Attie and Renatas Berniūnas and Jordane Boudesseul and James Beebe and Matteo Colombo and Fiery Cushman and Rodrigo Diaz and {N{\textquoteright}Djaye Nikolai van Dongen}, Noah and Vilius Dranseika and Earp, {Brian D.} and Torres, {Antonio Gait{\'a}n} and Ivar Hannikainen and Hern{\'a}ndez-Conde, {Jos{\'e} V.} and Wenjia Hu and Fran{\c c}ois Jaquet and Kareem Khalifa and Hanna Kim and Markus Kneer and Joshua Knobe and Miklos Kurthy and Anthony Lantian and Liao, {Shen yi} and Edouard Machery and Tania Moerenhout and Christian Mott and Mark Phelan and Jonathan Phillips and Navin Rambharose and Kevin Reuter and Felipe Romero and Paulo Sousa and Jan Sprenger and Emile Thalabard and Kevin Tobia and Hugo Viciana and Daniel Wilkenfeld and Xiang Zhou",

note = "Funding Information: Jos{\'e} Hern{\'a}ndez-Conde carried out his work as a Visiting Scholar at the University of Pittsburgh{\textquoteright}s HPS Department. He was financially supported by a PhD scholarship and mobility grant from the University of the Basque Country, and by the Spanish Ministry of Economy and Competitiveness research project No. FFI2014-52196-P. His replication research was supported by the Pittsburgh Empirical Philosophy Lab. Funding Information: Antonio Gait{\'a}n Torres and Hugo Viciana benefited from funding from the Ministerio de Econom{\'i}a y Competitividad for the project “La constituci{\'o}n del sujeto en la interacci{\'o}n social” (Grant No. FFI2015-67569-C2-1-P & FFI2015-67569-C2-2-P). Funding Information: This project could not have been possible without the financial support of multiple organizations. Florian Cova{\textquoteright}s work on this project was supported by a grant from the Cogito Foundation (Grant No. S-131/13, “Towards an Experimental Philosophy of Aesthetics”). Funding Information: This project could not have been possible without the financial support of multiple organizations. Florian Cova{\textquoteright}s work on this project was supported by a grant from the Cogito Foundation (Grant No. S-131/13, “Towards an Experimental Philosophy of Aesthetics”). Brent Strickland{\textquoteright}s work was supported by two grants from the Agence Nationale de la Recherche (Grants No. ANR-10-IDEX-0001-02 PSL*, ANR-10-LABX-0087 IEC). Matteo Colombo, Noah van Dongen, Felipe Romero and Jan Sprenger{\textquoteright}s work was supported by the European Research Council (ERC) through Starting Grant. No. 640638 (“Making Scientific Inferences More Objective”). Rodrigo Diaz and Kevin Reuter would like to acknowledge funding from the Swiss National Science Foundation, Grant No. 100012_169484. Antonio Gait{\'a}n Torres and Hugo Viciana benefited from funding from the Ministerio de Econom{\'i}a y Competitividad for the project “La constituci{\'o}n del sujeto en la interacci{\'o}n social” (Grant No. FFI2015-67569-C2-1-P & FFI2015-67569-C2-2-P). Jos{\'e} Hern{\'a}ndez-Conde carried out his work as a Visiting Scholar at the University of Pittsburgh{\textquoteright}s HPS Department. He was financially supported by a PhD scholarship and mobility grant from the University of the Basque Country, and by the Spanish Ministry of Economy and Competitiveness research project No. FFI2014-52196-P. His replication research was supported by the Pittsburgh Empirical Philosophy Lab. Hanna Kim{\textquoteright}s work was supported by the Pittsburgh Empirical Philosophy Lab. Shen-yi Liao{\textquoteright}s work was supported by the University of Puget Sound Start-up Funding. Tania Moerenhout carried out her work as a Visiting Researcher at the Center for Bioethics and Health Law, University of Pittsburgh, PA (Aug 2016-July 2017). Aur{\'e}lien Allard, Miklos Kurthy, and Paulo Sousa are grateful to Rashmi Sharma for her help in the replication of Knobe & Burra (2006), in particular for her help in translating the demographic questions from English to Hindi. Ivar Hannikainen and Florian Cova would like to thank Uri Simonsohn for his help in discussing the meaning and best interpretation of p-curves. Finally, we would like to thank all the authors of original studies who accepted to take the time to answer our questions, share their original material and data, and discuss the results of our replication attempts with us. Funding Information: Rodrigo Diaz and Kevin Reuter would like to acknowledge funding from the Swiss National Science Foundation, Grant No. 100012_169484. Funding Information: Brent Strickland{\textquoteright}s work was supported by two grants from the Agence Nationale de la Recherche (Grants No. ANR-10-IDEX-0001-02 PSL*, ANR-10-LABX-0087 IEC). Funding Information: Matteo Colombo, Noah van Dongen, Felipe Romero and Jan Sprenger{\textquoteright}s work was supported by the European Research Council (ERC) through Starting Grant. No. 640638 (“Making Scientific Inferences More Objective”). Publisher Copyright: {\textcopyright} 2018, Springer Science+Business Media B.V., part of Springer Nature.",

year = "2021",

month = mar,

doi = "10.1007/s13164-018-0400-9",

language = "Ingl{\'e}s",

volume = "12",

pages = "9--44",

journal = "Review of Philosophy and Psychology",

issn = "1878-5158",

publisher = "Springer Verlag",

number = "1",

}

Cova, F, Strickland, B, Abatista, A, Allard, A, Andow, J, Attie, M, Berniūnas, R, Boudesseul, J, Beebe, J, Colombo, M, Cushman, F, Diaz, R, N’Djaye Nikolai van Dongen, N, Dranseika, V, Earp, BD, Torres, AG, Hannikainen, I, Hernández-Conde, JV, Hu, W, Jaquet, F, Khalifa, K, Kim, H, Kneer, M, Knobe, J, Kurthy, M, Lantian, A, Liao, SY, Machery, E, Moerenhout, T, Mott, C, Phelan, M, Phillips, J, Rambharose, N, Reuter, K, Romero, F, Sousa, P, Sprenger, J, Thalabard, E, Tobia, K, Viciana, H, Wilkenfeld, D & Zhou, X 2021, 'Estimating the Reproducibility of Experimental Philosophy', Review of Philosophy and Psychology, vol. 12, no. 1, pp. 9-44. https://doi.org/10.1007/s13164-018-0400-9

TY - JOUR

T1 - Estimating the Reproducibility of Experimental Philosophy

AU - Cova, Florian

AU - Strickland, Brent

AU - Abatista, Angela

AU - Allard, Aurélien

AU - Andow, James

AU - Attie, Mario

AU - Berniūnas, Renatas

AU - Boudesseul, Jordane

AU - Beebe, James

AU - Colombo, Matteo

AU - Cushman, Fiery

AU - Diaz, Rodrigo

AU - N’Djaye Nikolai van Dongen, Noah

AU - Dranseika, Vilius

AU - Earp, Brian D.

AU - Torres, Antonio Gaitán

AU - Hannikainen, Ivar

AU - Hernández-Conde, José V.

AU - Hu, Wenjia

AU - Jaquet, François

AU - Khalifa, Kareem

AU - Kim, Hanna

AU - Kneer, Markus

AU - Knobe, Joshua

AU - Kurthy, Miklos

AU - Lantian, Anthony

AU - Liao, Shen yi

AU - Machery, Edouard

AU - Moerenhout, Tania

AU - Mott, Christian

AU - Phelan, Mark

AU - Phillips, Jonathan

AU - Rambharose, Navin

AU - Reuter, Kevin

AU - Romero, Felipe

AU - Sousa, Paulo

AU - Sprenger, Jan

AU - Thalabard, Emile

AU - Tobia, Kevin

AU - Viciana, Hugo

AU - Wilkenfeld, Daniel

AU - Zhou, Xiang

N1 - Funding Information: José Hernández-Conde carried out his work as a Visiting Scholar at the University of Pittsburgh’s HPS Department. He was financially supported by a PhD scholarship and mobility grant from the University of the Basque Country, and by the Spanish Ministry of Economy and Competitiveness research project No. FFI2014-52196-P. His replication research was supported by the Pittsburgh Empirical Philosophy Lab. Funding Information: Antonio Gaitán Torres and Hugo Viciana benefited from funding from the Ministerio de Economía y Competitividad for the project “La constitución del sujeto en la interacción social” (Grant No. FFI2015-67569-C2-1-P & FFI2015-67569-C2-2-P). Funding Information: This project could not have been possible without the financial support of multiple organizations. Florian Cova’s work on this project was supported by a grant from the Cogito Foundation (Grant No. S-131/13, “Towards an Experimental Philosophy of Aesthetics”). Funding Information: This project could not have been possible without the financial support of multiple organizations. Florian Cova’s work on this project was supported by a grant from the Cogito Foundation (Grant No. S-131/13, “Towards an Experimental Philosophy of Aesthetics”). Brent Strickland’s work was supported by two grants from the Agence Nationale de la Recherche (Grants No. ANR-10-IDEX-0001-02 PSL*, ANR-10-LABX-0087 IEC). Matteo Colombo, Noah van Dongen, Felipe Romero and Jan Sprenger’s work was supported by the European Research Council (ERC) through Starting Grant. No. 640638 (“Making Scientific Inferences More Objective”). Rodrigo Diaz and Kevin Reuter would like to acknowledge funding from the Swiss National Science Foundation, Grant No. 100012_169484. Antonio Gaitán Torres and Hugo Viciana benefited from funding from the Ministerio de Economía y Competitividad for the project “La constitución del sujeto en la interacción social” (Grant No. FFI2015-67569-C2-1-P & FFI2015-67569-C2-2-P). José Hernández-Conde carried out his work as a Visiting Scholar at the University of Pittsburgh’s HPS Department. He was financially supported by a PhD scholarship and mobility grant from the University of the Basque Country, and by the Spanish Ministry of Economy and Competitiveness research project No. FFI2014-52196-P. His replication research was supported by the Pittsburgh Empirical Philosophy Lab. Hanna Kim’s work was supported by the Pittsburgh Empirical Philosophy Lab. Shen-yi Liao’s work was supported by the University of Puget Sound Start-up Funding. Tania Moerenhout carried out her work as a Visiting Researcher at the Center for Bioethics and Health Law, University of Pittsburgh, PA (Aug 2016-July 2017). Aurélien Allard, Miklos Kurthy, and Paulo Sousa are grateful to Rashmi Sharma for her help in the replication of Knobe & Burra (2006), in particular for her help in translating the demographic questions from English to Hindi. Ivar Hannikainen and Florian Cova would like to thank Uri Simonsohn for his help in discussing the meaning and best interpretation of p-curves. Finally, we would like to thank all the authors of original studies who accepted to take the time to answer our questions, share their original material and data, and discuss the results of our replication attempts with us. Funding Information: Rodrigo Diaz and Kevin Reuter would like to acknowledge funding from the Swiss National Science Foundation, Grant No. 100012_169484. Funding Information: Brent Strickland’s work was supported by two grants from the Agence Nationale de la Recherche (Grants No. ANR-10-IDEX-0001-02 PSL*, ANR-10-LABX-0087 IEC). Funding Information: Matteo Colombo, Noah van Dongen, Felipe Romero and Jan Sprenger’s work was supported by the European Research Council (ERC) through Starting Grant. No. 640638 (“Making Scientific Inferences More Objective”). Publisher Copyright: © 2018, Springer Science+Business Media B.V., part of Springer Nature.

PY - 2021/3

Y1 - 2021/3

N2 - Responding to recent concerns about the reliability of the published literature in psychology and other disciplines, we formed the X-Phi Replicability Project (XRP) to estimate the reproducibility of experimental philosophy (osf.io/dvkpr). Drawing on a representative sample of 40 x-phi studies published between 2003 and 2015, we enlisted 20 research teams across 8 countries to conduct a high-quality replication of each study in order to compare the results to the original published findings. We found that x-phi studies – as represented in our sample – successfully replicated about 70% of the time. We discuss possible reasons for this relatively high replication rate in the field of experimental philosophy and offer suggestions for best research practices going forward.

AB - Responding to recent concerns about the reliability of the published literature in psychology and other disciplines, we formed the X-Phi Replicability Project (XRP) to estimate the reproducibility of experimental philosophy (osf.io/dvkpr). Drawing on a representative sample of 40 x-phi studies published between 2003 and 2015, we enlisted 20 research teams across 8 countries to conduct a high-quality replication of each study in order to compare the results to the original published findings. We found that x-phi studies – as represented in our sample – successfully replicated about 70% of the time. We discuss possible reasons for this relatively high replication rate in the field of experimental philosophy and offer suggestions for best research practices going forward.

UR - https://hdl.handle.net/20.500.12724/12311

UR - http://www.scopus.com/inward/record.url?scp=85131297329&partnerID=8YFLogxK

U2 - 10.1007/s13164-018-0400-9

DO - 10.1007/s13164-018-0400-9

M3 - Artículo (Contribución a Revista)

AN - SCOPUS:85131297329

SN - 1878-5158

VL - 12

SP - 9

EP - 44

JO - Review of Philosophy and Psychology

JF - Review of Philosophy and Psychology

IS - 1

ER -

Estimating the Reproducibility of Experimental Philosophy

Abstract

Access to Document

Other files and links

Cite this