EEGManyPipelines: A Large-scale, Grassroots Multi-analyst Study of Electroencephalography Analysis Practices in the Wild

Truebutschek, Darinka; Yang, Yu-Fang; Gianelli, Claudia; Cesnaite, Elena; Fischer, Nastassja L.; Vinding, Mikkel C.; Marshall, Tom R.; Algermissen, Johannes; Pascarella, Annalisa; Puolivaeli, Tuomas; Vitale, Andrea; Busch, Niko A.; Nilsonne, Gustav

Abstract

The ongoing reproducibility crisis in psychology and cognitive neuroscience has sparked increasing calls to reevaluate and reshape scientific culture and practices. Heeding those calls, we have recently launched the EEGManyPipelines project as a means to assess the robustness of electroencephalography (EEG) research in naturalistic conditions and experiment with an alternative model of conducting scientific research. 168 analyst teams, encompassing 396 individual researchers from 37 countries, independently analyzed the same unpublished, representative EEG dataset to test the same set of pre-defined hypotheses, and then provided  their analysis pipelines and reported outcomes. Here, we lay out how large scale scientific projects can be set up in a grass roots, community driven manner without a central organizing  laboratory. We explain our recruitment strategy, our guidance for analysts, the eventual outputs  of this project and how it might have a lasting impact on the field.