benchmarking-psychology

This repository contains code and data used for "Putting psychology to the test: rethinking model evaluation through benchmarking and prediction", published in Advances in Methods and Practices in Psychological Science (https://journals.sagepub.com/doi/pdf/10.1177/25152459211026864).
This repo is organized as follows:

• manylabs1_data contains data from Many Labs 1 (Klein et al., 2014; data available at https://osf.io/wx7ck/);
• Minor preprocessing of the data (i.e. removal of unused columns and regex replacement in empty cells) was performed using preprocess_data.py;
• manylabs1.ipynb contains the re-analysis of Many Labs 1 data discussed in the section Model evaluation in psychology;
• Code for the simulation reported in Figure 3 can be found in simulation.ipynb

Authors: Roberta Rocca & Tal Yarkoni - University of Texas at Austin