Forecasting Social Science: Evidence from 100 Projects

Abstract:

Increasingly, researchers gather forecasts ex ante about the results of their research studies. Ex post, this allows for a comparison of these forecasts with the results to capture the direction and extent of learning and to counter the “we knew it already” audience response. But what do we know about the accuracy of these forecasts? We use a unique data set from the Social Science Prediction Platform for all 100 of the projects posted in the 2020-24 period, including results for 60% of the projects. This unique data set contains detailed information on the projects and the forecasters, including tracking forecasts across projects. Using this data set, we examine 10 questions about these forecasts: (1) the average predicted effect size: (2) the comparison to realized effect sizes; (3) the predictability of effect sizes with average forecasts. Turning to the accuracy of individual forecasts, we compare (4) academics to non-academics, (5) experts in a field to non-experts, (6) panelists to other forecasters, (7) we estimate the extent of learning over time and (8) the impact of confidence on accuracy. Finally, we consider whether (9) there are superforecasters, with consistently higher accuracy and (10) we estimate the extent of wisdom of crowds. We draw the implications of the findings for forecasting and updating in science.