Elucidating the effects of peer review: a living synthesis of studies on manuscript changes
Objective: The ability of peer review to improve the scientific endeavour, e.g., conduct, reporting and validity of study findings, is increasingly being questioned (Tennant & Ross-Hellauer, 2020) and calls have been made to showcase changes that occurred to each study due to peer review (Limbu, 2020). Until such transparency is achieved, we are undertaking identification and synthesis of studies that analysed differences between preprints or submitted manuscript and peer reviewed journal publications.
Design: We identified studies based on our knowledge of the field and by checking all research at peer review conferences (presented as podium presentations or posters). We also checked references of identified studies. For all included studies, we then extracted: year of publication, sampling method, conflict of interest, funding, data and protocol sharing, number of analysed version-pairs, sample size calculation, scholarly discipline, method used to compare versions, variables (i.e., manuscript sections) analysed for changes, and metric with which the changes were quantified or qualitatively classified. Future steps will include a search of bibliographic databases (and preprint servers) and launch of an online form that will allow anyone to submit missed studies for inclusion in the review. Current results synthesis is only descriptive but meta-analyses are planned.
Results: Of 25 studies published from 1990 till the end of 2021, 16 (64%) analysed changes between submitted and published papers and 9 (36%) between preprints and published papers. Most commonly, changes were analysed by filling out questionnaires or scales separately for each of the 2 manuscript versions (n=11 [44%]), or by manual comparison of the 2 manuscript versions (n=6 [24%]). Median number of analysed version-pairs was 59 (IQR, 41-122). Most studies analysed changes that occurred to health (n=18 [72%]), or social sciences (n=4 [16%]) manuscripts. Overall, studies’ conclusions indicated very high similarity between version-pairs, with largest changes occurring in introduction and discussion sections.
Conclusions: Current results indicate that submitted or pre-printed manuscript versions and their peer-reviewed journal version are very similar, with main (analysis) methods and main findings rarely changing. Quantification of these results is pending. Large differences between studies, type of changes and methods with which they were measured, indicate greater need for collaboration in the peer-review field and creation of core outcomes measures for manuscript version changes.
Mario Malički
Stanford University
Meta-Research Innovation Center at Stanford (METRICS)
Stanford, CA, USA
ORCID ID: 0000-0003-0698-1930Ana Jerončić
University of Split School of Medicine
Department of Research in Biomedicine and Health
Split, Croatia
ORCID ID: 0000-0003-1621-1956Gerben ter Riet
Amsterdam University of Applied Sciences, Amsterdam University Medical Centers
Urban Vitality Centre of Expertise, Department of Cardiology
Amsterdam, The Netherlands
ORCID ID: 0000-0002-2231-7637Lex Bouter
Vrije Universiteit, Amsterdam University Medical Centers
Department of Philosophy, Department of Epidemiology and Data Science
Amsterdam, The Netherlands
ORCID ID: 0000-0002-2659-5482John P.A. Ioannidis
Stanford University
Meta-Research Innovation Center at Stanford (METRICS)
Stanford, CA, USA
ORCID ID: 0000-0003-3118-6859IJsbrand Jan Aalbersberg
Elsevier
Amsterdam, The Netherlands
ORCID ID: 0000-0002-0209-4480
Steven N. Goodman
Stanford University
Meta-Research Innovation Center at Stanford (METRICS)
Stanford, CA, USA