Linguistic and semantic characteristics of articles and peer review reports in social and medical sciences: analysis of articles published in Open Research Central
Recently, initiatives to share peer review data on peer review (Squazzoni et al, 2020) have brought about better understanding of the peer review process across different disciplines (Buljan et al, 2020, Squazzoni et al 2021a, Squazzoni et al 2021b). Some studies (Strang & Siler, 2015; Siler & Strang, 2016) examined the changes from initial submissions to journals to publication. Another study (Teplitskiy & Bakanic, 2016) explored how well peer reviews predict the article’s citation impact. A study of almost half a million peer review reports from 61 journals (Buljan et al, 2020) showed that peer review reports were longer in social science than in medical journals. It also showed that language characteristics were different across disciplines (Buljan et al., 2020).
We used the existence of Open Research Central (ORC) portal, which hosts several journal platforms for post-publication peer review and publishes articles from different research fields in the same platform/journals.
The aim of our study was to examine possible differences between articles and the peer review process in medicine and health sciences vs. social sciences. We examined the structural and linguistic differences between research articles, the characteristics of the peer review process, the language of peer review reports, and the outcomes of peer review reports.
To select the articles, we used the ORC search engine and applied the following filters: “Subject area: Medical and health sciences”, and “Subject area: Social science”. “Article type(s): Research article” filter was also applied. We ended up with 1912 medical and 477 social science articles. To create the samples of articles with clear medical/health vs. social sciences content, we excluded articles with a tag for medicine and health sciences and any other disciplinary field except biology and life sciences, and those with a tag both for social sciences, and biology and life sciences. This left 408 medical and 54 social science articles. Using the DOIs of filtered articles, we downloaded the articles manually in an XML format. Articles that had not been reviewed were excluded, yielding a total of 51 articles with a social sciences tag and 361 articles with a medicine and health sciences tag, as well as their peer review texts.
We analysed them using the Linguistic Inquiry Word Count (LIWC) text analysis software program and word embeddings, a method in which words are given mathematical vector representation. To assess possible differences between the articles and their reviews across Medical and Social sciences, one-way ANOVA and post hoc Tukey’s test were employed. For multivariate frequency distribution of the variables, contingency test was utilised. All analyses were carried out in JASP, Version 0.14.1.
Articles from medicine and health sciences, and social sciences differed in their structure: Introduction and Conclusion were longer in social sciences: median word count in Introduction section was 674 (IQR=308.0-637.0) for social sciences and 431 (IQR=420.5-1029.0) for medical sciences; and 263 (IQR=135.5-516.0) vs. 94 (IQR=64.0-168.0) in Conclusion, respectively (p<0.01, Mann Whitney). Percentage of articles with merged Discussion and Conclusion was higher in social sciences, medical articles followed IMRaD structure more frequently and contained fewer declarations and non-IMRaD sections; they also contained more figures. Linguistic analysis showed that social science articles had higher WC, higher clout, and less positive tone. No statistically significant differences were found in the characteristics of the peer review process and comments from all stages of review between social sciences and medical and health sciences. Peer review reports were significantly longer for articles in social sciences. There were no statistically significant differences in the outcome of the peer review process (approved, approved with reservation or rejected) between the two disciplines. There were also no statistically significant differences in manuscript changes between the disciplines, with the exception of text changes in the introduction section that changed more in the social sciences.
The review process for articles in social and medical/health sciences may not differ as much as is usually considered. This may be due in part to the same platform, which may have uniform policies and processes.
Andrijana Perković Paloš
University of Split School of Medicine
Split, Croatia
ORCID ID: 0000-0003-3048-2023Antonija Mijatović
University of Split School of Medicine
Split, Croatia
ORCID ID: 0000-0003-1733-582XIvan Buljan
University of Split School of Medicine
Split, Croatia
ORCID ID: 0000-0002-8719-7277Daniel Garcia-Costa
University of Valencia
Valencia, Spain
ORCID ID: 0000-0002-8939-8451Francisco Grimaldo
University of Valencia
Valencia, Spain
ORCID ID: 0000-0002-1357-7170Ana Marušić
University of Split School of Medicine
Split, Croatia
ORCID ID: 0000-0001-6272-0917