Documentation, code, and data for the study "Classifying Positive Results in Clinical Psychology Using Natural Language Processing" by Louis Schiekiera, Jonathan Diederichs & Helen Niemeyer, published in Zeitschrift für Psychologie (2024), Special Issue: Natural Language Processing in Psychology. DOI: 10.1027/2151-2604/a000563 · PDF · TLDR
The best-performing model, SciBERT, was deployed under the name NegativeResultDetector on HuggingFace. It can be used directly via a graphical user interface for single abstract evaluations, or for larger-scale inference by downloading the model files from HuggingFace and using this script from the GitHub repository.
Background: This study addresses the gap in machine learning tools for positive results classification by evaluating the performance of SciBERT, a transformer model pretrained on scientific text, and random forest in clinical psychology abstracts.
Methods: Over 1,900 abstracts were annotated into two categories: 'positive results only' and 'mixed or negative results'. Model performance was evaluated on three benchmarks. The best-performing model was utilized to analyze trends in over 20,000 psychotherapy study abstracts.
Results: SciBERT outperformed all benchmarks and random forest in in-domain and out-of-domain data. The trend analysis revealed non-significant effects of publication year on positive results for 1990–2005, but a significant decrease in positive results between 2005–2022. When examining the entire time-span, significant positive linear and negative quadratic effects were observed.
Discussion: Machine learning could support future efforts to understand patterns of positive results in large data sets. The fine-tuned SciBERT model was deployed for public use.
Table 1
Different metric scores for model evaluation of test data from the annotated MAIN corpus, consisting of n = 198 abstracts authored by researchers affiliated with German clinical psychology departments and published between 2012 and 2022.
| Accuracy | Mixed & Negative Results F1 | Mixed & Negative Results Recall | Mixed & Negative Results Precision | Positive Results Only F1 | Positive Results Only Recall | Positive Results Only Precision | |
|---|---|---|---|---|---|---|---|
| SciBERT | 0.864 | 0.867 | 0.907 | 0.830 | 0.860 | 0.822 | 0.902 |
| Random Forest | 0.803 | 0.810 | 0.856 | 0.769 | 0.796 | 0.752 | 0.844 |
| Extracted p-values | 0.515 | 0.495 | 0.485 | 0.505 | 0.534 | 0.545 | 0.524 |
| Extracted NL Indicators | 0.530 | 0.497 | 0.474 | 0.523 | 0.559 | 0.584 | 0.536 |
| Number of Words | 0.475 | 0.441 | 0.423 | 0.461 | 0.505 | 0.525 | 0.486 |
Across in-domain (MAIN test) and the two out-of-domain validation sets (VAL1, VAL2), SciBERT reached 85–88% accuracy, while the rule-based benchmarks performed near chance (47–57%). Only 9% of abstracts in the corpus mention p-values and only 14% contain predefined natural-language indicators, which leaves rule-based classifiers guessing on the remaining ~79%.
Figure 1
Comparing model performances across in-domain and out-of-domain data; coloured bars represent different model types; samples: MAIN test: n = 198 abstracts; VAL1: n = 150 abstracts; VAL2: n = 150 abstracts.
Trends in psychotherapy RCTs (1990–2022). SciBERT was applied to 20,212 unannotated psychotherapy RCT abstracts. We found no significant linear change in the share of positive results between 1990 and 2005, a significant decrease between 2005 and 2022, and, across the full span, a significant positive linear and negative quadratic effect (an inverted-U shape). A breakpoint analysis located the inflection around 2011, consistent with a time-lag between cultural shifts in research practice and their appearance in the published literature.
This study was conducted as part of the PANNE Project (German acronym for "publication bias analysis of non-publication and non-reception of results in a disciplinary comparison") at Freie Universität Berlin and was funded by the Berlin University Alliance.
If you use the data or the code, please cite the paper as follows:
Schiekiera, L., Diederichs, J., & Niemeyer, H. (2024). Classifying positive results in clinical psychology using natural language processing. Zeitschrift für Psychologie. https://doi.org/10.1027/2151-2604/a000563
BibTeX:
@article{schiekiera2024classifying,
author = {Schiekiera, Louis and Diederichs, Jonathan and Niemeyer, Helen},
title = {Classifying Positive Results in Clinical Psychology Using Natural Language Processing},
journal = {Zeitschrift f{\"u}r Psychologie},
year = {2024},
issue = {Special Issue: Natural Language Processing in Psychology},
doi = {10.1027/2151-2604/a000563}
}