Instruct: Identify the main category of Medrxiv papers based on the titles and abstracts
Query: Toward Assessing Clinical Trial Publications for Reporting Transparency ObjectiveTo annotate a corpus of randomized controlled trial (RCT) publications with the checklist items of CONSORT reporting guidelines and using the corpus to develop text mining methods for RCT appraisal.

MethodsWe annotated a corpus of 50 RCT articles at the sentence level using 37 fine-grained CONSORT checklist items. A subset (31 articles) was double-annotated and adjudicated, while 19 were annotated by a single annotator and reconciled by another. We calculated inter-annotator agreement at the article and section level using MASI (Measuring Agreement on Set-Valued Items) and at the CONSORT item level using Krippendorffs . We experimented with two rule-based methods (phrase-based and section header-based) and two supervised learning approaches (support vector machine and BioBERT-based neural network classifiers), for recognizing 17 methodology-related items in the RCT Methods sections.

ResultsWe created CONSORT-TM consisting of 10,709 sentences, 4,845 (45%) of which were annotated with 5,246 labels. A median of 28 CONSORT items (out of possible 37) were annotated per article. Agreement was moderate at the article and section levels (average MASI: 0.60 and 0.64, respectively). Agreement varied considerably among individual checklist items (Krippendorffs = 0.06-0.96). The model based on BioBERT performed best overall for recognizing methodology-related items (micro-precision: 0.82, micro-recall: 0.63, micro-F1: 0.71). Combining models using majority vote and label aggregation further improved precision and recall, respectively.

ConclusionOur annotated corpus, CONSORT-TM, contains more fine-grained information than earlier RCT corpora. Low frequency of some CONSORT items made it difficult to train effective text mining models to recognize them. For the items commonly reported, CONSORT-TM can serve as a testbed for text mining methods that assess RCT transparency, rigor, and reliability, and support methods for peer review and authoring assistance. Minor modifications to the annotation scheme and a larger corpus could facilitate improved text mining models. CONSORT-TM is publicly available at https://github.com/kilicogluh/CONSORT-TM.

Graphical abstract

O_FIG O_LINKSMALLFIG WIDTH=200 HEIGHT=114 SRC="FIGDIR/small/21249695v1_ufig1.gif" ALT="Figure 1">
View larger version (25K):
org.highwire.dtl.DTLVardef@6a6230org.highwire.dtl.DTLVardef@545939org.highwire.dtl.DTLVardef@1adae60org.highwire.dtl.DTLVardef@19359e7_HPS_FORMAT_FIGEXP  M_FIG C_FIG HighlightsO_LIWe constructed a corpus of RCT publications annotated with CONSORT checklist items.
C_LIO_LIWe developed text mining methods to identify methodology-related check-list items.
C_LIO_LIA BioBERT-based model performs best in recognizing adequately reported items.
C_LIO_LIA phrase-based method performs best in recognizing infrequently reported items.
C_LIO_LIThe corpus and the text mining methods can be used to address reporting transparency.
C_LI
26
The categories of this paper are: systems
41
The categories of this paper are: nephrology
31
The categories of this paper are: imaging
54
The categories of this paper are: gynecology
70
The categories of this paper are: otolaryngology
4
The categories of this paper are: reproductive
0
The categories of this paper are: neurology
3
The categories of this paper are: sexual
20
The categories of this paper are: psychiatry
36
The categories of this paper are: economics
18
The categories of this paper are: global
62
The categories of this paper are: orthopedics
57
The categories of this paper are: primary
53
The categories of this paper are: pathology
55
The categories of this paper are: obstetrics
42
The categories of this paper are: occupational