Case Studies

IEEE

Publications Ethics Monitor – An Automated System to Identify Ethical Violations During Peer Review   

Keywords: research integrity, peer review, citation coercion

​

Background

IEEE is the world’s largest technical professional organization dedicated to advancing technology for the benefit of humanity. IEEE publishes nearly a third of the world’s technical literature in electrical engineering, computing, and electronics. With peer review managed across more than 200 journals on the ScholarOne platform, IEEE required an automated solution capable of identifying and monitoring ethical violations at scale, including citation coercion and fraudulent authorship.

 

Solution

IEEE partnered with DCL to develop the Publications Ethics Monitor (PEM) solution to proactively detect citation coercion by monitoring occurrences of citation manipulation during the peer review lifecycle of a manuscript. ScholarOne Notification Services alerts DCL when a manuscript enters the peer-review system, and the PEM process commences. Manuscript files and metadata comprise one or more PDF files that DCL logs in its production control system, unpacks, and analyzes. 

 

Every citation in a PDF manuscript has its full text, corresponding authors, and other related metadata extracted and securely stored in DCL’s production control system. DCL employs GROBID (GeneRation of Bibliographic Data), a machine-learning tool, to extract, parse, and restructure citations that are then validated against CrossCheck, IEEE’s large database of published technical papers (as well as over 6 billion web pages). 

 

DCL engineers developed sophisticated comparison algorithms to account for the significant variation between how author names are entered in an author block versus how they appear in citation text or databases. 

 

As manuscripts go through the peer review process, the PEM system tracks and scores changes that occur during the editorial stages: 

• Differences between author lists throughout the peer review workflow 

• Changes to author names from original submission 

• Mismatches between original manuscript author lists and resubmitted author lists 

• Citations that were added during the peer review process 

• Citations that involve participants of the peer review process 

 

The complexities compound exponentially when you consider the massive volume of papers submitted as well as the 2 to 3 manuscript revision cycles that must be analyzed.  

 

Result

PEM receives notifications from the ScholarOne Notification Services API and initiates analysis when new or final manuscripts are available. The PEM stores the results of each analysis and makes them available to IEEE via an API that includes the ability to flag suspicious author or citation data for further review. With PEM in place to monitor conditions within the peer review system, IEEE can identify and remediate ethical violations before article publication. The system works at a level of scale that is simply not possible with human review. The PEM provides a platform for implementing future functionality for monitoring other issues that may be of concern in IEEE’s peer review system. The PEM has succeeded as a deterrent for bad actors seeking to exploit IEEE’s publications and results in the continued publication of trusted research and science.