SWIFT-ActiveScreener: research and development of an intelligent web-based document screening system

Period of Performance: 09/30/2017 - 05/31/2018

$212K

Phase 1 SBIR

Recipient Firm

Sciome, LLC
RESEARCH TRIANGLE PARK, NC 27709
Principal Investigator

Abstract

Project Summary More than 4,000 systematic reviews are performed each year in the fields of environmental health and evidence- based medicine, with each review requiring, on average, between six months to one year of effort to complete. In order to remain accurate, systematic reviews require regular updates after their initial publication, with most reviews out of date within five years. In the screening phase of systematic review, researchers use detailed inclusion/exclusion criteria to decide whether each article in a set of candidate citations is relevant to the research question under consideration. For each article considered, a researcher reads the title and abstract and evaluates its content with respect to the prespecified criteria. A typical review may require screening thousands or tens of thousands of articles in this manner. Under the assumption that it takes a skilled reviewer 30-90 seconds, on average, to screen a single abstract, dual-screening a set of 10,000 abstracts may require between 150 to 500 hours of labor. We have shown in previous work that automated machine learning methods for article prioritization can reduce by more than 50% the human effort required to screen articles for inclusion in a systematic review. Recently, we have further extended these methods and packaged them into a web-based, collaborative systematic review software application called SWIFT-Active Screener. Active Screener has been used successfully to reduce the effort required to screen articles for systematic reviews conducted at a variety of organizations including the National Institute of Environmental Health Science (NIEHS), the United States Environmental Protection Agency (EPA), the United States Department of Agriculture (USDA), The Endocrine Disruption Exchange (TEDX), and the Evidence Based Toxicology Collaboration (EBTC). These early adopters have provided us with an abundance of useful data and user feedback, and we have identified several areas where we can continue to improve our methods and software. Our goal for the current proposal is to conduct additional research and development to make significant improvements to SWIFT-Active Screener, including several innovations that will be necessary for commercial success. The research we propose encompasses three specific aims: (1) Investigate several improvements to statistical algorithms used for article prioritization and recall estimation. We will explore promising avenues for further improving the performance of our existing algorithms and address critical technical issues that limit the applicability of our current methods (Aim 1 ? Improved Statistical Models). (2) Explore ways in which we can improve our models and methods to handle the scenario in which an existing systematic review is updated with new data several years after its initial publication (Aim 2 ? New Methods for Systematic Review Updates). (3) Investigate several questions related to scaling the system to support hundreds to thousands of simultaneous screeners (Aim 3 - Software Engineering for Scalability, Usability and Full Text Extraction).