Automated Diagnosis of Usability Problems Using Statistical Computational Methods

Period of Performance: 09/02/2003 - 09/02/2004


Phase 1 STTR

Recipient Firm

Aptima, Inc.
12 Gill Street Array
Woburn, MA 01801
Principal Investigator
Firm POC

Research Institution

University of Massachusetts Lowell
1 University Ave.
Lowell, MA 01854
Institution POC


The effects of poor usability range from mere inconvenience to disaster. Human factors specialists employ usability analysis to reduce the likelihood or impact of such failures. However, good usability analysis requires usability reports that are rarely collected, rarely complete, and difficult to analyze. Aptima and the Center for Intelligent Information Retrieval (U. Mass. Amherst) have partnered to develop a usability analysis system that addresses these problems. The system will consist of (1) an interface to elicit useful usability reports in natural language, (2) a text analysis engine that classifies these reports (or existing usability reports) using a validated taxonomy, and (3) an analysis interface for analyzing individual usability reports and trends in usability problems. We will train and test the system using a very large corpus of publicly available usability reports categorized into an extension of the User Action Framework. Aptima and CIIR will deliver a report of results, a software prototype, and a corpus of manually categorized usability reports. The anticipated benefits of the proposed Usability Reports Diagnostic Tool are as follows. The proposed tool will help to collect and classify usability reports. During usability tests, the tool will facilitate the articulation of observed usability problems. The tool's automatic classifier will analyze the resulting usability reports using a text analysis algorithm and categorize them according to a classification schema (also to be developed under this project). The output will identify the problem observed, classify it by type and by cause, and allow further refine by the user. The Usability Reports Diagnostic Tool will be useful for both usability professionals within software development organizations who must generate reports for internal use and customers, and human factors or other usability professionals in customer organizations who are evaluating the usability of the products. Our solution will enable users (and observers such as usability engineers) to express usability problems in their own terms, while systematically categorizing those problems in meaningful way for designers.