New Event Detection

Period of Performance: 11/21/2001 - 09/25/2002

$98.4K

Phase 1 SBIR

Recipient Firm

Knowledge Analysis Technologies, LLC
4940 Pearl East Circle, Suite 200
Boulder, CO 80301
Principal Investigator

Abstract

Intelligence organizations want to know when an unprecedented event or new information is reported. While there is good technology for searching, tracking, and filtering on known topics, current methods do poorly at detecting something new. The chief mechanism of search and topic tracking, spotting important words, is innapropriate-new stories are not ones with no important words. Because the degree of difference of new and old is different for different topics, uniform thresholds for overlap, as used in current filtering technologies, are also inapproriate. This project approaches the problem in three new ways. First, it applies Latent Semantic Analysis (LSA), a machine-learning technology that simulates human understanding of discourse. After automatic training on a large body of representative text, LSA accurately measures amount of meaning similarity between two passages using all the words in both. Texts with a few words in common are not judged similar if their meaning is different, but are, even if they use entirely different terminology, if their meaning is the same. Second, the system interacts with human users to adapt its criteria to their interests and the characteristics of the data. Third, it uses novel LSA-based storage and retrieval techniques to increase efficiency and capacity. The target innovative new event and new information technology will be of benefit to intelligence agencies and government and private organizations needing to keep abreast of news and developments in commerce, science, and world affairs.