Multimodal Knowledge Acquisition and Management

Period of Performance: 09/27/2011 - 03/27/2013


Phase 2 STTR

Recipient Firm

The Friedland Group, Inc.
330 SW 43rd St., Suite K #489
Seattle, WA 98118
Principal Investigator

Research Institution

University of Rochester
518 Hylan, River Campusbox 27014
Rochester, NY 14627
Institution POC


Automated techniques for harvesting knowledge from documents will make tasks like intelligence gathering significantly faster and more reliable. Information in documents is often distributed between text and non-textual components, which rely upon each other to create a comprehensive picture of what is being conveyed to the reader. The Friedland Group, working with our partner, The University of Rochester, is developing a multimodal knowledge acquisition and management (MKAM) technology specifically to be able to harvest knowledge that is distributed among different modalities in documents. For example, an image containing people could provide details like the age, race, gender, hair and eye color and relative positions of the individuals in the photo. The addition of a caption, e.g. "The Adams Family at their daughter's graduation." provides a deeper context - specifically, who the people are, how they relate to each other and what event are they participating in. Taken individually, text and images each provide only a piece of the puzzle, but adding them together allows us to recover not only the broader context, but important specifics that would otherwise have been missed.