Matching and integration across heterogeneous data sources

ABSTRACT

A sea of undifferentiated information is forming from the body of data that is collected by people and organizations, across government, for different purposes, at different times, and using different methodologies. The resulting massive data heterogeneity requires automatic methods for data alignment, matching and/or merging. In this poster, we describe two systems, Guspin™ and Sift™, for automatically identifying equivalence classes and for aligning data across databases. Our technology, based on principles of information theory, measures the relative importance of data, leveraging them to quantify the similarity between entities. These systems have been applied to solve real problems faced by the Environmental Protection Agency and its counterparts at the state and local government level.

REFERENCES

• 1: Ricardo A. Baeza-Yates , Berthier Ribeiro-Neto, Modern Information Retrieval, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1999
• 2: Patrick Pantel , Andrew Philpot , Eduard Hovy, Aligning database columns using mutual information, Proceedings of the 2005 national conference on Digital government research, May 15-18, 2005, Atlanta, Georgia
• 3: Patrick Pantel , Andrew Philpot , Eduard H. Hovy, An information theoretic model for database alignment, Proceedings of the 17th international conference on Scientific and statistical database management, p.14-23, June 27-29, 2005, Santa Barbara, CA
• 4: Shannon, C. E. 1948. A Mathematical Theory of Communication. Bell System Technical Journal, 20:50--64.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.5 Heterogeneous Databases

General Terms:
Algorithms, Experimentation

Keywords:
database alignment, equivalence class detection, information theory, mutual information