Incorporating string transformations in record matching

ABSTRACT

Today's record matching infrastructure does not allow a flexible way to account for synonyms such as "Robert" and "Bob" which refer to the same name, and more general forms of string transformations such as abbreviations. We expand the problem of record matching to take such user-defined string transformations as input. These transformations coupled with an underlying similarity function are used to define the similarity between two strings. We demonstrate the effectiveness of this approach via a fuzzy match operation that is used to lookup an input record against a table of records, where we have an additional table of transformations as input. We demonstrate an improvement in record matching quality and efficient retrieval based on our index structure that is cognizant of transformations.

REFERENCES

• 1: Nick Koudas, Sunita Sarawagi, Divesh Srivastava. Record linkage: Similarity Measures and Algorithms. ACM SIGMOD 2006.
• 2: A. Arasu, S. Chaudhuri and R. Kaushik. Transformation based Framework for Record Matching. IEEE ICDE 2008.
• 3: A. Arasu, V. Ganti and R. Kaushik. Efficient Exact Set Similarity Joins. VLDB 2006.
• 4: S. Chaudhuri, K. Ganjam, V. Ganti, R. Kapoor, V. R. Narasayya, T. Vassilakis. Data cleaning in Microsoft SQL server 2005. ACM SIGMOD 2005.
• 5: A. Gionis, P. Indyk and R. Motwani. Similarity search in high dimensions via hashing. VLDB 1999.
• 6: Microsoft SQL Server Integration Services. http://msdn2.microsoft.com/en-us/library/ms141026.aspx

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.4 Systems

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications

General Terms:
Algorithms, Experimentation, Performance

Keywords:
data cleaning, record matching, transformation rules