A hit-miss model for duplicate detection in the WHO drug safety database

ABSTRACT

The WHO Collaborating Centre for International Drug Monitoring in Uppsala, Sweden, maintains and analyses the world's largest database of reports on suspected adverse drug reaction incidents that occur after drugs are introduced on the market. As in other post-marketing drug safety data sets, the presence of duplicate records is an important data quality problem and the detection of duplicates in the WHO drug safety database remains a formidable challenge, especially since the reports are anonymised before submitted to the database. However, to our knowledge no work has been published on methods for duplicate detection in post-marketing drug safety data. In this paper, we propose a method for probabilistic duplicate detection based on the hit-miss model for statistical record linkage described by Copas & Hilton. We present two new generalisations of the standard hit-miss model: a hit-miss mixture model for errors in numerical record fields and a new method to handle correlated record fields. We demonstrate the effectiveness of the hit-miss model for duplicate detection in the WHO drug safety database both at identifying the most likely duplicate for a given record (94.7% accuracy) and at discriminating duplicates from random matches (63% recall with 71% precision). The proposed method allows for more efficient data cleaning in post-marketing drug safety data sets, and perhaps other applications throughout the KDD community.

REFERENCES

• 1: A. Bate, M. Lindquist, I. R. Edwards, S. Olsson, R. Orre, A. Lansner, and R. M. De Freitas. A Bayesian neural network method for adverse drug reaction signal generation. European Journal of Clinical Pharmacology, 54:315--321, 1998.
• 2: T. Belin and D. Rubin. A method for calibrating false-match rates in record linkage. Journal of the American Statistical Association, 90:694--707, 1995.
• 3: Mikhail Bilenko , Raymond J. Mooney, Adaptive duplicate detection using learnable string similarity measures, Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining, August 24-27, 2003, Washington, D.C. [doi>10.1145/956750.956759]
• 4: M. Bilenko and R. J. Mooney. On evaluation and training-set construction for duplicate detection. In Proceedings of the KDD-2003 workshop on data cleaning, record linkage and object consolidation, pages 7--12, 2003.
• 5: E. A. Bortnichak, R. P. Wise, M. E. Salive, and H. H. Tilson. Proactive safety surveillance. Pharmacoepidemiology and Drug Safety, 10:191--196, 2001.
• 6: A. D. Brinker and J. Beitz. Spontaneous reports of thrombocytopenia in association with quinine: clinical attributes and timing related to regulatory action. American Journal of Hematology, 70:313--317, 2002.
• 7: J. Copas and F. Hilton. Record linkage: statistical models for matching computer records. Journal of the Royal Statistical Society: Series A, 153(3):287--320, 1990.
• 8: I. R. Edwards. Adverse drug reactions: finding the needle in the haystack. British Medical Journal, 315(7107):500, 1997.
• 9: I. P. Fellegi and A. B. Sunter. A theory for record linkage. Journal of the American Statistical Association, 64:1183--1210, 1969.
• 10: Mauricio A. Hernández , Salvatore J. Stolfo, The merge/purge problem for large databases, Proceedings of the 1995 ACM SIGMOD international conference on Management of data, p.127-138, May 22-25, 1995, San Jose, California, United States
• 11: M. Lindquist. Data quality management in pharmacovigilance. Drug Safety, 27(12):857--870, 2004.
• 12: A. E. Monge and C. Elkan. An efficient domain-independent algorithm for detecting approximately duplicate database records. In Research Issues on Data Mining and Knowledge Discovery, 1997.
• 13: H. B. Newcombe. Record linkage: the design of efficient systems for linking records into individual family histories. American Journal of Human Genetics, 19:335--359, 1967.
• 14: J. N. Nkanza and W. Walop. Vaccine associated adverse event surveillance (VAEES) and quality assurance. Drug Safety, 27:951--952, 2004.
• 15: R. Orre , A. Lansner , A. Bate , M. Lindquist, Bayesian neural networks with confidence estimations applied to data mining, Computational Statistics & Data Analysis, v.34 n.4, p.473-493, Oct. 28 2000 [doi>10.1016/S0167-9473(99)00114-0 ]
• 16: M. D. Rawlins. Spontaneous reporting of adverse drug reactions. II: Uses. British Journal of Clinical Pharmacology, 1(26):7--11, 1988.
• 17: Sunita Sarawagi , Anuradha Bhamidipaty, Interactive deduplication using active learning, Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, July 23-26, 2002, Edmonton, Alberta, Canada [doi>10.1145/775047.775087]

INDEX TERMS

Primary Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Statistical computing

  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.m Miscellaneous

Additional Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Statistical computing

  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.7 Database Administration
      H.2.8 Database applications
          Subjects: Data mining
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.2 Information Storage
          Subjects: Record classification**

  J. Computer Applications
  J.3 LIFE AND MEDICAL SCIENCES
      Subjects: Health

Keywords:
duplicate detection, hit-miss model, mixture models