An event based framework for improving information quality that integrates baseline models, causal models and formal reference models

ABSTRACT

We introduce a framework for improving information quality in complex distributed systems that integrates: 1) Analytic models that describe baseline values for attributes and combinations of attributes and components that detect statistically significant changes from baselines. These models determine whether a significant change has occurred, and if so, when. 2) Casual models that help determine why a statistically significant change has occurred and what its impact is. These models focus on the reasons for a change. 3) Formal business and technical reference models so that data and information quality problems are less likely to occur in the future. In this note, we focus on the first two types of models and describe how this framework applies to data quality problems associated with electronic payments transactions and highway traffic patterns.

REFERENCES

• 1: Alan Agresti, An Introduction to Categorical Data Analysis, John Wiley and Sons, Inc., New York, 1996.
• 2: Michèle Basseville , Igor V. Nikiforov, Detection of abrupt changes: theory and application, Prentice-Hall, Inc., Upper Saddle River, NJ, 1993
• 3: Tamraparni Dasu , Theodore Johnson, Exploratory Data Mining and Data Cleaning, John Wiley & Sons, Inc., New York, NY, 2003
• 4: W. Edwards Deming, Elementary Principles of the Statistical Control of Quality: A Series of Lectures, JUSE, Tokyo, 1952.
• 5: Data Mining Group, The Predictive Model Markup Language, Version 3.0, retrieved from www.dmg.org on March 20, 2005.
• 6: DOD Guidelines on Data Quality Management (Summary), retrieved from tricare.osd.mil/rm/documents/fa/DoDGuidelinesOnDataQualityManagement.pdf on March 20, 2004.
• 7: David S. Frankel, Model Driven Architecture, Wiley Publishing Inc., Indianapolis, 2003.
• 8: Glenn W. Goodman Jr., Taming the River of Data: New Software Tools Fuse Intelligence from Many Sources, Defense News, March 14, 2005.
• 9: Robert L. Grossman, H. Bodek, D. Northcutt, and H. V. Poor, Data Mining and Tree-based Optimization, Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, E. Simoudis, J. Han and U. Fayyad, editors, AAAI Press, Menlo Park, California, 1996, pp 323--326.
• 10: Robert L. Grossman , Mark F. Hornick , Gregor Meyer, Data mining standards initiatives, Communications of the ACM, v.45 n.8, August 2002 [doi>10.1145/545151.545180]
• 11: Robert L. Grossman , Richard G. Larson, An Algebraic Approach to Data Mining: Some Examples, Proceedings of the 2002 IEEE International Conference on Data Mining (ICDM'02), p.613, December 09-12, 2002
• 12: Robert L. Grossman, Alert Management Systems: A Quick Introduction, in Managing Cyber Threats: Issues, Approaches and Challenges, edited by Vipin Kumar, Jaideep Srivastava, Aleksandar Lazarevic, Kluwer Academic Publisher, 2004.
• 13: Trevor Hastie, Robert Tibshirani, and Jerome Friedman, The Elements of Statistical Learning, Springer, New York, 2001.
• 14: Yang W. Lee , Diane M. Strong , Beverly K. Kahn , Richard Y. Wang, AIMQ: a methodology for information quality assessment, Information and Management, v.40 n.2, p.133-146, December 2002 [doi>10.1016/S0378-7206(02)00043-5]
• 15: Ken Orr, Data quality and systems theory, Communications of the ACM, v.41 n.2, p.66-71, Feb. 1998 [doi>10.1145/269012.269023]
• 16: Pantheon Gateway Testbed, retrieved from highway.ncdm.uic.edu on March 20, 2005. (A SVG plug in for your browser is required to see the map.)
• 17: Leo L. Pipino , Yang W. Lee , Richard Y. Wang, Data quality assessment, Communications of the ACM, v.45 n.4, April 2002 [doi>10.1145/505248.506010]
• 18: Thomas C. Redman, Data quality: the field guide, Digital Press, Newton, MA, 2001
• 19: James J. Rooney and Lee N. Vanden Heuvel, Root Cause Analysis for Beginners, Quality Prgoress, 2004, pages 45--53.
• 20: Diane M. Strong , Yang W. Lee , Richard Y. Wang, Data quality in context, Communications of the ACM, v.40 n.5, p.103-110, May 1997 [doi>10.1145/253769.253804]
• 21: Shawn Turner, Defining and Measuring Traffic Data Quality, Proceedings of the Traffic Data Quality Workshop, Washington, DC, December 31, 2002.