What you see is not what you get!: Detecting Simpson's Paradoxes during Data Exploration

ABSTRACT

Visual data exploration tools, such as Vizdom or Tableau, significantly simplify data exploration for domain experts and, more importantly, novice users. These tools allow to discover complex correlations and to test hypotheses and differences between various populations in an entirely visual manner with just a few clicks, unfortunately, often ignoring even the most basic statistical rules. For example, there are many statistical pitfalls that a user can "tap" into when exploring data sets.

As a result of this experience, we started to build QUDE [1], the first system to Quantifying the Uncertainty in Data Exploration, which is part of Brown's Interactive Data Exploration Stack (called IDES). The goal of QUDE is to automatically warn and, if possible, protect users from common mistakes during the data exploration process. In this paper, we focus on a different type of error, the Simpson's Paradox, which is a special type of error in which a high-level aggregate/visualization leads to the wrong conclusion since a trend reverts when splitting the visualized data set into multiple subgroups (i.e., when executing a drill-down)..

References

1. C. Binnig et al. Toward sustainable insights, or why polygamy is bad for you. In CIDR, 2017.
2. Y. Chung et al. Estimating the impact of unknown unknowns on aggregate query results. In SIGMOD, pages 861--876, 2016.
3. A. Crotty et al. Vizdom: Interactive analytics through pen and touch. PVLDB, 8(12), 2015.
4. C. C. Fabris et al. Discovering surprising instances of simpson's paradox in hierarchical multidimensional data. IJDWM, 2(1):27--49, 2006.
5. C. Glymour et al. Statistical themes and lessons for data mining. Data Min. Knowl. Discov., 1(1):11--28, 1997.
6. R. Kievit et al. Simpson's paradox in psychological science: a practical guide. Frontiers in psychology, 4, 2013.
7. R. Kievit et al. Simpson's paradox in psychological science: a practical guide. Frontiers in Psychology, 4:513, 2013.
8. Y. Kuo et al. Mining surprising patterns and their explanations in clinical data. Applied Artificial Intelligence, 28(2):111--138, 2014.
9. Z. Liu et al. The effects of interactive latency on exploratory visual analysis. IEEE Trans. Vis. Comput. Graph., 20(12), 2014.
10. Mimic ii data set. https://mimic.physionet.org/. Accessed: 2014-11-03.
11. B. Padmanabhan et al. A belief-driven method for discovering unexpected patterns. In KDD, pages 94--100, 1998.
12. A. Silberschatz et al. What makes patterns interesting in knowledge discovery systems. IEEE Trans. Knowl. Data Eng., 8(6):970--974, 1996.
13. E. Suzuki et al. Discovery of surprising exception rules based on intensity of implication. In PKDD, pages 10--18, 1998.
14. Z. Zhao et al. Controlling false discoveries during interactive data exploration. In SIGMOD, 2017.