Adaptive rule monitoring system

ABSTRACT

Rule-based techniques are gaining importance with their ability to augment large scale data processing systems. However, there still remain key challenges amongst current rule-based techniques, including rule monitoring, adapting and evaluation. Among these challenges, monitoring the precision of rules is highly important as it enables analysts to maintain the accuracy of a rule-based system. In this paper, we propose an Adaptive Rule Monitoring System (ARMS) for monitoring the precision of rules. The approach employs a combination of machine learning and crowdsourcing techniques. ARMS identifies rules deteriorating the performance of a rule based system, using the feedback receives from the crowd. To enable analysts identifying the imprecise rules, ARMS leverage machine learning algorithms to analyze the crowd's feedback. The evaluation results show that ARMS can identify the imprecise rules more successfully compared to the default practice of the system, which rely exclusively on analysts.

References

1. Bilal Abu-Salih, Pornpit Wongthongtham, Seyed-Mehdi-Reza Beheshti, and Dengya Zhu. 2015. A Preliminary Approach to Domain-Based Evaluation of Users' Trustworthiness in Online Social Networks. In 2015 IEEE International Congress on Big Data, New York City, NY, USA, June 27 - July 2, 2015. 460--466.
2. Michael R Anderson, Michael Cafarella, Yixing Jiang, Guan Wang, and Bochun Zhang. 2014. An integrated development environment for faster feature engineering. Proceedings of the VLDB Endowment 7, 13 (2014), 1657--1660.
3. Peter Bak, Dotan Dolev, and Tali Yatzkar-Haham. 2014. Rule adjustment by visualization of physical location data. (Sept. 11 2014). US Patent App. 14/483,158.
4. Elena Baralis and Paolo Garza. 2002. A lazy approach to pruning classification rules. In Data Mining, 2002. ICDM 2003. Proceedings. 2002 IEEE International Conference on. IEEE, 35--42.
5. Tiffany Barnes and John Stamper. 2008. Toward automatic hint generation for logic proof tutoring using historical student data. In International Conference on Intelligent Tutoring Systems. Springer, 373--382.
6. Joseph E Beck and Beverly Park Woolf. 2000. High-level student modeling with machine learning. In International Conference on Intelligent Tutoring Systems. Springer, 584--593.
7. Amin Beheshti, Boualem Benatallah, and Hamid Reza Motahari-Nezhad. 2018. ProcessAtlas: A scalable and extensible platform for business process analytics. Softw., Pract. Exper. 48, 4 (2018), 842--866.
8. Amin Beheshti, Boualem Benatallah, Reza Nouri, Van Munin Chhieng, HuangTao Xiong, and Xu Zhao. 2017. Coredb: a data lake service. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. ACM, 2451--2454.
9. Seyed-Mehdi-Reza Beheshti, Boualem Benatallah, and Hamid Reza Motahari-Nezhad. 2016. Scalable graph-based OLAP analytics over process execution data. Distributed and Parallel Databases 34, 3 (2016), 379--423.
10. Seyed-Mehdi-Reza Beheshti, Boualem Benatallah, Sherif Sakr, Daniela Grigori, Hamid Reza Motahari-Nezhad, Moshe Chai Barukh, Ahmed Gater, and Seung Hwan Ryu. 2016. Process Analytics - Concepts and Techniques for Querying and Analyzing Process Data. Springer.
11. Seyed-Mehdi-Reza Beheshti, Boualem Benatallah, Srikumar Venugopal, Seung Hwan Ryu, Hamid Reza Motahari-Nezhad, and Wei Wang. 2017. A systematic review and comparative analysis of cross-document coreference resolution methods and tools. Computing 99, 4 (2017), 313--349.
12. Seyed-Mehdi-Reza Beheshti, Srikumar Venugopal, Seung Hwan Ryu, Boualem Benatallah, and Wei Wang. 2013. Big Data and Cross-Document Coreference Resolution: Current State and Future Opportunities. CoRR abs/1311.3987 (2013).
13. Seyed-Mehdi-Reza Beheshti, Alireza Tabebordbar, Boualem Benatallah, and Reza Nouri. 2016. Data Curation APIs. arXiv preprint arXiv:1612.03277 (2016).
14. Seyed-Mehdi-Reza Beheshti, Alireza Tabebordbar, Boualem Benatallah, and Reza Nouri. 2017. On automating basic data curation tasks. In Proceedings of the 26th International Conference on World Wide Web Companion. International World Wide Web Conferences Steering Committee, 165--169.
15. Ron Bekkerman and Matan Gavish. 2011. High-precision phrase-based document classification on a modern scale. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 231--239.
16. Olivier Chapelle and Lihong Li. 2011. An empirical evaluation of thompson sampling. In Advances in neural information processing systems. 2249--2257.
17. Laura Chiticariu, Yunyao Li, and Frederick R Reiss. 2013. Rule-based information extraction is dead! long live rule-based information extraction systems!. In Proceedings of the 2013 conference on empirical methods in natural language processing. 827--832.
18. Benjamin Clement, Didier Roy, Pierre-Yves Oudeyer, and Manuel Lopes. 2014. Online optimization of teaching sequences with multi-armed bandits. In 7th International Conference on Educational Data Mining.
19. Paul Suganthan GC, Chong Sun, Haojun Zhang, Frank Yang, Narasimhan Rampalli, Shishir Prasad, Esteban Arcaute, Ganesh Krishnan, Rohit Deep, Vijay Raghavendra, and others. 2015. Why big data industrial systems need rules and what we can do about it. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data. ACM, 265--276.
20. Chaitanya Gokhale, Sanjib Das, AnHai Doan, Jeffrey F Naughton, Narasimhan Rampalli, Jude Shavlik, and Xiaojin Zhu. 2014. Corleone: hands-off crowdsourcing for entity matching. In Proceedings of the 2014 ACM SIGMOD international conference on Management of data. ACM, 601--612.
21. Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M Henne. 2009. Controlled experiments on the web: survey and practical guide. Data mining and knowledge discovery 18, 1 (2009), 140--181.
22. Yun-En Liu, Travis Mandel, Emma Brunskill, and Zoran Popovic. 2014. Trading Off Scientific Knowledge and User Learning with Multi-Armed Bandits.. In EDM. 161--168.
23. Zakaria Maamar, Sherif Sakr, Ahmed Barnawi, and Seyed-Mehdi-Reza Beheshti. 2015. A Framework of Enriching Business Processes Life-Cycle with Tagging Information. In Databases Theory and Applications - 26th Australasian Database Conference, ADC 2015, Melbourne, VIC, Australia, June 4--7, 2015. Proceedings. 309--313.
24. Tova Milo, Slava Novgorodov, and Wang-Chiew Tan. 2016. Rudolf: interactive rule refinement system for fraud detection. Proceedings of the VLDB Endowment 9, 13 (2016), 1465--1468.
25. Dan Shen, Jean-David Ruvini, and Badrul Sarwar. 2012. Large-scale item categorization for e-commerce. In Proceedings of the 21st ACM international conference on Information and knowledge management. ACM, 595--604.
26. Chong Sun, Narasimhan Rampalli, Frank Yang, and AnHai Doan. 2014. Chimera: Large-scale classification using machine learning, rules, and crowdsourcing. Proceedings of the VLDB Endowment 7, 13 (2014), 1529--1540.
27. Joseph Jay Williams, Juho Kim, Anna Rafferty, Samuel Maldonado, Krzysztof Z Gajos, Walter S Lasecki, and Neil Heffernan. 2016. Axis: Generating explanations at scale with learner sourcing and machine learning. In Proceedings of the Third (2016) ACM Conference on Learning@ Scale. ACM, 379--388.
28. Jun Xie, Chong Sun, Fan Yang, and Narasimhan Rampalli. 2014. Automatic rule coaching. (Sept. 2 2014). US Patent App. 14/475,470.

Index Terms

• Computer systems organization

  • Dependable and fault-tolerant systems and networks

    • Redundancy

  • Embedded and cyber-physical systems

    • Embedded systems

    • Robotics

• Networks

  • Network properties

    • Network reliability