Reducing overfitting in process model induction

ABSTRACT

In this paper, we review the paradigm of inductive process modeling, which uses background knowledge about possible component processes to construct quantitative models of dynamical systems. We note that previous methods for this task tend to overfit the training data, which suggests ensemble learning as a likely response. However, such techniques combine models in ways that reduce comprehensibility, making their output much less accessible to domain scientists. As an alternative, we introduce a new approach that induces a set of process models from different samples of the training data and uses them to guide a final search through the space of model structures. Experiments with synthetic and natural data suggest this method reduces error and decreases the chance of including unnecessary processes in the model. We conclude by discussing related work and suggesting directions for additional research.

REFERENCES

• 1: Hilde Adé , Luc De Raedt , Maurice Bruynooghe, Declarative Bias for Specific-to-General ILP Systems, Machine Learning, v.20 n.1-2, p.119-154, July/Aug. 1995 [doi>10.1007/BF00993477]
• 2: Arrigo, K. R., Worthen, D. L., & Robinson, D. H. (2003). A coupled ocean-ecosystem model of the Ross Sea: 2. Iron regulation of phytoplankton taxonomic variability and primary production. Journal of Geophysical Research, 108, 3231.
• 3: Åström, K. J., & Eykhoff, P. (1971). System identification---A survey. Automatica, 7, 123--167.
• 4: Elizabeth Bradley , Matthew Easley , Reinhard Stolle, Reasoning about nonlinear system identification, Artificial Intelligence, v.133 n.1-2, p.139-188, December 2001 [doi>10.1016/S0004-3702(01)00143-6 ]
• 5: Leo Breiman, Bagging predictors, Machine Learning, v.24 n.2, p.123-140, Aug. 1996 [doi>10.1023/A:1018054314350 ]
• 6: David S. Bunch , David M. Gay , Roy E. Welsch, Algorithm 717; subroutines for maximum likelihood and quasi-likelihood estimation of parameters in nonlinear regression models, ACM Transactions on Mathematical Software (TOMS), v.19 n.1, p.109-130, March 1993 [doi>10.1145/151271.151279]
• 7: Domingos, P. (1998). Knowledge discovery via multiple models. Intelligent Data Analysis, 2, 187--202.
• 8: Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. New York City: Chapman & Hall.
• 9: Brian Falkenhainer , Kenneth D. Forbus, Compositional modeling: finding the right model for the job, Artificial Intelligence, v.51 n.1-3, p.95-143, Oct. 1991 [doi>10.1016/0004-3702(91)90109-W ]
• 10: Kenneth D. Forbus, Qualitative process theory, Artificial Intelligence, v.24 n.1-3, p.85-168, Dec. 1984 [doi>10.1016/0004-3702(84)90038-9 ]
• 11: Langley, P. (1981). Data-driven discovery of physical laws. Cognitive Science, 5, 31--54.
• 12: Langley, P., George, D., Bay, S., & Saito, K. (2003). Robust induction of process models from time-series data. Proceedings of the Twentieth International Conference on Machine Learning (pp. 432--439). Washington, D.C.: AAAI Press.
• 13: Pat Langley , Javier Sánchez , Ljupco Todorovski , Saso Dzeroski, Inducing Process Models from Continuous Data, Proceedings of the Nineteenth International Conference on Machine Learning, p.347-354, July 08-12, 2002
• 14: J. Ross Quinlan, C4.5: programs for machine learning, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 15: Todorovski, L. (2003). Using domain knowledge for automated modeling of dynamic systems with equation discovery. Doctoral dissertation, Faculty of Computer and Information Science, University of Ljubljana, Ljubljana, Slovenia.
• 16: Todorovski, L., Bridewell, W., Shiran, O., & Langley, P. (in press). Inducing hierarchical process models in dynamic domains. Proceedings of the Twentieth National Conference on Artificial Intelligence. Pittsburgh, PA: AAAI Press.
• 17: Takashi Washio , Hiroshi Motoda , Yuji Niwa, Enhancing the Plausibility of Law Equation Discovery, Proceedings of the Seventeenth International Conference on Machine Learning, p.1127-1134, June 29-July 02, 2000
• 18: Williams, R., & Zipser, D. (1989). A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1, 270--280.
• 19: Zytkow, J. M., Zhu, J., & Hussam, A. (1990). Automated discovery in a chemistry laboratory. Proceedings of the Eighth National Conference on Artificial Intelligence (pp. 89--894). Boston, MA: AAAI Press.