Cyrus Omar
Jonathan Aldrich
ABSTRACT
One of the pillars of the modern scientific method is model validation: comparing a scientific model's predictions against empirical observations. Today, a scientist demonstrates the validity of a model by making an argument in a paper and submitting it for peer review, a process comparable to code review in software engineering. While human review helps to ensure that contributions meet high-level goals, software engineers typically supplement it with unit testing to get a more complete picture of the status of a project.
We argue that a similar test-driven methodology would be valuable to scientific communities as they seek to validate increasingly complex models against growing repositories of empirical data. Scientific communities differ from software communities in several key ways, however. In this paper, we introduce SciUnit, a framework for test-driven scientific model validation, and outline how, supported by new and existing collaborative infrastructure, it could integrate into the modern scientific process.
- K. Beck. Test Driven Development: By Example. Addison Wesley, 2003. Google Scholar
Digital Library
- G. E. Box and N. R. Draper. Empirical model-building and response surfaces. John Wiley & Sons, 1987. Google ScholarDigital Library
- J. Howison and J. Herbsleb. Scientific software production: incentives and collaboration. In CSCW, pages 513–522. ACM, 2011. Google ScholarDigital Library
- A. E. Jinha. Article 50 million: an estimate of the number of scholarly articles in existence. Learned Publishing, 23(3):258–263, July 2010.Google Scholar
- R. Jolivet, F. Schürmann, T. Berger, R. Naud, W. Gerstner, and A. Roth. The quantitative single-neuron modeling competition. Biological Cybernetics, 99(4):417–426, Nov. 2008. Google ScholarDigital Library
- C. F. Kemerer and M. C. Paulk. The impact of design and code reviews on software quality: An empirical study based on PSP data. IEEE Transactions on Software Engineering, 35(4):534–550, 2009. Google ScholarDigital Library
- F. Perez and B. E. Granger. IPython: a system for interactive scientific computing. Computing in Science & Engineering, 9(3):21––29, 2007. Google ScholarDigital Library
- M. F. Sanner et al. Python: a programming language for software integration and development. J Mol Graph Model, 17(1):57–61, 1999.Google Scholar
- J. Segal. Models of scientific software development. In SECSE, May 2008.Google Scholar
- H. Siy and L. Votta. Does the modern code inspection have value? In ICSM, pages 281–289, 2001. Google ScholarDigital Library
Index Terms
- Collaborative infrastructure for test-driven scientific model validation
Recommendations
Model validation: A bibliometric analysis of the literature
AbstractValidation is a crucial step in environmental and economic modeling that establishes the reliability of models to be used in decision-making contexts. It is often said that validation approaches proposed in the literature are not ...
Highlights- We conduct citation and text-mining analyses on a broad model validation literature.
Test-Driven Development in scientific software: a survey
Scientific software developers are increasingly employing various software engineering practices. Specifically, scientists are beginning to use Test-Driven Development (TDD). Even with this increasing use of TDD, the effect of TDD on scientific software ...
Model validation failure in class imbalance problems
Highlights- Model validation is inherently difficult under class imbalance where minority class is rare in absolute sense.
AbstractFor a classification task, multiple classification models can be built from the training set in various ways. In general, the best-performing model is selected for deployment through a model validation procedure. However, even if the ...
Comments