Article

White noise assumptions revisited: regression metamodels and experimental designs in practice

Author:

Jack P. C. KleijnenAuthors Info & Claims

WSC '06: Proceedings of the 38th conference on Winter simulation

Pages 107 - 117

Published: 03 December 2006 Publication History

Abstract

Classic linear regression metamodels and their concomitant experimental designs assume a univariate (not multivariate) simulation response and white noise. By definition, white noise is normally (Gaussian), independently (implying no common random numbers), and identically (constant variance) distributed with zero mean (valid metamodel). This advanced tutorial tries to answer the following questions: (i) How realistic are these classic assumptions in simulation practice? (ii) How can these assumptions be tested? (iii) If assumptions are violated, can the simulation's I/O data be transformed such that the analysis becomes correct? (iv) If such transformations cannot be applied, which alternative statistical methods (for example, generalized least squares, bootstrapping, jackknifing) can then be applied?

References

[1]

Angün, E., D. den Hertog, G. Gürkan, and J. P. C. Kleijnen. 2006. Response surface methodology with stochastic constrains for expensive simulation. Working Paper, Tilburg University, Tilburg, Netherlands.]]

[2]

Arcones, M. A., and Y. Wang. 2006. Some new tests for normality based on U-processes. Statistics & Probability Letters 76(1): 69--82.]]

[3]

Atkinson, A., and M. Riani. 2000. Robust diagnostic regression analysis. New York: Springer.]]

[4]

Ayanso, A., M. Diaby, and S. K. Nair. 2006. Inventory rationing via drop-shipping in Internet retailing: a sensitivity analysis. European Journal of Operational Research 171(1): 135--152.]]

[5]

Breukers, A. 2006. Bio-economic modelling of brown rot in the Dutch potato production chain. Doctoral dissertation, Wageningen University, Wageningen, The Netherlands.]]

[6]

Conover, W. J. 1980. Practical nonparametric statistics, second edition. New York: Wiley:]]

[7]

Conover, W. J., and R. L. Iman. 1981. Rank transformations as a bridge between parametric and nonparametric statistics. The American Statistician 35(3): 124--133.]]

[8]

Davidson, R., and J. G. MacKinnon. 2006. Improving the reliability of bootstrap tests with the fast double bootstrap. Computational Statistics & Data Analysis, in press.]]

Digital Library

[9]

Davison, A. C., and D. V. Hinkley. 1997. Bootstrap methods and their application. Cambridge: Cambridge University Press.]]

[10]

Dykstra, R. L. 1970. Establishing the positive definiteness of the sample covariance matrix. The Annals of Mathematical Statistics 41(6): 2153--2154.]]

[11]

Efron, B., and R. J. Tibshirani. 1993. An introduction to the bootstrap. New York: Chapman & Hall.]]

[12]

Freeman, J., and R. Modarres. 2006. Inverse Box Cox: the power-normal distribution. Statistics & Probability Letters 76(8): 764--772.]]

[13]

Godfrey, L. G. 2006. Tests for regression models with heteroskedasticity of unknown form. Computational Statistics & Data Analysis 50(10): 2715--2733.]]

Digital Library

[14]

Good, P. I. 2005. Resampling methods: a practical guide to data analysis, third edition. Boston: Birkhäuser.]]

Digital Library

[15]

Helton, J. C., J. D. Johnson, C. J. Sallaberry, and C. B. Storlie. 2006. Survey of sampling-based methods for uncertainty and sensitivity analysis. Reliability Engineering and Systems Safety, in press.]]

[16]

Ivanescu, C., W. Bertrand, J. Fransoo, and J. P. C. Kleijnen. 2006. Bootstrapping to solve the limited data problem in production control: an application in batch processing industries. Journal of the Operational Research Society 57(1): 2--9.]]

[17]

Kleijnen, J. P. C. 1987. Statistical tools for simulation practitioners. New York: Marcel Dekker.]]

Digital Library

[18]

Kleijnen, J. P. C. 1992. Regression metamodels for simulation with common random numbers: comparison of validation tests and confidence intervals. Management Science 38(8): 1164--1185.]]

Digital Library

[19]

Kleijnen, J. P. C. 1993. Simulation and optimization in production planning: a case study. Decision Support Systems 9: 269--280.]]

Digital Library

[20]

Kleijnen, J. P. C. 1995. Sensitivity analysis and optimization of system dynamics models: regression analysis and statistical design of experiments. System Dynamics Review 11(4): 275--288.]]

[21]

Kleijnen, J. P. C. 2006. Regression models and experimental designs: a tutorial for simulation analysts. CentER Discussion Paper, number 2006--10.]]

[22]

Kleijnen, J. P. C. 2007. DASE: design and analysis of simulation experiments. Springer Science + Business Media.]]

Digital Library

[23]

Kleijnen, J. P. C., R. C. H. Cheng, and B. Bettovil. 2001. Validation of trace-driven simulation models: bootstrapped tests. Management Science 47(11): 1533--1538.]]

Digital Library

[24]

Kleijnen, J. P. C., P. Cremers, and F. van Belle. 1985. The power of weighted and ordinary least squares with estimated unequal variances in experimental designs. Communications in Statistics, Simulation and Computation 14(1): 85--102.]]

[25]

Kleijnen, J. P. C., and D. Deflandre. 2006. Validation of regression metamodels in simulation: bootstrap approach.European Journal of Operational Research 170(1): 120--131.]]

[26]

Kleijnen, J. P. C., and J. Helton. 1999. Statistical analyses of scatter plots to identify important factors in large-scale simulations, 1: review and comparison of techniques. Reliability Engineering and Systems Safety 65(2): 147--185.]]

[27]

Kleijnen, J. P. C., P. C. A. Karremans, W. K. Oortwijn, and W. J. H. van Groenendaal. 1987. Jackknifing estimated weighted least squares: JEWLS. Communications in Statistics, Theory and Methods 16(3): 747--764.]]

[28]

Kleijnen, J. P. C., J. Kriens, H. Timmermans, and H. Van den Wildenberg. 1989. Regression sampling in statistical auditing: a practical survey and evaluation (including Rejoinder). Statistica Neerlandica 43(4): 193--207 (225).]]

[29]

Kleijnen, J. P. C., and W. C. M. van Beers. 2004. Application-driven sequential designs for simulation experiments: Kriging metamodeling. Journal of the Operational Research Society 55(9): 876--883.]]

[30]

Kleijnen, J. P. C., and W. van Groenendaal. 1992. Simulation: a statistical perspective. Chichester (England): John Wiley.]]

Digital Library

[31]

Kleijnen, J. P. C., and W. van Groenendaal. 1995. Twostage versus sequential sample-size determination in regression analysis of simulation experiments. American Journal of Mathematical and Management Sciences 15(1&2): 83--114.]]

[32]

Kleijnen, J. P. C., G. van Ham, and J. Rotmans. 1992. Techniques for sensitivity analysis of simulation models: a case study of the CO2 greenhouse effect. Simulation 58(6): 410--417.]]

[33]

Law, A. M., and W. D. Kelton. 2000. Simulation modeling and analysis, third edition. Boston: McGraw-Hill.]]

Digital Library

[34]

Lehmann, E. L. 1999. Elements of large-sample theory. New York: Springer.]]

[35]

Lunneborg, C. E. 2000. Data analysis by resampling: concepts and applications. Pacific Grove, California: Duxbury Press.]]

[36]

Rao, C. R. 1959. Some problems involving linear hypothesis in multivariate analysis. Biometrika 46: 49--58.]]

[37]

Rao, C. R. 1967. Least squares theory using an estimated dispersion matrix and its application to measurement of signals. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability 1, 355--372.]]

[38]

Ruud, P. A. 2000. An introduction to classical econometric theory. New York: Oxford University Press.]]

[39]

Salibian-Barrera, M. 2006. Bootstrapping MM-estimators for linear regression with fixed designs. Statistics & Probability Letters, in press.]]

[40]

Saltelli, A., and I. M. Sobol. 1995. About the use of rank transformation in sensitivity analysis of model output. Reliability Engineering and System Safety 50: 225--239.]]

[41]

Simpson, T. W., A. J. Booker, D. Ghosh, A. A. Giunta, P. N. Koch, and R.-J. Yang. 2004. Approximation methods in multidisciplinary analysis and optimization: a panel discussion. Structural and Multidisciplinary Optimization 27(5): 302--313.]]

[42]

Van Beers, W. C. M., and J. P. C. Kleijnen. 2006. Customized sequential designs for random simulation experiments: Kriging metamodeling and bootstrapping. Working Paper, Tilburg University, Tilburg, Netherlands.]]

[43]

Wu, C. F. J., and M. Hamada. 2000. Experiments; planning, analysis, and parameter design optimization. New York: Wiley.]]

Cited By

Horn DDagge MSun XBischl B(2017)First Investigations on Noisy Model-Based Multi-objective Optimization9th International Conference on Evolutionary Multi-Criterion Optimization - Volume 1017310.1007/978-3-319-54157-0_21(298-313)Online publication date: 19-Mar-2017
https://dl.acm.org/doi/10.1007/978-3-319-54157-0_21
Kleijnen JTew J(2007)Regression models and experimental designsProceedings of the 39th conference on Winter simulation: 40 years! The best is yet to come10.5555/1351542.1351580(183-194)Online publication date: 9-Dec-2007
https://dl.acm.org/doi/10.5555/1351542.1351580

White noise assumptions revisited: regression metamodels and experimental designs in practice
1. Computing methodologies
  1. Modeling and simulation
2. Mathematics of computing

Recommendations

Testing parametric assumptions on band- or time-limited signals under noise

This paper considers the problem of testing parametric assumptions on signals f from which only noisy observations y_k=f(τk)+&epsi;_k are available, and where the signal is assumed to be either band-limited or time-limited. To this end, the signal is ...
Robust Noise Suppression Algorithm with the Kalman Filter Theory for White and Colored Disturbance

We propose a noise suppression algorithm with the Kalman filter theory. The algorithm aims to achieve robust noise suppression for the additive white and colored disturbance from the canonical state space models with (i) a state equation composed of the ...
Suboptimal FIR Filtering of Nonlinear Models in Additive White Gaussian Noise

The first- and second-order extended finite impulse response (EFIR1 and EFIR2, respectively) filters are addressed for suboptimal estimation of nonlinear discrete-time state-space models with additive white Gaussian noise. It is shown that, unlike the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WSC '06: Proceedings of the 38th conference on Winter simulation

December 2006

2429 pages

ISBN:1424405017

Editors:
L. Felipe Perrone
Bucknell University
,
Barry G. Lawson
University of Richmond
,
Jason Liu
Colorado School of Mines
,
Frederick P. Wieland
Sensis Corporation
,
General Chair:
David Nicol
University of Illinois, Urbana-Champaign
,
Program Chair:
Richard Fujimoto
College of Computing, Georgia Tech

Sponsors

IIE: Institute of Industrial Engineers
ASA: American Statistical Association
IEICE ESS: Institute of Electronics, Information and Communication Engineers, Engineering Sciences Society
IEEE-CS\DATC: The IEEE Computer Society
SIGSIM: ACM Special Interest Group on Simulation and Modeling
NIST: National Institute of Standards and Technology
(SCS): The Society for Modeling and Simulation International
INFORMS-CS: Institute for Operations Research and the Management Sciences-College on Simulation

Publisher

Winter Simulation Conference

Publication History

Published: 03 December 2006

Check for updates

Qualifiers

Article

Conference

WSC06

Sponsor:

IIE
ASA
IEICE ESS
IEEE-CS\DATC
SIGSIM
NIST
(SCS)
INFORMS-CS

WSC06: Winter Simulation Conference 2006

December 3 - 6, 2006

California, Monterey

Acceptance Rates

WSC '06 Paper Acceptance Rate 177 of 252 submissions, 70%;

Overall Acceptance Rate 3,413 of 5,075 submissions, 67%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
324
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)3

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Horn DDagge MSun XBischl B(2017)First Investigations on Noisy Model-Based Multi-objective Optimization9th International Conference on Evolutionary Multi-Criterion Optimization - Volume 1017310.1007/978-3-319-54157-0_21(298-313)Online publication date: 19-Mar-2017
https://dl.acm.org/doi/10.1007/978-3-319-54157-0_21
Kleijnen JTew J(2007)Regression models and experimental designsProceedings of the 39th conference on Winter simulation: 40 years! The best is yet to come10.5555/1351542.1351580(183-194)Online publication date: 9-Dec-2007
https://dl.acm.org/doi/10.5555/1351542.1351580

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten