Performance evaluations of comparison-with-a-standard procedures
Abstract
We investigate the performance of a heuristic sequential procedure to compare a finite number of designs with respect to a single standard. The goal is to identify the best design, and if the chosen best design is not the standard, to determine whether the chosen best design is better than the standard. We give preferential status to the standard because there are costs and time involved in replacing the standard. We accomplish this goal by extending indifference-zone selection procedures. An experimental performance evaluation demonstrates the validity and efficiency of our sequential procedures.
References
[1]
Branke, J., S. Chick, and C. Schmidt. 2005. New developments in ranking and selection: an empirical comparison of the three main approaches. In Proceedings of the 2005 Winter Simulation Conference, ed. M. E. Kuhl, N. M. Steiger, F. B. Armstrong, and J. A. Joines, 708--717.
[2]
Chen, E. J. 2004. Using ordinal optimization approach to improve efficency of selection procedures. Journal of Discrete Event Dynamic Systems 14(2):153--170.
[3]
Chen, E. J. 2005. Using Parallel and Distributed Computing to Increase the Capability of Selection Procedures. In Proceedings of the 2005 Winter Simulation Conference, ed. M. E. Kuhl, N. M. Steiger, F. B. Armstrong, and J. A. Joines, 723--731.
[4]
Chen, E. J. 2006. Comparison with a standard via allpairwise comparisons. Journal of Discrete Event Dynamic Systems 16(3):385--403.
[5]
Chen, E. J., and W. D. Kelton. 2005. Sequential selection procedures: using sample means to improve efficiency. European Journal of Operational Research 166(1):133--153.
[6]
Chen, C. H., J. Lin, E. Yücesan, and S. E. Chick. 2000. Simulation budget allocation for further enhancing the efficiency of ordinal optimization. Journal of Discrete Event Dynamic Systems 10: 251--270.
[7]
Edwards, D. G., and J. C. Hsu. 1983. Multiple comparisons with the best treatment. Journal of the American Statistical Association 78:965--971.
[8]
Kim, S.-H. 2005. Comparison with a standard via fully sequential procedures. ACM Transactions on Modeling and Computer Simulation 11:251--273.
[9]
Law, A. M., and W. D. Kelton. 2000. Simulation modeling and analysis. Third ed. New York: McGraw-Hill.
[10]
Nakayama, M. K. 1997. Multiple-comparison procedures for steady-state simulations. Annals of Statistics 25: 2433--2450.
[11]
Nelson, B. L., and D. Goldsman. 2001. Comparisons with a standard in simulation experiments. Management Science 47:449--463.
[12]
Nelson, B. L., and F. J. Matejcik. 1995. Using common random numbers for indifference-zone selection and multiple comparisons in simulation. Management Science 41:1935--1945.
[13]
Rice, J. A. 1995. Mathematical statistics and data analysis. 2nd ed. Belmont, California: Duxbury Press.
[14]
Rinott, Y. 1978. On two-stage selection procedures and related probability inequalities. Communications in Statistics A7:799--811.
[15]
Scheffé, H. 1970. Practical solutions of the Behrens-Fisher problem, Journal of the American Statistical Association 65:1501--1508.
[16]
Tong, Y. L. 1980. Probability inequalities in multivariate distributions. New York:Academic Press.
Recommendations
Comparison with a standard via fully sequential procedures
We develop fully sequential procedures for comparison with a standard. The goal is to find systems whose expected performance measures are larger or smaller than a single system referred to as a standard and, if there is any, to find the one with the ...
A comparison of the ISO working draft standard for SQL and a commercial implementation of SQL
Our is one of many organizations which have adopted the SQL language for use with their database systems. Since portability is a major requirement in our work, we must know how the emerging ISO working draft standard for SQL and the implementation of ...
Performance comparison of the emerging H.264 video coding standard with the existing standards
ICME '03: Proceedings of the 2003 International Conference on Multimedia and Expo - Volume 2The emerging ITU-T H.264 video coding standard has been developed to achieve significant improvements over the existing standards in the compression performance. Although the basic coding framework of the standard is similar to that of the existing ...
Comments
Information & Contributors
Information
Published In
December 2006
2429 pages
ISBN:1424405017
- Editors:
- L. Felipe Perrone,
- Barry G. Lawson,
- Jason Liu,
- Frederick P. Wieland,
- General Chair:
- David Nicol,
- Program Chair:
- Richard Fujimoto
Sponsors
- IIE: Institute of Industrial Engineers
- ASA: American Statistical Association
- IEICE ESS: Institute of Electronics, Information and Communication Engineers, Engineering Sciences Society
- IEEE-CS\DATC: The IEEE Computer Society
- SIGSIM: ACM Special Interest Group on Simulation and Modeling
- NIST: National Institute of Standards and Technology
- (SCS): The Society for Modeling and Simulation International
- INFORMS-CS: Institute for Operations Research and the Management Sciences-College on Simulation
Publisher
Winter Simulation Conference
Publication History
Published: 03 December 2006
Check for updates
Qualifiers
- Article
Conference
WSC06
Sponsor:
- IIE
- ASA
- IEICE ESS
- IEEE-CS\DATC
- SIGSIM
- NIST
- (SCS)
- INFORMS-CS
Acceptance Rates
WSC '06 Paper Acceptance Rate 177 of 252 submissions, 70%;
Overall Acceptance Rate 3,413 of 5,075 submissions, 67%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 81Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 07 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in