Wk 06. Computational Techniques to Support Simulation Model Validation

Lecture Date: September 25, 2023 - Monday
Lecturer: Dr. Hamdi Kavak

Simulation models require employing many algorithmic techniques or heuristics to increase confidence in models. This lecture will focus on three such techniques addressing verification and validation challenges:

1. Warm up periods and steady-state simulation.
2. Calibration and parameter estimation.
3. Sensitivity analysis and sampling.

Slides:

Download PDF

Assigned Reading (choose one for your SWA 2):

• Cevik, M., Ergun, M. A., Stout, N. K., Trentham-Dietz, A., Craven, M., & Alagoz, O. (2016). Using active learning for speeding up calibration in simulation models. Medical Decision Making, 36(5), 581-593.
• Schmidt, R., Voigt, M., & Mailach, R. (2019). Latin hypercube sampling-based Monte Carlo simulation: extension of the sample size and correlation control. In Uncertainty Management for Robust Industrial Design in Aeronautics (pp. 279-289). Springer, Cham.

Recommended Reading:

• Balci, Osman. “Verification, validation, and testing.” Handbook of simulation 10.8 (1998): 335-393.
• Kang, J.-Y., Michels, A., Crooks, A., Aldstadt, J., & Wang, S. (2021). An integrated framework of global sensitivity analysis and calibration for spatially explicit agent-based models. Transactions in GIS, 00, 1– 29. https://doi.org/10.1111/tgis.12837
• Mahajan, P. S., & Ingalls, R. G. (2004, December). Evaluation of methods used to detect warm-up period in steady state simulation. In Proceedings of the 2004 Winter Simulation Conference, 2004. (Vol. 1). IEEE.
• Matala, A. (2008). Sample size requirement for Monte Carlo simulations using Latin hypercube sampling. Helsinki University of Technology, Department of Engineering Physics and Mathematics, Systems Analysis Laboratory, 25.
• Kleijnen, J. P. (2005). An overview of the design and analysis of simulation experiments for sensitivity analysis. European Journal of Operational Research, 164(2), 287-300.
• Saltelli, A., Aleksankina, K., Becker, W., Fennell, P., Ferretti, F., Holst, N., … & Wu, Q. (2019). Why so many published sensitivity analyses are false: A systematic review of sensitivity analysis practices. Environmental modelling & software, 114, 29-39.
• Yu, M., & Fan, W. D. (2017). Calibration of microscopic traffic simulation models using metaheuristic algorithms. International Journal of Transportation Science and Technology, 6(1), 63-77.

Code Examples:

Python Notebook for Calibration - Brittleness to parameter value change
Python Notebook for Latin Hypercube Sampling - Choosing number of runs