Wk 09. Validation of Machine Learning Models (Basics)

Lecture Date: October 16, 2023 - Monday
Lecturer: Dr. Hamdi Kavak

In this lecture, we will go through basic steps and techniques of machine learning model validation. Particularly, we will cover machine learning model development steps (supervised and unsupervised), cross-validation, evaluation measures and scores, and finally data-related assessments. This lecture will give methodological rigor to machine learning model evaluation. Two student will present papers assigned for this week (PP 3 and PP 4).

Slides:

Download PDF

Assigned Reading (choose one for your SWA 5):

• Breck, E., Polyzotis, N., Roy, S., Whang, S., & Zinkevich, M. (2019, April). Data Validation for Machine Learning. In MLSys.
• Vabalas, A., Gowen, E., Poliakoff, E., & Casson, A. J. (2019). Machine learning algorithm validation with a limited sample size. PloS one, 14(11), e0224365.

Recommended Reading:

• Ferdinandy, B., Gerencsér, L., Corrieri, L., Perez, P., Újváry, D., Csizmadia, G., & Miklósi, Á. (2020). Challenges of machine learning model validation using correlated behaviour data: evaluation of cross-validation strategies and accuracy measures. PloS one, 15(7), e0236092.
• Ho, S. Y., Wong, L., & Goh, W. W. B. (2020). Avoid oversimplifications in machine learning: going beyond the class-prediction accuracy. Patterns, 1(2), 100025.
• Liem, C. C., & Panichella, A. (2020, June). Oracle Issues in Machine Learning and Where to Find Them. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops (pp. 483-488).
• Maleki, F., Muthukrishnan, N., Ovens, K., Reinhold, C., & Forghani, R. (2020). Machine learning algorithm validation: from essentials to advanced applications and implications for regulatory certification and deployment. Neuroimaging Clinics, 30(4), 433-445.
• Myllyaho, L., Raatikainen, M., Männistö, T., Mikkonen, T., & Nurminen, J. K. (2021). Systematic literature review of validation methods for AI systems. Journal of Systems and Software, 111050.
• Ramezan, C., A Warner, T., & E Maxwell, A. (2019). Evaluation of sampling and cross-validation tuning strategies for regional-scale machine learning classification. Remote Sensing, 11(2), 185.
• Raschka, S. (2018). Model evaluation, model selection, and algorithm selection in machine learning. arXiv preprint arXiv:1811.12808.
• Roberts, D. R., Bahn, V., Ciuti, S., Boyce, M. S., Elith, J., Guillera‐Arroita, G., … & Dormann, C. F. (2017). Cross‐validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecography, 40(8), 913-929.
• Robinson, M. C., & Glen, R. C. (2020). Validating the validation: reanalyzing a large-scale comparison of deep learning and machine learning models for bioactivity prediction. Journal of computer-aided molecular design, 1-14.
• Saeb, S., Lonini, L., Jayaraman, A., Mohr, D. C., & Kording, K. P. (2017). The need to approximate the use-case in clinical machine learning. Gigascience, 6(5), gix019.
• Walsh, I., Fishman, D., Garcia-Gasulla, D., Titma, T., Pollastri, G., Harrow, J., … & Tosatto, S. C. (2021). DOME: recommendations for supervised machine learning validation in biology. Nature methods, 1-6.