Validation methods to promote real-world applicability of machine learning in medicine
DescriptionThe impact of AI on health care has been dramatic; however, there is a considerable degree of skepticism among clinicians about the real-world applicability of advanced predictive models;
for this reason, it is particularly important to emphasize to students the need for proper model validation in machine learning education. Often model skepticism is well-placed as modelers
may overclaim the real-world replicability for their models, understate the known limitations, or simply not be aware of the hidden limits of the modeling approach. Rigorous and thorough justification of all model design decisions may not be practical given model complexity. This also becomes more challenging as state-of-the-art models with the highest benchmark accuracy are becoming less interpretable, e.g. ensemble methods or deep learning. However, in the same way, that test-driven development has been a successful paradigm to navigate the complex coding landscape through a focus on testable results, we have observed a similar improvement in modeling strategy when the focus of a predictive model is driven by validation targets rather than more abstract, theoretical concerns. In this paper, we provide an overview of the common limitations of model validation methods
typically used in medicine. We then present solutions to address such limitations, with a focus on strengthening the replicability of predictive models. We anticipate that healthcare providers prepared with properly trained and validated models will thus be more likely to adopt machine learning strategies to guide clinical decisions and practice.