The AAAiH Algorithm Team have been working hard over the last 18 months to develop a framework which they’ve termed ‘Translational Evaluation of Healthcare Artificial Intelligence (TEHAI)’. The framework has been developed to assess the functional, utility and ethical aspects of AI systems and was reviewed by an eight-member international expert panel from the UK, USA and New Zealand prior to finalisation.
About TEHAI
The TEHAI evaluation framework is now available as an open access publication in BMJ Health & Care Informatics. TEHAI can be applied to the evaluation of clinical AI in research settings and more broadly to guide evaluation of working clinical systems.
It addresses the gaps in implementing AI in healthcare that have limited its adoption including inappropriate or incomplete evaluation of AI systems, a lack of internationally recognised AI standards on evaluation and little guidance on assessing the functional, utility and ethical components of AI systems in healthcare. In developing the framework, the Team undertook an extensive literature review of existing evaluation frameworks.
The key elements of TEHAI that differentiates it from other frameworks are its emphasis on translational and ethical features and its ability to be applied at any stage of the development and deployment of the AI system.
TEHAI has three main components:
- Capability: assesses the intrinsic technical capability of the AI system to perform its expected purpose by reviewing key aspects as to how the AI system was developed.
- Utility: evaluates the usability of the AI system across different dimensions including the contextual relevance and safety and ethical considerations regarding eventual deployment into clinical practice. It also assesses the efficiency of the system.
- Adoption: assesses the translational value of the AI system by evaluating key elements that demonstrate the adoption of the model in real life settings.
Next steps for TEHAI
The team is now applying the evaluation framework via the COVIDENCE systematic review platform to hundreds of COVID-related AI studies that were sourced from a systematic review of over 2000 articles. The results of the study are expected to be published later in 2021.
Meet the TEHAI Team
- Sandeep Reddy, Deakin University
- Wendy Rogers, Macquarie University
- Ville-Petteri Makinen, South Australian Health and Medical Research Institute
- Enrico Coiera, Macquarie University
- Pieta Brown, Orion Health (NZ)
- Markus Wenzel, Fraunhofer Institute for Telecommunications Heinrich-Hertz-Institute (Berlin, Germany)
- Eva Weicken, Fraunhofer Institute for Telecommunications Heinrich-Hertz-Institute (Berlin, Germany)
- Saba Ansari, Deakin University
- Piyush Mathur, Anesthesiology Institute Cleveland Clinic (Ohio, USA)
- Aaron Casey, South Australian Health and Medical Research Institute
- Blair Kelly, Deakin University
For more information
Please contact sandeep.reddy@deakin.edu.au
Reddy S, Rogers W, Makinen V, et al. Evaluation framework to guide implementation of AI systems into healthcare settings, BMJ Health & Care Informatics 2021;28:e100444.