Healthdirect is harnessing AI to improve clinical decision-making and enhance virtual care. We spoke with Dr Nirvana Luckraj, Chief Medical Officer at healthdirect, about the organisation’s AI-based Clinical Decision Support System (CDSS).
The CDSS is an AI tool used by healthdirect nurses to perform symptom-based assessments on the helpline. After a successful trial phase in 2022, the system was officially rolled out across Australia in March 2023. When consumers call the healthdirect helpline with health concerns, the CDSS helps guide nurses through a structured assessment, recommending next steps like seeking emergency care, visiting a doctor, or managing the issue at home.
The healthdirect CDSS is now integrated into their updated online Symptom Checker, launched in December 2023, and the healthcare options available to consumers following a virtual triage process.
Dr Luckraj shares insights into the CDSS below.
- What problem was healthdirect trying to solve?
“We were transitioning our technology platform from an external vendor to internal and as part of that platform transition, we needed to acquire our own CDSS.
It needed to accept multiple symptoms as an input, because our CDSS at the time could only accept a single symptom entry, which doesn’t reflect real-life consultations. You don’t go to the doctor and say, “I have a headache”, and that’s it. You say, “I have a headache, I have a fever, I have a sore throat”. It’s unusual just to go with the one symptom.
It also needed to consider individual risk factors, because two different people presenting with the same symptoms – one with an existing chronic disease and one who’s normally fit and healthy – may require different triage because of their underlying risk.
It needed to have Australia-specific diseases and circumstances such as poisonous snake bites, jellyfish stings, and consider the epidemiology of chronic diseases in Australia.
In the end, it was an AI-enabled model that met those criteria. Our previous model was a decision-tree algorithm. But we wanted something more sophisticated for better accuracy in triage.”
- Could you provide an overview of the AI system?
“It’s a probabilistic Bayesian model that’s used in the CDSS tool and it has an inference engine. So, it has at its core a knowledge base – an evidence-based database that contains relationships between symptoms, diseases, and risk factors – and on top of that lies the inference engine.
Basically, having a core knowledge base with an inference engine helps us then to give a more accurate triage output. It is also easier to maintain because with any changes in evidence you update the knowledge base and that automatically has follow-on effects for the accuracy of the triage.”
- How you know that the CDSS is safe and effective?
“We performed clinical validation together with AIHI and Professor Farah Magrabi. We determined the testing protocol, looking at previous published studies comparing the efficacy and safety of symptom checkers, and used the same clinical vignettes as was done in the Hill (2020) study to conduct the testing.
When we compared our results to the results of that study, we found that it had comparable results. We also did a comparative study to the old CDSS and found the new CDSS was just as safe if not more accurate than the old CDSS, and safe to be implemented as a national virtual triage tool.
Since implementation, we’ve applied strict clinical governance processes to monitoring the tool’s performance. We have encouraged feedback from the nurses on how the CDSS has performed, and if needed we can go back and look at that flow.
We’ve also got an extremely collaborative relationship with the CDSS vendor and a mechanism by which we provide feedback to them to continuously improve the product.”
- What are some barriers and enablers to acceptance of the tool?
“One of the main challenges we faced was getting the nurses to adjust to it to the point where they trusted it. I think we had underestimated the change management required in getting the agents used to a new system.
They had to take a more comprehensive look at, “what should I actually be entering in here?”. If you answer questions with, “I don’t know” where there’s an option of “yes”, “no” or “don’t know”, it’s going to keep asking you questions because it can’t eliminate certain possibilities. The quality of the information they enter at the beginning will determine the quality of the questions asked by the CDSS and the accuracy of the following triage.
So, we ran a few sessions with our service providers’ clinical leaders, taking the nurses through “This is the CDSS, this is how it works” to eliminate that black box effect, where nurses just enter information as something it churns through, and then something is thrown back out at them.
I’m really pleased to say that after a year of using it, the nurses now are far more trusting of the tool. We’ve also put in user interface improvements that have helped with their interactions with the tool.”
- What are your plans for the AI tool’s future?
“The scalability of the CDSS and its capability to integrate additional pathways through the helpline and the Symptom Checker are pivotal to healthdirect’s strategic goals. By enhancing virtual triage and delivering precise recommendations, the CDSS significantly alleviates pressure on the healthcare system. This seamless integration ensures that consumers receive timely and appropriate care, whether it be emergency services, a doctor’s consultation, or self-management at home.”
- Any words of advice for another company or organisation looking to adopt an AI tool or CDSS tool?
“Ensure that a clinical governance framework is applied to it from the very beginning – from deciding what your requirements are, what functions you want it to perform, and if the AI tool is therefore the best choice of product for your requirements. If it is, ensure that the principle of having a clinician in the loop at all times applies, that you validate its safety and quality, that you perform a risk assessment of implementing a new technology.
But what we found at healthdirect was that applying our basic clinical governance principles to the adoption and continuous monitoring of an AI product has worked very well. And if you’ve got strong clinical governance foundations in your organisation, adopting new technology safely should not be too much of a challenge.”