Medical researchers at Mount Sinai’s Icahn School of Medicine have completed a new study on artificial intelligence (AI) models to test their capabilities in independent medical practice.
The report highlights that large language models (LLMs) have showcased proficiency in evidence-based medicine (EBM), marking a breakthrough that could open up numerous possibilities for healthcare.
EBM involves using the best available research evidence in treating patients rather than traditional methods or personal beliefs. Employing EBM in modern clinical practice is a herculean task, given the avalanche of new medical research available to healthcare practitioners. Still, researchers say AI chatbots offer a “promising solution” for the complexity.
To arrive at their conclusions, the researchers experimented with OpenAI’s ChatGPT, Gemini, LLAMA v2, and Mixtral-8x7B, giving them access to previously curated case files. The models made clinical decisions based on available data, with researchers testing their performance on several metrics.
In the report, the researchers gauged the effectiveness of LLMs on the resistance to hallucinations, the validity of their clinical decisions, and conformity to guidelines. In the final report, ChatGPT-4 showed the most capability to operate in a clinical setting without human interference over other LLMs.
“LLMs can be made to function as autonomous practitioners of evidence-based medicine,” read the report. “Their ability to utilize tooling can be harnessed to interact with the infrastructure of a real-world healthcare system and perform the tasks of patient management in a guideline-directed manner.”
Despite the proficiency of LLMs in EBM, the researchers identified several areas for improvement in their operations, including the fact that mainstream LLMs typically have a training cutoff in 2021, making them unaware of new medical data after the date. Fine-tuning models with new medical information is a cost-intensive project that researchers say could make their application unfeasible.
Other downsides include the risks of hallucination stemming from asking a model to generate information on an unfamiliar subject matter and the lack of data on cultural concerns and antibiotic resistance.
Circumnavigating the challenges
To improve the performances of LLMs in EBM, the researchers developed a new tool dubbed Retrieval Augmented Generation (RAG) involving the input of task-specific information to AI models.
The researchers opine that using prompt engineering, such as telling the LLM that “You are a professor of medicine,” can improve the quality of responses from the models.
“Limitations were observed in terms of model ability to handle complex guidelines and diagnostic nuances,” said the researchers. “Retrieval Augmented Generation made recommendations more tailored to patients and healthcare systems.”
Emerging technologies like AI and blockchain are making significant inroads in medicine and public health with studies probing cancer detection and epidemic tracking.
In order for artificial intelligence (AI) to work right within the law and thrive in the face of growing challenges, it needs to integrate an enterprise blockchain system that ensures data input quality and ownership—allowing it to keep data safe while also guaranteeing the immutability of data. Check out CoinGeek’s coverage on this emerging tech to learn more why Enterprise blockchain will be the backbone of AI.
Watch: AI, Blockchain, and secret to winning in technology
New to blockchain? Check out CoinGeek’s Blockchain for Beginners section, the ultimate resource guide to learn more about blockchain technology.
