AI recommendations for common conditions superior to those from physicians

Key takeaways:

• AI recommendations were frequently rated as more optimal and less potentially harmful vs. those by physicians.
• AI was also responsible, only making recommendations when sufficiently confident.

NEW ORLEANS — AI recommendations for managing conditions often treated in primary care were rated superior to those made by physicians, according to a study published in Annals of Internal Medicine.

“I feel like it has the potential to improve the way we practice,” Zehavi Horowitz-Kugler, MD, vice president of medical sciences at K Health, said during the plenary session at the ACP Internal Medicine Meeting.

In the study, four physician adjudicators reviewed 461 primary care visits involving respiratory, dental, urinary, vaginal or eye symptoms — areas “for which prior analysis has shown high AI accuracy,” the researchers noted.

Patients described their symptoms in AI-assisted intake questionnaires, which the AI used to create diagnosis and treatment recommendations such as referrals, prescriptions or laboratory tests.

Adjudicators then compared these recommendations with the final physician recommendations, scoring them as optimal, reasonable, inadequate or potentially harmful.

The AI and physician recommendations were concordant for 56.8% of the visits.

Horowitz-Kugler and colleagues reported that AI recommendations were likelier to be rated as optimal (77.1%; 95% CI, 72.7%-80.9%) vs. physician recommendations (67.1%; 95% CI, 62.9%-71.1%).

AI recommendations were also less likely to be rated as potentially harmful (2.8%; 95% CI, 1.4%-5.2%) compared with decisions from physicians (4.6%; 95% CI, 2.9% to 7.3%).

The scores of AI and physicians were equal in 67.9% (95% CI, 64.8%-70.9%) of cases, better for AI in 20.8% (95% CI, 17.8%-24%) and better for physicians in 11.3% (95% CI, 9%-14.2%).

AI recommendations were rated higher than physician recommendations in 14.4% to 40.8% of visits across all symptom types.

The researchers explained that the AI “outperformed physicians in avoiding unjustified empirical treatments and recognizing key risk factors that may trigger a change in diagnosis or management.”

Horowitz-Kugler also pointed out that the AI acted responsibly, only making recommendations “when sufficiently confident.”

The researchers acknowledged several study limitations, such as a lack of patient follow-up data and limited generalizability, the latter due to the single-center design and limited category of symptoms.

Still, the data “show the potential of AI … to improve medical decision-making in primary care settings,” Horowitz-Kugler said.

In a related editorial, Jerome P. Kassirer, MD, a professor at Tufts University School of Medicine, underlined the importance of creating a criterion for assessing AI technology in clinical settings.

“Medicine is not the only field wrestling with how to implement AI in its quotidian activities, but it is one of the disciplines where the consequences include life, quality of life, and death,” he said. “Thus, the technology must be rigorously examined.”

Ultimately, clinicians “must apply AI tools with sufficient comprehension and discerning clinical judgment,” Kassirer concluded.

References: