ESCRS - POOTH007 - Performance Of Five Large Language Models On Ophthalmology Multiple-Choice Questions: Focus On Cataract, Cornea And Refractive Surgery

Performance Of Five Large Language Models On Ophthalmology Multiple-Choice Questions: Focus On Cataract, Cornea And Refractive Surgery

Published 2026 - 44th Congress of the ESCRS

Reference: POOTH007 | Type: Free paper | DOI: 10.82333/k08k-v503

Authors: Prof. Dr Giulia Coco* 1 , Dr Andrea Taloni 2 , Dr Antonia Carmen Sangregorio 3 , Dr Filippo Lixi 4 , Dr Mario Troisi 5 , Dr Francesco Paolo Mancini 1 , Dr Valerio Calabresi 4 , Dr Feyza Çukurova 4 , Dr Angeli Christy Yu 2 , Prof. Dr Giuseppe Giannaccare 4

1Department of Clinical Sciences and Translational Medicine,University of Rome Tor Vergata,Rome,Italy, 2Department of Translational Medicine,University of Ferrara,Ferrara,Italy, 3Department of Ophthalmology,Magna Graecia University of Catanzaro,Catanzaro,Italy, 4Department of Surgical Sciences,University of Cagliari,Cagliari,Italy, 5Department of Ophthalmology,University of Naples Federico II,Naples,Italy

Purpose

To compare the performance of five publicly available large language models (LLMs) in answering multiple-choice questions from the American Academy of Ophthalmology (AAO) self-assessment tests, with specific focus on cataract/anterior segment, cornea/external diseases, and refractive management/intervention.

Setting

Multicenter academic study conducted at the Departments of Clinical and Translational Medicine, University of Rome Tor Vergata; Translational Medicine, University of Ferrara; Ophthalmology, University Magna Graecia of Catanzaro; Surgical Sciences, University of Cagliari; and Ophthalmology, University of Napoli “Federico II”, Italy.

Methods

In this cross-sectional study, 1020 AAO multiple-choice questions were submitted to five LLMs (ChatGPT-5.2, Claude 4.5 Opus, Gemini 3.0 Pro, DeepSeek-R1, and Grok 4.1). Human performance data were obtained from the AAO platform. Questions were categorized into 10 subspecialties and 3 practice areas (diagnostics, medical treatment, and surgery). Items were defined as “difficult” if >50% of human respondents answered incorrectly and “easy” if >95% answered correctly. Accuracy rates were compared at the question level using chi-square or Fisher’s exact tests, with pairwise LLM comparisons performed using McNemar’s test and Bonferroni correction for multiple testing.

Results

Gemini 3.0 Pro achieved the highest accuracy (96.2%), followed by ChatGPT-5.2 (95.8%), Claude Opus 4.5 (94.3%), Grok 4.1 (93.8%), and DeepSeek R1 (90.7%) (χ²₄=35.055, p<0.001). Humans scored significantly lower (75.6±17.1%, all p<0.001). Only Claude Opus 4.5 and Grok 4.1 varied across subspecialties (all p<0.008). In cataract/anterior segment and cornea/external diseases, humans scored 77.6% and 77.1% versus 89.7–96.9% for LLMs. Refractive management/intervention was most challenging for both (humans 68.3±19.8%; LLMs 83.8–89.7%), and surgery was the hardest domain. LLM performance was consistent across areas (all p>0.06). All LLMs answered all easy questions (n=122) correctly and outperformed humans on difficult ones (n=90; all p<0.001).

Conclusions

All evaluated LLMs achieved over 90% accuracy on AAO self-assessment questions and significantly outperformed human respondents across subspecialties and difficulty levels, highlighting their potential role as advanced educational support tools in ophthalmology.