Improving The Generalisability Of Chatbot Responses To Cataract Surgery Patient Questions Using Structured Prompt Design: A Comparative Multi-Model Evaluation Of Understandability And Readability

To assess whether large language model (LLM) responses to real patient questions regarding cataract surgery are presented in language understandable to the general population, and whether understandability, actionability, and readability can be improved through the application of a structured prompt compared with the standard, unprompted approach.

Comparative evaluation study of publicly available LLMs (ChatGPT 5.2, Google Gemini 3, Claude Sonnet 4.5, and Microsoft Copilot) using patient-derived cataract surgery questions sourced from online forums of the American Academy of Ophthalmology, Patient.info, and Reddit/AskDocs.

Thirty patient questions regarding cataract surgery were entered verbatim into four publicly available and free-to-use LLMs (ChatGPT 5.2, Google Gemini 3, Claude Sonnet 4.5, and Microsoft Copilot). Responses were assessed using the Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P) for understandability and actionability, alongside Flesch-Kincaid Reading Ease scores. The process was repeated after applying a predefined structured prompt before question entry. Mean scores before and after prompt application were compared. Statistical significance was determined using paired t-tests and a repeated-measures ANOVA to assess inter-model differences. A p-value <0.05 was considered statistically significant.

Baseline PEMAT understandability scores were 69.73 (GPT), 74.07 (Gemini), 69.93 (Copilot), and 70.20 (Claude); p<0.001. Following structured prompting, scores increased to 81.00, 78.87, 76.93, and 79.20, respectively (p<0.0001). Actionability improved from 26.67-36.00 to 60.00-61.63 (p<0.0001). Flesch-Kincaid Reading Ease scores increased from 50.87-56.80 to 63.23-71.83 (p<0.0001), reflecting a shift from high school senior/college student level toward 8th-9th grade readability. Improvements were consistent in all LLMs, demonstrating the robustness of structured prompting in improving patient comprehensibility. Although all models improved significantly, Claude demonstrated comparatively greater gains across domains relative to other LLMs.

The application of a structured prompt significantly improves the communicative quality of LLM responses to cataract surgery queries, particularly regarding actionability. Because post-prompt performance was comparable across all tested models, the structured prompt acts as a model-agnostic tool that helps bridge the gap between technical surgical information and patient understanding. This suggests that rather than relying on inherent model architecture, clinical utilisation can be improved by standardised, validated prompting strategies to ensure AI-generated patient education is accessible and actionable for the target population.