AI for Eyes

The lack of data available for use in machine learning in ophthalmology greatly limits the clinical value of the diagnostic algorithms the technology provides, warns Professor D Robert Iskander PhD.

“The focus for the future should be on large data sets. Large data sets are expensive, but without them, I would say that machine learning is like a preschool football game,” he said.

TWIN CURSES OF A LOW DATA REGIME AND DIMENSIONALITY

Prof Iskander noted the accuracy of machine learning models increases with the amount of data available. However, data sets in machine learning studies in glaucoma have included an average of only about 200 images, ranging from as few as 60 images, at the low end, to about 3,000 images at the high end.

By contrast, the CIFAR-10 data set—a collection of images used to train machine learning models— consists of 60,000 images split into 10 different classes, with 6,000 images for each.

Moreover, the 10 classes of the CIFAR-10 data set represent a diverse range of images, such as airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. Such classes are much more easily distinguished than the sometimes very subtle changes occurring in the optic nerve of eyes with glaucoma, Prof Iskander observed.

Added to that is the curse of dimensionality, whereby each additional variable exponentially increases the amount of data needed to efficiently separate images into groups. Another of the limitations of working in a small data regime is the potential for overfitting—where a model corresponds too closely to a particular data set to have the same predictive value in a larger and more diverse data set.

TWIDDLING THE KNOBS

The scarcity of data available for machine learning in ophthalmic diagnoses has led some investigators to adopt a range of data manipulation methods designed to enhance an algorithm’s ability to distinguish between groups of images, Prof Iskander said.

As an example, he cited a study conducted by two of his PhD students, Dominika Sulot and Dr David Alonso-Caneiro, which he also co-authored. They investigated glaucoma classification based on scanning laser ophthalmoscopic images using a machine learning technique, with a sample size of only 122 healthy control subjects and 105 glaucoma patients.ⁱ

To reduce the complexity of the process, they used a task-specific convolutional neural network. The team also augmented the data set with image transformations and rotations. They achieved their final algorithm using a deep learning ensemble method, combining multiple learning algorithms to provide a more accurate model than any of the constituent learning algorithms.

The team found by using this approach, they could develop an algorithm that could distinguish between normal and glaucomatous eyes with an accuracy of 93%. Prof Iskander noted that while this is an impressive result, he nonetheless likened the approach to tuning a radio while adding more tuning knobs until achieving the desired result.

“My pragmatic interpretation is that even five- or maybe tenfold cross-validation does not guarantee I will end up with this 93% accuracy if we take another set of 100 images. The result is also post hoc—I mean, I had to provide ground truth for this result for the neural network to tell me I was right,” he said.

A WASTEFUL ENTERPRISE

Because of their limited clinical value, machine learning studies in ophthalmology that use small data sets are ultimately little more than academic exercises, mainly useful for training purposes, Prof Iskander said. The proliferation of such studies is therefore an expensive waste of resources. Meanwhile, gathering and labelling data sets of sufficient size—tens of thousands of images—remains a daunting prospect.

“We should also seek real-world solutions that will provide us with an algorithm that can really be used in a machine: for example, in [segmenting] retinal and corneal layers,” he said.

Prof Iskander noted that a search in Google Scholar using the keywords “machine learning” and “ophthalmology” shows the number of studies published on those topics declined from 70,000 articles between 2001 and 2010 to 60,000 articles between 2011 and 2020. This does not follow the usual trend in scientific inquiry, where there is an exponential increase in the number of studies with each succeeding decade.

“Maybe interest in this particular field is waning because people are starting to think that this is not the way to go. And don’t be surprised if you end up with an essentially similar picture if you change the word ‘ophthalmology’ to ‘medicine’,” he added.

Professor Iskander presented at the Artificial Intelligence in Ophthalmology 2022 virtual meeting.

ⁱ D Sulot et al, PLoS ONE 16(6): e0252339.

Professor D Robert Iskander PhD, Biomedical Signal processing group, Wrocław University of Science and Technology, Wrocław, Poland.

robert.iskander@pwr.edu.pl