Study shows artificial intelligence can determine race based on medical images

Artificial intelligence (AI) is used in a variety of healthcare settings, from analyzing medical images to assisting with surgical procedures. While AI can sometimes surpass trained clinicians, these superhuman abilities are not always fully understood.

In a recently published study in The Lancet’s digital healthResearchers found that AI models could accurately predict self-reported race in several different types of X-ray images — a task human experts cannot do. These results suggest that racial information could be unknowingly incorporated into image analysis models, potentially exacerbating racial disparities in the medical setting.

“AI has immense potential to revolutionize the diagnosis, treatment and monitoring of numerous diseases and conditions, and could dramatically transform the way we approach healthcare,” said the study’s first author and NIBIB Data and Technology Advancement ( DATA) National Service Scholar Judy Gichoya, MD “However, for AI to truly benefit all patients, we need to better understand how these algorithms make their decisions to avoid unintended biases.”

The concept of bias in AI algorithms is not new. Research studies have shown that AI performance can be influenced by demographics, including race. There are several potential factors that could introduce bias into AI algorithms, such as: B. Using data sets that are not representative of a patient population (e.g. using data sets where most patients are white). In addition, confounders—traits or phenotypes that are disproportionately present in subgroup populations (e.g., racial differences in breast or bone density)—can also introduce bias. The current study highlights another potential factor that could introduce unintended biases into AI algorithms.

For their study, Gichoya and colleagues first wanted to see if they could develop AI models that could identify breeds based on chest X-rays alone. They used three large datasets spanning a diverse patient population and found that their models could predict race with high accuracy — a remarkable finding since human experts cannot make such predictions by looking at X-rays. The researchers also found that the AI ​​could identify self-reported breeds even when the images were severely degraded, cropped to a ninth of their original size, or when the resolution was changed so much that the images were barely recognized as X-. rays. The research team then used other extra-thoracic X-ray datasets, including mammograms, cervical spine X-rays, and computed tomography (CT) scans of the chest, and found that the AI ​​was still able to determine the self-reported race regardless of the type of scan or anatomy location .

“Our results suggest that there are ‘hidden signals’ in medical images that drive AI to predict race,” Gichoya said. “We need to accelerate our understanding of why these algorithms have this capability so that the downstream applications of AI — like creating image-based algorithms to make predictions about health — aren’t potentially harmful to minority and underserved patient populations.”

The researchers tried to understand how the AI ​​could make these predictions. They examined a variety of different confounders that could potentially affect features in X-ray images, such as B. Body mass index (BMI), breast density, bone density or disease distribution. They could not identify a specific factor that could explain the AI’s ability to accurately predict self-reported breeds. In short, while AI can be trained to predict races from medical images, the information the models use to make those predictions has yet to be uncovered.

“There was a consideration that if developers ‘hide’ demographic factors – such as race, gender or socioeconomic status – from the AI ​​model, the resulting algorithm will not be able to discriminate based on such characteristics and therefore ‘fair .’ This work underscores that this simplistic view is not a viable option to ensure equity in AI and machine learning,” said NIBIB DATA grantee Rui Sá, Ph.D. “We need to recognize the potential limitations of AI and adapt our methods to make sure AI is fair to everyone.”

Provided by the National Institute of Biomedical Imaging and Bioengineering