Scientists at Johns Hopkins University and the US Food and Drug Administration have created G-AUDIT, a new instrument that spots concealed problems inside the datasets used to train medical artificial intelligence systems.

Although medical AI shows strong promise for advancing healthcare via disease detection, risk forecasting, opportunistic screening, and quantitative analysis, these systems have often fallen short and displayed serious hidden biases.

The team concentrated on biases that arise because the medical data used for training frequently fails to represent the broader population where the AI tools are meant to operate.

The models that drive precision medicine learn to infer clinical outcomes from the data they are trained on. In many cases that works well, but it can also produce surprising failures that are not obvious at first glance, said Mathias Unberath, an expert in AI-assisted medicine at Johns Hopkins University.

Data bias can trigger shortcut learning, fairness problems in AI predictions, and poor generalization to new settings.

AI Relies on Patterns Whether Pertinent or Not

Shortcut learning leads AI to base conclusions on irrelevant patterns. In one case an AI learned to link skin markings made by clinicians with malignant lesions, resulting in 40 percent more false positives when markings appeared on scans during testing.

Another AI was trained on images from two clinics, one focused on cancer and one offering general dermatology care. The clinics employed different imaging equipment of varying quality, and rulers often appeared in cancer clinic photos to monitor tumor growth. The AI readily associated camera quality and ruler presence with cancer risk.

Envision taking this algorithm into the real world with a smartphone camera. There is no ruler. The camera quality is completely irrelevant. But the model learned to associate ruler and camera. You have just created health disparity, Unberath said.

We train models to be the most predictive, but we have zero control over what the model uses to make the prediction, Unberath added.

These shortcuts allow an AI to seem effective at first, yet it will make serious mistakes when applied beyond the original sample data.

G-AUDIT Examines Datasets Directly

While much prior work on medical AI bias has centered on modifying the models themselves, the Johns Hopkins and FDA researchers targeted the datasets and developed Generalized Attribute Utility and Detectability-Induced Bias Testing, or G-AUDIT.

G-AUDIT is built to detect potential issues in datasets before they are used to train medical AI. The tool was evaluated on an image dataset and on a collection of health record texts used to study stigmatizing language by physicians.

In the image dataset G-AUDIT identified image height, image width, and year of collection as the factors carrying the highest risk of causing bias.

In the text data G-AUDIT found that clinical specialty posed the greatest risk of bias, exceeding the influence of patient sex or race for several tasks.

The research team plans to expand the tool and noted that it could be adapted for use outside healthcare as well.