Article originally published February 2026 in JAMIA OPEN
SUMMARY | Interpretability of an FDA-authorized AI/ML sepsis diagnostic tool improved by SHAP values
Artificial intelligence (AI) tools in healthcare have the potential to improve patient care, but they often operate as “black boxes,” giving answers without explaining how they arrived at them. This lack of transparency makes clinicians hesitant to use them. This study investigated whether adding explanations to an FDA-authorized AI tool for detecting sepsis would help clinicians better understand and trust the technology. We showed 30 clinicians real patient cases using the AI tool. The tool included explanations called SHAP values that show how much each patient characteristic (vitals, labs, etc.) influenced the model’s sepsis risk assessment. We tested whether clinicians could understand these explanations and if they found them helpful. We found clinicians correctly interpreted the explanations 98% of the time, and in every case said they improved their understanding of how the AI generated sepsis risk scores. All 30 clinicians preferred having the explanations versus not having them. These results indicate that SHAP values are a valuable approach to making AI medical tools more transparent, which could help overcome clinician hesitation to use them. This could lead to better adoption of AI tools in patient care and may enhance the impact of this particular AI tool on sepsis care.
Read the full article in JAMIA OPEN.