Safe AI in Healthcare: We Cannot Manage the Risks We Cannot See
Safe AI in Healthcare
Artificial intelligence is moving into healthcare faster than the patient safety approaches around it are developing.
Healthcare organizations are adopting AI to summarize clinical information, support decision-making, prioritize patients, draft documentation, identify risks, and streamline clinical and administrative workflows. The potential is enormous, but AI also introduces a different patient safety challenge: organizations are increasingly being asked to evaluate, implement, and oversee technologies that can change rapidly, behave unpredictably, and influence care in ways that may be difficult to detect.
At the Mid-Atlantic Patient Safety Center’s 2026 Annual Patient Safety Conference, we explored this challenge with Raj Ratwani and Lucy Bockneck from the MedStar Health National Center for Human Factors in Healthcare. During that discussion, we asked the audience a simple but important question:
Does your organization currently have the ability to identify and collect patient safety issues associated with the use of AI?
The response was a nearly unanimous “no,” which should concern all of us.
As AI becomes increasingly integrated into healthcare, we may not yet have reliable ways to recognize when it contributes to a near miss, unsafe condition, or patient harm. Before we can analyze AI-related safety events, learn from them, or prevent them from recurring, we first have to be able to see them.
Today, collectively, we often cannot.
AI Creates a Different Kind of Safety Challenge
Healthcare has decades of experience evaluating medications, devices, clinical practices, and improvement interventions. AI does not always fit neatly into those traditional approaches.
Generative AI is probabilistic, so the same input may not always produce the same output. Models change, vendors update products, and clinical workflows evolve. A tool that performs well in one environment may perform differently in another.
Even keeping a “human in the loop” is not, by itself, an adequate safety strategy. Clinicians can over-rely on recommendations, fail to recognize subtle errors, or become less likely to question technology they have come to trust. Safe implementation requires thoughtful workflow design, clear expectations, ongoing monitoring, and an understanding of where AI may fail.1
Healthcare organizations are often having to develop these approaches from the ground up.
They are asking many of the same questions: What should we evaluate before adopting an AI tool? What safeguards should be required? How do we know whether it continues to perform safely? What constitutes an AI-related patient safety event? How do we investigate one when it occurs?
There is an even bigger question: How will we recognize risks that may not be visible within any single organization?
The Risks We See Together May Be Different From the Risks We See Alone
Imagine that one hospital identifies an unusual event involving an AI-supported workflow. Viewed in isolation, it may appear to be a one-time occurrence; perhaps a workflow problem, an individual error, or an anomaly that does not rise to the level of a broader concern.
But what if five hospitals are seeing it? Or 20?
What looks insignificant within one organization may, when viewed across organizations, reveal an emerging patient safety signal that none could have recognized independently. This is one of the most important reasons healthcare organizations must learn together.
Patient safety has long benefited from looking beyond individual events to identify patterns. AI makes that collective perspective even more important. Technologies may be deployed across many organizations, embedded in different workflows, and used with different populations. No single organization will necessarily have enough experience, or enough events, to recognize every risk early.
If we only learn within our own walls, we may miss the larger picture. Worse, one organization may experience a safety problem that others have already encountered without knowing it.
Safe AI Should Not Depend on an Organization’s Resources
Some large healthcare systems have informaticists, data scientists, human factors specialists, technology experts, clinicians, attorneys, and patient safety professionals helping them navigate AI. However, many organizations do not.
Smaller hospitals, rural and geographically isolated organizations, long-term care providers, and other resource-constrained healthcare organizations may be evaluating many of the same technologies without comparable expertise.
Patient safety cannot depend on the size, geography, or resources of the organization adopting AI. Nor should every healthcare organization have to independently create solutions to the same emerging challenges.
That is why the Mid-Atlantic Patient Safety Center and the MedStar Health National Center for Human Factors in Healthcare launched the Consortium for Safe AI in Healthcare. And it’s not too late to join us!
Learning Together and Giving Organizations a Better Starting Point
The Consortium is not another forum simply to talk about AI. It brings healthcare organizations and experts together to address common patient safety challenges and develop practical resources that organizations can use and adapt to their own environments.
Initially, participants are focusing on three priorities:
Establishing practical best practices and safeguards for evaluating, implementing, and monitoring AI.
Developing approaches for identifying, reporting, investigating, and learning from AI-related patient safety events.
Exploring how AI itself can help patient safety professionals identify patterns and emerging risks across safety reports and other information.
There will not be one approach that works identically in every healthcare organization. Technology, workflows, governance, resources, and patient populations differ.
Every organization should not have to begin with a blank page. The Consortium allows experts and healthcare organizations to work through common questions together, develop approaches and resources others can adapt, and critically, to learn across organizational boundaries.
This is where MPSC can play a role no individual healthcare organization can easily play alone: convening organizations around patient safety, identifying common and emerging risks, bringing different expertise to the same table, and translating collective learning into tools and guidance others can use.
Rather than simply reducing duplication, organizations will be equipped to see what they cannot see alone.
AI has extraordinary potential to improve healthcare. Realizing that potential safely will require the same principle that has always been fundamental to patient safety: we make greater progress when we learn together.
Every organization will need to make AI safe within its own environment. But none of us should have to discover the risks, or figure out how to address them, alone. Join the Consortium today!
Reference
Frank C. Why a ‘human-in-the-loop’ is not an adequate safety strategy: the case for vendor accountability in clinical AI safety. BMJ Quality & Safety Published Online First: 31 July 2026. doi: 10.1136/bmjqs-2026-020713