Event recap: «Shape responsible clinical AI»
The DSI recently brought together clinicians, researchers, and industry experts to examine why AI tools in healthcare so often fall short of their clinical potential. The panel unpacked what it actually takes for AI to work in medical practice – from trust and accountability to workflow fit, local context, and the question of who bears responsibility when things go wrong.
On May 8th, the DSI brought together experts in the use of AI in healthcare, to explore why AI tools in healthcare often struggle to move from technical promise to practical impact. Dr. Tariq Andersen (Associate Professor at the University of Copenhagen), Dr. Afua van Haasteren (Director of Health Policy and External Affairs at Roche Diagnostics), Dr. David Sasu (Technology Director at LIGHT Lab and EPFL SwissAI Fellow), Dr. Daphné Chopard (AI Research Lead in ICU Research Group at University Children’s Hospital Zurich) and Dr. Sintieh Ekongefeyin (Physician, Scientist and DSI Excellence Program Fellow at the University of Zurich) joined as panellists, and Gabriela Morgenshtern (Doctoral Researcher at the University of Zurich) and Dr. Andrea Farnham (Digital Epidemiologist at the University of Zurich) joined as moderators.
A recurring theme was that AI development too often begins with the technology rather than the clinical problem. Speakers challenged the assumption that a sophisticated tool is automatically useful, arguing that healthcare AI must be judged by whether it solves a real problem, fits into clinical workflows, and improves outcomes for patients and professionals.
Trust emerged as one of the central issues. In healthcare, even a high-performing model can be difficult to accept if clinicians cannot understand or rely on it. An accuracy rate that may look impressive from an engineering perspective can still be unacceptable in clinical practice, where the remaining margin of error may involve patient harm, liability, and professional responsibility. The discussion also emphasized that doctors are not simply worried about being replaced by AI; they are concerned about accountability when AI-supported decisions go wrong.
Another major point was context. The same AI tool may be helpful in one setting and disruptive in another. A feature that supports clinicians in a small or under-resourced hospital may confuse specialists in a large university hospital. Similarly, tools designed without attention to infrastructure, internet access, local resources, or patient realities may fail despite strong technical design. One example discussed was a health advice tool that recommended foods unavailable in the communities it was meant to serve, showing that usefulness depends on environment as much as sophistication.
The panel also addressed the limits of data and generalizability. AI models trained in one hospital, population, or age group may not work elsewhere. Clinical practices differ across institutions, regions, and patient groups, meaning that models can reproduce local assumptions rather than universal medical knowledge. This raised broader questions about bias, cross-border data use, and the need for regulatory frameworks that both protect patients and allow responsible innovation.
A final thread was the need for collaboration between developers, clinicians, institutions, and patients. The most promising approach was not to design AI in isolation and then impose it on healthcare settings, but to involve clinicians early, test tools in real workflows, and adapt them to local needs. Rather than replacing medical expertise, the panel framed AI as a form of augmentation: potentially useful when it helps clinicians process information, work more efficiently, and spend more time with patients, but only if it is trustworthy, context-sensitive, and properly integrated into the healthcare system.
A big thank-you to the expert panellists and moderators for their valuable contribution, as well as the DSI Communities Health, Ethics and AI & Law for organising this event. For further information on the CITL project (Clinician-in-the-Loop AI & Clinical Uncertainty), view the project website here.