Vision-language models answer questions about medical images in fluent, confident language. That confidence is the problem: a model sounds just as certain whether or not its answer rests on visual evidence it has actually seen and can handle. In medicine, a confident answer about an image the model does not understand is worse than no answer at all. This project asks how to make the answers of multimodal LLMs traceable to the evidence they were given. Four questions guide it:

1. Where inside a vision-language model does visual evidence enter the answer, and where does it get lost?
2. How can grounding in the image be interleaved with a model's free-text reasoning, so each reasoning step points to the evidence behind it?
3. Can a model tell, at test time and without labels, that an input lies outside what it can reason about?
4. Can a deployed model adapt at test time to new visual features and previously unseen labels, without being taken out of service?

We are building towards the first co-working space for medical multimodal imaging, one place to analyse and automate more than 15 tasks across medical imaging and analysis, built on models whose answers can be traced to their evidence.

Collaborations

Prof. Pranav Rajpurkar

Associate Professor of Biomedical Informatics, Harvard Medical School

Rajpurkar Lab, Department of Biomedical Informatics: medical AI for complex clinical work

Prof. Zongwei Zhou

Assistant Professor of Computer Science, Johns Hopkins University

Joint appointment in Oncology, Johns Hopkins School of Medicine