2.5 million encounters documented by AI scribes in a single year — and curricula that don't teach note review

Ambient AI scribes, which listen to consultations and produce draft notes on their own, are among the fastest-spreading AI tools in health care. Now researchers at Carnegie Mellon University and the University of Pittsburgh School of…

File photo: Scaife Hall, University of Pittsburgh
File photo: Piotrus, CC BY-SA 3.0 · source This adaptation is licensed under CC BY-SA 3.0.

2.5 million encounters documented by AI scribes in a single year — and curricula that don't teach note review

Ambient AI scribes, which listen to consultations and produce draft notes on their own, are among the fastest-spreading AI tools in health care. Now researchers at Carnegie Mellon University and the University of Pittsburgh School of Medicine are proposing something medical schools have barely confronted: that reviewing AI-generated notes should be treated as a distinct, teachable competency — with four defined domains and simulation as the teaching method.

What's new

In an article in the journal npj Digital Medicine, the research team, led by Yash Raka, presents a framework titled "From authoring to editing." The point, put simply: where physicians in training have traditionally learned to write clinical notes, they must now learn to evaluate notes that an AI has written for them. The researchers define this review as a competency that can be taught and practiced — not something clinicians simply already know how to do.

The framework was developed jointly by Heinz College at Carnegie Mellon University and the School of Medicine at the University of Pittsburgh.

Why now

The background is that adoption has run well ahead of training. Ambient AI scribes — tools that passively capture the clinical consultation and produce a draft note — are described in the reporting as among the fastest-adopted AI applications in health care. One integrated health system, according to the source material, reported that the tools documented more than 2.5 million clinical consultations in a single year. The figure is attributed to one, unnamed system and has not been independently verified.

It is against this picture that the authors direct their argument: the tools are in widespread use, but according to them, nothing in today's medical curricula targets directly the skills required to review AI documentation competently. The early benefits AI scribes have shown — related to documentation time, cognitive workload and burnout — are described in the source material in general terms, without specific studies or figures.

The four domains

The article identifies four domains involved in reviewing AI-generated clinical documentation:

  1. Detecting errors and omissions. The note may contain factual errors or omit information that was present in the consultation.
  2. Judging whether the note reflects clinical reasoning. A note can be superficially correct yet still miss how the clinician actually thought and prioritized.
  3. Generating notes independently. Candidates must still master writing notes without AI — partly in order to be able to evaluate the machine's drafts.
  4. Calibrating trust across contexts. How much can you rely on the tool in different clinical situations, and when should distrust be turned up?

Together, the domains describe a shift in what kind of skill note work actually is. The authors are explicit about their own position: the editing skills required to review AI documentation are specific and teachable — not innate extensions of clinical practice. And that, they argue, is precisely what today's curricula fail to capture.

Simulation as the method

As for how the skills should be learned, the authors bet on simulation. Students are to practice review in safe settings, with the possibility of targeted feedback on what they miss.

"Simulation lets trainees practice catching an AI's mistakes in a low-stakes way where it won't affect patient outcomes," says Reed Van Deusen, associate professor of general internal medicine and pediatrics at the University of Pittsburgh School of Medicine, who is a co-author of the article.

The reasoning is the classic argument for simulation in medicine, transplanted to new content: the errors an inexperienced trainee overlooks in an AI note can, in practice, become the patient's problem. In a simulated consultation, the same errors can become teaching material instead.

A preliminary, untested framework

There are no measured results yet. The authors themselves describe the framework as preliminary and call for empirical testing to develop it further. That means the benefits of teaching according to the model — such as better error detection among trainees — are proposed, not demonstrated. The framework is a response to an identified gap in training, not a documented solution to it.

It should also be said how the reporting stands: all details about the framework, the adoption figures and the quotations come via Medical Xpress's coverage of the article, based on a release from Carnegie Mellon's Heinz College. The underlying peer-reviewed article was not directly available for this coverage, and the details of the four domains are therefore reproduced as they appear in the secondary source.

What needs to come next

The authors themselves point to the way forward: the framework must be tested empirically and adjusted. Open questions follow from that. Are trainees trained in review actually better at catching errors and omissions? How do you calibrate trust in practice, when contexts range from routine check-ups to complex courses of illness? And does the balance between independent note writing and editing hold as the tools get even better?

Until such studies exist, the proposal stands as a diagnosis of a training need rather than a proven cure: adoption has already happened, the million-scale figures are documented at one health system, and the question the authors raise is whether medical education will keep up — or let a generation of clinicians learn to edit AI notes without a curriculum.

(Medical Xpress, source ID: e02029b3-b70d-47e8-9aa0-9fabf86a0331)

AIMag.no
AIMag.no
The AIMag.no editorial team covers artificial intelligence, tools, research, and regulation.

Get the best of AI MAG in your inbox

News, analysis, and ideas at the intersection of AI and society.