The quality of radiology reports produced by residents can be quantitatively assessed, according to research published September 24 in Radiology.
After retrospectively using the Jaccard index to compare nearly 250,000 radiology reports, the researchers found that the method quantitatively demonstrated an improvement in resident performance as they gained experience over the course of their training. As a result, the Jaccard similarity coefficient could serve as a proxy for report quality improvement over time, according to the group led by Jan Vosshenrich, MD, of the University of Basel in Switzerland.
“Data could potentially be used to establish performance benchmarks and better comprehend the developmental trajectories of residents, although variability among subspecialties and imaging modalities suggests that one size does not fit all,” the authors wrote.
Of the 246,984 radiology reports dictated by 53 residents at their institution between September 2017 and November 2022, 22% were dictated by first-year residents, 24% were dictated by second-year residents, 19.1% by third-year residents, 15.9% by fourth-year residents, and 19% by fifth-year residents.
Over the course of the residency, the Jaccard similarity coefficient showed a positive correlation between years of training and report quality, increasing by 14% for draft reports and 6% for preliminary reports. These improvements were statistically significant (p < 0.001). Improvements were most evident in musculoskeletal imaging.
What’s more, the standard deviations decreased by 4% for draft reports and by 7% with years of training.
“Future studies should aim at refining the tools for quantitative assessment of report quality to provide a more nuanced and clinically relevant understanding of the skills residents acquire throughout their education,” the authors wrote.
In an accompanying editorial, Michael Bruno, MD, of Penn State Milton S. Hershey Medical Center and Penn State College of Medicine noted that the research presents a major step forward and provides a gateway to further research, such as for evaluating a variety of interventions designed to improve trainee report writing.
“The effectiveness of both currently used and novel strategies to improve trainee report writing skills can now be objectively evaluated, longitudinally, across a large cohort of trainees with use of the new report comparison tool,” Bruno wrote. “This also encompasses whether large language models, such as OpenAI’s ChatGPT, could be used to improve report writing by residents, including automated generation of the impression section of a radiology report and creating layperson versions of reports for the benefit of patients.”
He also suggested that perhaps a higher standard should be used to assess the quality of the reports.
“In other words, perhaps those of us who teach radiology residents ought to aspire to help our trainees to become even better report writers than we currently are so that the specialty advances and future patients are better served,” the authors wrote. “Why should we set our goal to merely make our trainees reach our own level, when there now exist tools that can be used to raise the quality bar even higher?”
The full article can be found here and the accompanying editorial can be found here.
