BACKGROUND: As bedside ultrasound (BUS) is being increasingly taught and incorporated into emergency medicine practice, measurement of BUS competency is becoming more important. The commonly adopted experiential approach to BUS competency has never been validated on a large scale, and has some limitations by design.

OBJECTIVE: Our aim was to introduce and report preliminary testing of a novel emergency BUS image rating scale (URS).

METHODS: Gallbladder BUS was selected as the test case. Twenty anonymous BUS image sets (still images and clips) were forwarded electronically to 16 reviewers (13 attendings, 3 fellows) at six training sites across the United States. Each reviewer rated the BUS sets using the pilot URS that consisted of three components, with numerical values assigned to each of the following aspects: Landmarks, Image Quality, and Annotations. Reviewers also decided whether or not each BUS set would be "Clinically Useful." Kendall taus were calculated as a measure of concordance among the reviewers.

RESULTS: Among the 13 attendings, image review experience ranged from 2-15 years, 5-300 scans per week, and averaged 7.8 years and 60 images. Kendall taus for each aspect of the URS were: Landmarks: 0.55; Image Quality: 0.57; Annotation: 0.26; Total Score: 0.63, and Clinical Usefulness: 0.45. All URS elements correlated significantly with clinical usefulness (p < 0.001). The correlation coefficient between each attending reviewer and the entire group ranged from 0.48-0.69, and was independent of image review experience beyond fellowship training.

CONCLUSION: Our novel URS had moderate-to-good inter-rater agreement in this pilot study. Based on these results, the URS will be modified for use in future investigations.