OBJECTIVE: Little is known about the best approaches and format for measuring physicians' communication skills in an online environment. This study examines the reliability and validity of scores from two Web-based communication skill assessment formats.

METHODS: We created two online communication skill assessment formats: (a) MCQ (multiple-choice questions) consisting of video-based multiple-choice questions; (b) multi-format including video-based multiple-choice questions with rationales, Likert-type scales, and free text responses of what physicians would say to a patient. We randomized 100 general internists to each test format. Peer and patient ratings collected via the American Board of Internal Medicine (ABIM) served as validity sources.

RESULTS: Seventy-seven internists completed the tests (MCQ: 38; multi-format: 39). The adjusted reliability was 0.74 for both formats. Excellent communicators, as based on their peer and patient ratings, performed slightly better on both tests than adequate communicators, though this difference was not statistically significant. Physicians in both groups rated test format innovative (4.2 out of 5.0).

CONCLUSION: The acceptable reliability and participants' overall positive experiences point to the value of ongoing research into rigorous Web-based communication skills assessment.

PRACTICE IMPLICATIONS: With efficient and reliable scoring, the Web offers an important way to measure and potentially enhance physicians' communication skills.