CONSTRUCT: This article describes the development and implementation of an assessment intended to provide objective scores that would be valid indications of medical students' abilities to counsel patients about tobacco dependence.

BACKGROUND: Assessing medical students' advanced communication skills, particularly in the context of providing tobacco-dependence treatment, consistently and accurately is challenging; doing so across multiple medical schools is even more difficult.

APPROACH: Ten medical schools implemented a tobacco-dependence treatment case as part of an Objective Structured Clinical Examination for 3rd-year medical students. A 33-item checklist with detailed criteria and examples was developed for scoring students' performances. Trained coders viewed and coded 660 videotaped encounters; approximately 10% also were coded by the coding supervisor to check accuracy.

RESULTS: Average time required to code an encounter was approximately 31 minutes; accuracy (i.e., agreement with the gold standard coder) was excellent. Overall, students performed an average of 1 in 4 of the 33 behaviors included on the checklist, and only 1 in 10 discussed setting a quit date. Most students (almost 9 in 10) asked how much the patient smoked in a day, and just over 7 in 10 informed the patient that the cough was due to smoking.

CONCLUSIONS: The authors developed and implemented a rigorous assessment that will be used to evaluate medical students' tobacco-dependence treatment skills. Operationalizing the specific counseling behaviors, training coders to accurately capture students' performances using a structured checklist, and conducting the coding all required substantial time commitments but will provide confidence in the objectivity of the assessment results. In addition, this assessment can be used to provide formative information on medical students' tobacco-dependence treatment skills and to tailor ongoing training for medical students in this area.