BACKGROUND: Multisource feedback can provide a comprehensive picture of a medical trainee's performance. The utility of a multisource feedback system could be undermined by lack of direct observation and accurate knowledge.

METHOD: The National Board of Medical Examiners conducted a national survey of medical students, interns, residents, chief residents, and fellows to learn the extent to which certain behaviors were observed, to examine beliefs about knowledge of each other's performance, and to assess feedback.

RESULTS: Increased direct observation is associated with the perception of more accurate knowledge, which is associated with increased feedback. Some evaluators provide feedback in the absence of accurate knowledge of a trainee's performance, and others who have accurate knowledge miss opportunities for feedback.

CONCLUSIONS: Direct observation is a key component of an effective multisource feedback system. Medical educators and residency directors may be well advised to establish explicit criteria specifying a minimum number of observations for evaluations.