SummaryThis paper describes a study in which students were faced with a series of problems presented as patient management problems and as simulated patients (individuals trained to accurately portray a clinical problem). The subjects were sixty‐five final‐year medical students in a clinical clerkship in family medicine. Four clinical problems were used—each was developed in the PMP and simulated patient format. Each student completed one PMP and one simulated patient encounter (SPE) during the 2nd week of the 8‐week clerkship, and a second PMP and SPE in the 7th week of the clerkship. Performance on the two formats was compared by determining the number of options, and the number of critical options (weighted +1 or +2 by a criterion panel) elicited in each section of the problem—history, physical examination, investigations and management. Students were found to elicit significantly more options in the PMP in all sections of the problem, an increase of from 20 to 150%. This difference due to format was of similar magnitude to the difference between problems and the proportion of observed variance in response due to the differences between formats and cases was consistently greater than the variance due to systematic differences between students. The findings of this study are consistent with previous studies comparing performance on PMPs to oral examinations and medical records, and raise some concern about the use of PMPs as a measure of competence in certification and licensure