SummarySkills in reviewing medical charts have been demonstrated components of clinical competence related to knowledge base, level of clinical experience, and basic observational skills. A study of the generalizability of performance on chart review exercises, which controlled for knowledge in the content area, was undertaken to determine their potential in evaluating students' problem‐solving ability. Results of the study indicate that the case specificity which has characterized simulated problem‐solving tasks is largely case, rather than content, specificity: correlations between scores on single charts demonstrated no consistent relationships for measures of proficiency, efficiency, and diagnostic accuracy. However, averaging the scores on two charts and then computing correlations resulted in significant positive measures for both proficiency and efficiency. The effects of length and difficulty level on the generalizability of tests of problem‐solving ability are areas suggested for future res