SummaryIn a previous study we described a problem‐based criterion‐referenced test of the clinical competence of medical students which was felt to offer advantages over the traditional final‐year examination. This paper reports the validity and reliability studies on which it is possible to judge the value of this new test when compared to the traditional approach.The results demonstrate a high level of content validity and provide evidence of the construct validity of the test. Efforts to obtain measures of concurrent and predictive validity were thwarted by a failure to attain reliable assessments of ward performance from resident and consultant staff. Satisfactory levels of internal consistency were established for the whole test. Marker reliability was satisfactory in all sections of the test except for those requiring examiners to rate practical clinical skills. This was so despite the use of simulated patients, behavioural check‐lists and rater training. Possible solutions to this problem are discussed.It is concluded that this new approach overcomes many of the measurement problems inherent in the traditional final examination. It has been shown to be feasible to construct and administer in the medical school setting without the need for the allocation of additional re