The laboratory ranking test is a nonparametric procedure that W. J. Youden recommended in 1963 for identifying outlying laboratories in collaborative studies. Though potential problems with the performance of this test have been noted by Youden and others, its use still is being promoted. One problem with the test is that its critical values were developed for a different application and do not allow for the between-laboratory variation ordinarily observed in collaborative studies. This paper presents a quantitative evaluation of the impact of between-laboratory variation and other factors on the performance of the laboratory ranking test. The results document some of the test's shortcomings, including its propensity to identify unexceptional laboratories as outliers.