A counterbalanced experimental design was used to investigate the construct validity of four 10‐point scales of oral proficiency aimed at grammar, vocabulary, pronunciation, and fluency. Interviews with 10 foreign students were tape recorded and evaluated by 15 native speakers of English. Under condition 1, on four of five consecutive hearings, the raters evaluated the performances on only one of the four scales. Under condition 2, on the remaining occasion, they rated each interview on all four scales at one hearing. Construct validity was assessed by contrasting the amount of agreement across raters across scales with the amount of agreement within scales. A single general factor accounted for 96% to 113% of the estimated reliable variance in each scale. There was great fluctuation in specificity estimates which by the least biased method averaged between ‐12% and +3%. This result seems to indicate that there is no unique reliable variance (i.e., specificity) which can be attributed to the separate constructs of grammar, vocabulary, pronunciation, and fluency. The contrast between agreement and indexes for scales rated on the same occasion versus scales rated on different occasions revealed a halo effect which tends to reduce the reliable variance in scales rated at a single hearing by about 3%. Agreement across raters within and across scales tended to increase with repeated listenings. The global proficiency factor is sustained and no consistent specificity is found in the four scales stud