This paper is a sequel to a study which showed that the dominant dimension for perceptual discrimination among normal voices was the male–female categorization and which also suggested that discrimination within the male–female categories utilized distinct dimensions. The present study eliminates the male–female axis by treating the gender groups separately and making the within‐category dimensions available for more sensitive analysis. The purpose was to determine the number and nature of perceptual parameters needed to explain judgments of voice similarity depending on talker sex and whether the stimulus sample was a sustained vowel or a short phrase. The similarity judgments were submitted to multidimensional analysis via INDSCAL and the resulting dimensions were interpreted in terms of available acoustic measures and unidimensional voice‐quality ratings of pitch, breathiness, hoarseness, nasality, and effort. The decisions of the listeners appeared to be influenced by both the sex of the speaker and the stimulus sample, although fundamental frequency (f0), was important for all judgments. Aside from thef0dimensions, judgments concerning male voices were related to vocal tract parameters, while similarity judgments of female voices were related to perceived glottal/vocal tract differences. Formant structure was apparently important in judging the similarity of vowels for both sexes while perceptual glottal/temporal attributes may have been used as cues in the judgments of phrases.