The security of frequency domain analogue speech scramblers is investigated. It is shown that a vector codebook similar to that deployed in vector quantisation of speech can be used to attack the speech scramblers even under the stringent condition that no section of the original speech is available to the attacker and that the encryption key of the system is varied frequently in a random and unknown manner. Subjective as well as objective results demonstrating the success of the attack is given. A technique for improving the security of the frequency domain encryption scheme under the proposed attack is discussed.