This paper presents a semi‐automatic phonetic labeling method for processing in the MAT (Mandarin across Taiwan) speech database. MAT speech data are collected through the telephone networks. Each utterance has been transcribed into Chinese characters and Pinyin symbols. The proposed phonetic labeling method will mark the syllable and sub‐syllable boundaries in an utterance. Phonetic symbols are assigned to each segmented syllable. The segmentation process is accomplished by using hidden Markov modeling (HMM) and Viterbi decoding. The accuracy of syllable segmentation is detected by measuring the syllable length and the distance of a syllable from its state models. The experimental results show that the proposed labeling method can achieve segmentation accuracy around 90% for an allowed tolerance of 16 ms.