Accent Detection and Speech Recognition for Shanghai-Accented MandarinYanli Zheng⊕, Richard Sproat⊕, Liang Gu†, Izhak Shafran‡,Haolang Zhou‡, Yi Su‡, Dan Jurafsky, Rebecca Starr, Su-Youn Yoon⊕University ofIllinois⊕, IBM T. J. Watson Research Center†Johns Hopkins University‡, Stanford University{zheng3, rws, syoon9}@uiuc. edu⊕, lianggu@us. ibm. com†,{zakshafran, haolangzhou, suy}@jhu. edu‡, jurafsky@stanford. edu, starr@sundell. netAbstractAs speech recognition systems are used in ever more applica-tions, it is crucial for the systems to be able to deal with ac-cented speakers. Various techniques, such as acoustic modeladaptation and pronunciation adaptation, have been reported toimprove the recognition of non-native or accented speech. Inthis paper, we propose a new approach that combines accent de-tection, accent discriminative acoustic features, acoustic adap-tation and model selection foraccented Chinese speech recogni-tion. Experimental results show that this approach can improvethe recognition of accented speech.1. IntroductionAccent is by far the most critical issue for the state-of-the-artChinese automatic speech recognition (ASR) systems. This isbecause Chinese is a language with so many dialects includingMandarin, Wu (spoken by Shanghainese), Yue (spoken by Can-tonese), Min (spoken by Taiwanese), etc. Although the officialspoken language is Putonghua (also called Standard Mandarinor Mandarin in the speech recognition literature), it is spokenextremely differently by speakers living in different dialectalregions of China. As a result, current ASR systems trained onPutonghua or Standard Mandarin often experience a dramaticaccuracy loss for speakers with strong accents.Active research has been carried out on dialectal or for-eign accented speech recognition during the past few years.The proposed methods vary from simply collecting data in thataccent and training a recognizer, to various ways of adapt-ing recognizers trained on unaccented speech. Wang, Schultz,and Waibel [1] investigated German-accented English speakersin the VERBMOBIL (conversational meeting planning) task.Tomokiyo and Waibel [2] examined the task of recognizingJapanese-accented English in the VERBMOBIL domain. Inboth tasks, it was found that training on non-native speech data,achieves the most obvious gains in performance on accenteddata. The simplest use of adaptation was merely the direct useof MLLR (Maximum Likelihood Linear Regression) to adaptindividually to each test speaker. In [3], in order to recognizeShanghainese-accented Putonghua, Huang et al. applied stan-dard speaker MLLR adaptation to a Microsoft Whisper systemthat had been trained on 100,000 sentences from 500 speakersliving in the Beijing area. In [1, 2], MLLR was adapted not justto the single accented test speaker, but to a larger number ofac-cented speakers. Research in [1, 2, 3] shows the effectivenessof MLLR or MAP (Maximum A Posteriori) adaptation on ac-cented speech. but it did not report whether combining MLLRand MAP could be helpful for accented ASR.While some promising results have been published on ac-cented speech recognition using the above approaches, therecognition accuracy on accented speech is still low and defi-nitely needs further improvement. In particular, some researchissues remain open. First, more sophisticated forms of MLLRor MAP may be applied, such as MLLR using phone-specifictransforms rather than a single global transform. Furthermore,our research shows that current adaptation schemes have variedperformance on different groups ofspeakers. Second, the effectof combining MLLR and MAP in accented ASR needs to beexplored. Third, the accent ofeach speaker should be treated asa matter of degree. Previous work on accented Chinese speechrecognition [3, 4, 5, 6] typically treats speakers from a given di-alectal region as a single class. In reality, these speakers clearlyhave different degrees of accent.In this work we optimized the MAP/MLLR combinationfor our task. Then, building on this optimal MAP/MLLR com-bination, we developed new approaches to detecting and utiliz-ing degree of accent in accented ASR. A series of new algo-rithms is proposed: phoneme-based automatic accent detection,formant-augmented acoustic features for accented speech, andaccent-based model selection during acoustic model decoding.For the sake of simplifying our experiments we focus hereonly on one form of accented Putonghua, namely the accentof people of Shanghai whose native language is Shanghainese,which belongs to the Wu dialect group, a group with 87 millionspeakers. Thus all experiments in this paper were performed onWu-accented conversational Chinese speech. Nevertheless, webelieve that our proposed approaches will also be helpful for ac-cented speech with other Chinese dialects or in other languages.2. Data Collection and TranscriptionIn this paper, we use spontaneous speech data collected from50 male and 50 female speakers of Wu-accented Standard Chi-nese, henceforth Putonghua. The spontaneous speech consistedof free-form monologues where the speaker was asked to dis-cuss a topic of their choice from a small set of predeterminedtopics. It was recorded at 16KHz using a head-mounted micro-phone. The data were orthographically transcribed, and phonet-ically transcribed into syllable initials (onsets) and finals (nu-cleus+coda) — henceforth “IF” refers to initial-final phoneset.Canonical pinyin transliterations were also derived from the or-thographic transcription. Finally, speakers were classified byexperts into their “Putonghua level” on a 6-point scale rang-ing from 1A (most standard) to 3B (least standard); all of ourspeakers fell in the range 2A–3B. Ofthe 100 speakers, 80 wereused fortraining data and the remaining 20 for test data. Furtherdetails on the data can be found in [7].
