Learning to Boost GMM Based Speaker Verication

Stan Z. Li, Dong Zhang, Chengyuan Ma, Heung-Yeung Shum, and Eric Chang Microsoft Research China, Beijing Sigma Center, Beijing 100080, China
szli@microsoft.com, http://research.microsoft.com/ szli

Abstract
The Gaussian mixture models (GMM) has proved to be an effective probabilistic model for speaker verication, and has been widely used in most of state-of-the-art systems. In this paper, we introduce a new method for the task: that using AdaBoost learning based on the GMM. The motivation is the following: While a GMM linearly combines a number of Gaussian models according to a set of mixing weights, we believe that there exists a better means of combining individual Gaussian mixture models. The proposed AdaBoost-GMM method is non-parametric in which a selected set of weak classiers, each constructed based on a single Gaussian model, is optimally combined to form a strong classier, the optimality being in the sense of maximum margin. Experiments show that the boosted GMM classier yields 10.81% relative reduction in equal error rate for the same handsets and 11.24% for different handsets, a signicant improvement over the baseline adapted GMM system.

1. Introduction
The speaker verication task is essentially a hypothesis testing problem or that of a binary classication between the targetspeaker model and impostor model. The Gaussian mixture models (GMM) method [8] has proved to be an effective probabilistic model for speaker verication, and has been widely used in most of state-of-the-art systems [8]. Each speaker is characterized by a GMM, and the classication is performed based on the log likelihood ratio (LLR) of the two classes [8]. A GMM as aims to approximate a complex nonlinear distribution using a mixture of simple Gaussian models, each parameterized by its mean vector, covariance matrix and the mixing parameter. These parameters are learned by using, e.g. an EM algorithm. In adapted GMM based speaker verication systems, a GMM is learned for the imposter class, and the target-speaker model is approximated by adaptation from the imposter GMM, i.e. modication of the imposter GMM model. An complete adaptation should be done on the mean vector, covariance matrix and the mixing parameter; but it is usually performed for the mean vector only since it is empirically observed that adaptation also on covariance matrix and the mixing parameter would yield less favorable results. As such, the GMM modeling, i.e. the estimation of the parameters, and especially the adaptation for the target-speaker model are not optimal. This motivated us to investigate into a method which would rely less on the estimated parameters and could rectify inaccuracies therein. AdaBoost methods, introduced by Freund and Schapire [2], provides a simple yet effective stagewise learning approach: It learns a sequence of more easily learnable weak classiers, each of them needing only slightly better than random guessing, and boosts them into a single strong classier by a linear combi-

nation of them. The weak classiers, each derived based on some simple, coarse estimates, need not to be optimal. Yet, the AdaBoost learning procedure provides an optimal approach for combining them into the strong classier. Originating from the PAC (probably approximately correct) learning theory [11, 5], AdaBoost provably achieves arbitrarily good bounds on its training and generalization errors [2, 10] provided that weak classiers can perform slightly better than random guessing on every distribution over the training set. It is also shown that such simple weak classiers, when boosted, can capture complex decision boundaries [1]. Relationships of AdaBoost to functional optimization and statistical estimation are established recently. It is shown that the AdaBoost learning procedure minimizes an upper error bound which is an exponential function of the margin on the training set [9]. Several gradient boosting algorithms are proposed [3, 6, 12], which provides new insights into AdaBoost learning. A signicant advance is made by Friedman et al. [4]. It is shown that the AdaBoost algorithms can be interpreted as stagewise estimation procedures that t an additive logistical regression model. Both the discrete AdaBoost [2] and the real version [10] optimize an exponential loss function, albeit in different ways. The work [4] links AdaBoost, which was advocated from the machine learning viewpoint, to the statistical theory. In this paper, we propose a new method for the speaker verication: that using AdaBoost learning based on the GMM. We start with an imposter GMM and a target-speaker GMM, more specically, their mean vectors and covariance matrices but not the mixing weights. Although the GMM models are inaccurate, we are able to construct a sequence of weak classiers based on these GMMs, weak classiers meaning slightly better than random guessing. The AdaBoost procedure (1) sequentially and adaptively adjusts weights associated with the training examples which helps to construct and select the next good weak classiers, and (2) combine them sensibly to constitute a boosted strong classier. The combination is optimality in the sense of maximum margin. Experiments show that the boosted GMM classier yields 10.81% relative reduction in equal error rate for the same handsets and 11.24% for different handsets, a signicant improvement over the baseline adapted GMM system.

2. GMM Representation of Speaker Voices

In speaker verication, the task is to verify the target speaker and to reject imposter speakers. Two GMM are built from training data, the universal background model (UBM) for the imposter class and the target speaker model. This section describes the GMM modeling, on which most of the state-of-the-art systems are based, as the starting point of the proposed method. Mel-frequency cepstral coefcients (MFCCs) are used as acoustic features for signals of speaker voices. All utterances

are pre-emphasized with a factor of 0.97. A Hamming window with 32ms window length and 16ms window shift is used for each frame. Each feature frame consists of 10 MFCC coefcients and 10 delta MFCC coefcients. Finally, the relative spectral (RASTA) lter and cepstral mean subtraction (CMS) are used to remove linear channel convolutional effects on the cepstral features. Therefore, each window of signal frames is represented by a 20-dimensional feature vector, which we call a feature frame. A sequence of feature frames, denoted    , are observed from the utterance of the same speaker. Assuming the feature vectors are independent, the probability (likelihood) of the observation can be obtained as follow: !$)()0 21 % !#" !$&% ' " (1) 6 7   " $% 3" $4 5 % (2) 8 @9
3" 4 5 % $4 5 % A 3" C8 B

observation , the LLR is dened as 3" 4 5 wx!% yy " % (10) P U" 4 5WV!X`Y %  7 $24 5 V!X#Y %f U" $4 5gwx % P 3" $ 8 P s Because the corresponding means and covariances have been estimated, the LLR can be computed analytically. The decision function is r yy " %R " % (11) p f tIf (12) sp dee fge The threshold can be adjusted to balance between the accuracy and false alarm rates (i.e. to choose a point on the ROC curve).

3. AdaBoost Learning
(3) The basic form of AdaBoost [2] is for two class prob lems. A set of labelled training examples  is given as " h %ijjjk "  h  % , where h $ml r is the class !$ lonWp p sp . AdaBoost assumes thata label for the example " % procedure available for learning a weak classiers ituis jj jEv deq (rs ) from the training examples, with respect p to a distribution of the examples determined by the associated fw$ weight . It aims to learn a stronger classier as a linear comv bination of the weak classiers x 7 " % x " % q (13) 8 qydeq q % The classication of is obtained as the sign of x " . It may be advantageous to use a normalized version of the above: x " % 8 q " % {z e d q (14) q x 8 q| q q z q %f # } ~ P " T Now, the classier function is and the normalized 4 " % 4 condence score is . The learning procedure, described  % " . in Fig.1, is aimed to derive q and q d q In the simplest case, the error is to be minimized. An error " !% h , or h x " %w . The margin of an occurs when U ) ` h % " !% ln on the training set examexample " achieved h " % by d ples is dened as . This can be considered as a measure of d the condence of the s prediction. The upper bound on clasd sication error achieved by x can be derived as the following exponential loss function [9] 7 h $ x " $ %f " x % (15) $ T` s x 7 h$ 7 " %& ` T q $ 8 qydeq s q " % by stagewise minimization of AdaBoost construct deq Eq.(16). A characteristics of AdaBoost is that during the learning, the AdaBoost re-weights each example, i.e. adaptively updates f$ qg according to the classication performance of the learned weak classiers (Step 2(3)). More difcult examples are associated with large weights so that more emphasis in the next round

" $ (0 21 % with Here, 9 0 is the weight of Gaussian 5 ' 1 mixture D0 E1  and covariance matrix , and is mean 9 also used to represent the model parameters. 5 5 Using the EM algorithm, we can get a local optimum F for under the MLE criterion by maximizing the probabilistic density for all observation frames. 5 45 % GIH2PRQSGT U" (4) F

The universal background model (UBM) is obtained by the MLE training. The target speaker model may also be obtained in a similar way. However, to reduce computation and to improve performance when only a limited number of training utterances are available, some adaptation techniques were proposed, e.g. [8], in which MAP adaptation outperforms the other two, maximum likelihood linear regression (MLLR) adaptation and eigenvoices method. MAP adaptation derived target speaker model by adapting the parameters of the UBM using target speakers training data under the MAP criterion. Experiments show that when only the mean vectors of the UBM model are adapted, 5WV!X`Y we can get and training the best performance. Given a UBM model data from the hypothesized speaker, the adaptation can be done as follow,  d" e$f% a "cb 4 !$&% 9 (5) 3" !$ 4 5gV!X#Y %
h F i d" % 0 q 7 A C8 p hU 7 a "cb 4 $ %

(6)

a "cb 4 !$&%D!$ A C8 % r " i #"  q q p s g hU h rut

(7)
%f0

(8) (9)

t t where is 5g a w constant relevance factor xed at . The x pv parameters of the target speaker GMM is thus obtained. The classication for speaker verication may be performed based on the the log likelihood ratio LLR. Given the

0. (Input) " h %ijkjj "  h  %2 ,$ (1) Training examples ru h@ ; of which examples have where h $ and examples have ; (2) Termination condition sp v) (e.g. based on Recall/False Alarm, or ); 1. (Initialization) f $ h@$ r or D for those examples with f $ h$ p ) for those examples with . v sp ; 2. (Stagewise Learning) not while v satised vr (1) ; p (2) Learn x ;x h@$ x " !$&%f d f $ , and (3) Update T# $ f$ x s normalize to ; p z 3. (Output) u w " % { e .
r

Table 1: DCF, EER and relative reduction (SH: Same Handset. DH: Different Handset)
p

Method Adapted GMM Boosted GMM Rel. Reduction Method Adapted GMM Boosted GMM Rel. Reduction

DCF (SH) 0.0460 0.0395 14.13% EER (SH) 4.90% 4.37% 10.81%

DCF (DH) 0.1816 0.1729 4.79% EER (DH) 21.96% 19.49% 11.24%

4. Learning Weak Classiers

An algorithm is needed to learn weak classiers (Step 2.(2) in Fig.1). In this work, we dene a set of weak classi0 candidate E1 ers based on the model parameters ( ) provided by the two GMMs (but without using the mixing parameter 9 in this stage), and nd the v best one from this set in each stage. At iteration , we dene the set of base weak classier using the component LLRs 3" 45gwx f x % x " % x P U" 45 V!X`Y )f x % (20) d s (21) x where the is set to ensure a required accuracy. The best weak classier is the one for which the false alarm is minimized: x be GIH2PRQ~ " " %% (22) d x " % where (also w.r.t. is the false alarm caused by f xS d ). This gives us the best weak classier as x " % x " %  (23) d d

Figure 1: Discrete AdaBoost Algorithm.

of learning weak classier will be put on those examples. The weighted error is
q 7 $ f$ qg p h $ " $ %f

(16)

Minimizing the above is the objective of x .  % e d g q " % , and 8 q Given the current x " | q e d q z x q weak classier the newly according to the (learned f learned x d " )h`% ), the best combining weights coefcient q x for % xS " %r q x x " % is the new strong classier x " d the one which leads to the minimum cost: q x GIH2PQ~ " xS " %r q d x " %%

(17)

The minimizer is
q x a "h t p P a " h

5. Experiments
4 f xS % r f xS % p 4 s p

5.1. Data Set (18) All experiments are conducted on the 1996 NIST speaker recognition evaluation data set. Only male speakers are investigated. There are 21 male target speakers and 204 male impostors in the evaluation data set. The training utterances for each target speaker are extracted from two sessions originating from two different handsets, one minute per session. As for the testing, there are 321 target trials and 1060 impostor trials. The duration of each test utterance is about 30 seconds. 5.2. Results The system performance is evaluated using the detection error tradeoff (DET) curve, detection cost function (DCF) and equal error rate (EER). DCF can be dened as follow [7],   aU r   a3 (24) R  R  Here,  and  are false rejection rate and false acceptance rate respectively at a operating point. and aU a3 are costs for and false rejection and false acceptance. are # the j prior a probability of target trials and impostor trials. and dj a3 p .

h`% The weight for each " are then updated f x " Uh`% f x " U` h % " x h x " %% T` s q d

(19)

which is equivalent to Step 2.(3). The basic AdaBoost algorithm is for two-class problem. In t multi-class problems, where the class number , the nal decision could be obtained by combining multiple binary Adaboost classiers. There are two popular schemes for extending binary classiers to the multi-class case. One is the one-againstall strategy, in which classication is performed between each class and the remaining. This needs to train a total of boosted classiers. The other is the one-against-one strategy, in which classication is performed %t between each pair. The later strategy will require " classiers. In our system, we adopt sp the former scheme to save the computational cost. The output of the strong classiers are normalized as Eq.(14) to provide the score for comparison. The nal decision is made by maximizing the score.

EER and DCF for Boosted GMM and Adapted GMM under the same and the different handsets are shown in Table 1. DET curves for Adapted GMM and Boosted GMM for the same handset data are shown in Figure 2, and those for the different handsets are shown in Figure 3. From the results, we could conclude that boosted GMM yields a signicant improvement on relatively reduction of over 10% than the baseline adapted GMM approach. The improvement is due to the optimal combination of individual GMM learned by non-parametric AdaBoost method.
Speaker Detection Performance

adapted GMM system. The boosted GMM optimally combines weak classiers, i.e. component log likelihood ratio in GMM, to a strong classier. AdaBoost provides an effective framework for fusing different models. In future, it would be interesting to investigate how to use AdaBoost at low level feature (such as cepstral) level to learn better speaker models.

6. References
[1] L. Breiman. Arcing classiers. The Annals of Statistics, 26(3):801849, 1998. [2] Y. Freund and R. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1):119 139, Aug 1997. [3] J. Friedman. Greedy function approximation: A gradient boosting machine. The Annals of Statistics, 29(5), October 2001. [4] J. Friedman, T. Hastie, and R. Tibshirani. Additive logistic regression: a statistical view of boosting. The Annals of Statistics, 28(2):337374, April 2000. [5] M. J. Kearns and U. Vazirani. An Introduction to Computational Learning Theory. MIT Press, Cambridge, MA, 1994. [6] L. Mason, J. Baxter, P. Bartlett, and M. Frean. Functional gradient techniques for combining hypotheses. In A. Smola, P. Bartlett, B. Sch olkopf, and D. Schuurmans, editors, Advances in Large Margin Classiers, pages 221 247. MIT Press, Cambridge, MA, 1999. [7] NIST. the 1996 speaker recognition evaluation plan. http://www.nist.gov/speech/tests/spk/. [8] D. A. Reynolds, T. F. Quatieri, and R. B. Dunn. Speaker verication using adapted gaussian mixture models. Digital Signal Processing, 10:1941, 2000. [9] R. Schapire, Y. Freund, P. Bartlett, and W. S. Lee. Boosting the margin: A new explanation for the effectiveness of voting methods. The Annals of Statistics, 26(5):1651 1686, October 1998. [10] R. E. Schapire and Y. Singer. Improved boosting algorithms using condence-rated predictions. In Proceedings of the Eleventh Annual Conference on Computational Learning Theory, pages 8091, 1998. [11] L. Valiant. A theory of the learnable. Communications of ACM, 27(11):11341142, 1984. [12] R. Zemel and T. Pitassi. A gradient-based boosting algorithm for regression problems. In Advances in Neural Information Processing Systems, volume 13, Cambridge, MA, 2001. MIT Press.

20

Miss probability (in %)

10

Adapted GMM Boosted GMM

0.5

0.5

5 10 False Alarm probability (in %)

20

Figure 2: DET curves for boosted GMM and Adapted GMM on the same handset.
Speaker Detection Performance 60

40

Miss probability (in %)

20

10

Adapted GMM Boosted GMM

10 20 False Alarm probability (in %)

40

60

Figure 3: DET curves for boosted GMM and Adapted GMM on the different handsets

5.3. Conclusion We proposed a novel boosting learning based algorithm that could signicantly improve the performance of the baseline