2012 International Conference on Communication Systems and Network Technologies

Speech and Speaker Recognition System using Artificial Neural Networks and
Hidden Markov Model

Niladri Sekhar Dey* Ramakanta Mohanty K. L. chugh

Department of Comp. Sci. and Engg. Department of Comp. Sci. and Engg. Department of Comp. Sci. and Engg.
MLR Institute of Technology, MLR Institute of Technology, MLR Institute of Technology,
Hyderabad, AP, India Hyderabad, AP, India Hyderabad, AP, India
niladridey@mlrinstitutions.ac.in ramakanta5a@gmail.com kl_chugh@hotmail.com

* Corresponding author

Abstract— Aiming towards automatic machine learning by spectrum analysis depicted in figure 1 to machine voice
human, a methodology for speech recognition with speaker spectrum deprecated in figure 3 for speech recognition.
identification based on Hidden Markov Model for security is a
demand of science. Inspiring by the same, we propose a
methodology to identify speaker and detection of speech.
Within our research acquisition of speech signal, analysis of
spectrogram, neutralization, extraction of features for
recognition, mapping of speech using Artificial Neural
networks is presented. In our investigation such a method of
mapping is realized using back propagation rules of neural Fig. 1. Sample Input Spectrogram
networks. This algorithm is especially suitable for huge set of
input and output speech mapping. Additionally recognition of
speaker using Hidden Markov Model also will be presented in
this paper.

Keywords- Neutralization of Speech; Speaker Identification;

Speech Recognition; Spectrogram Analysis of Speech
I. INTRODUCTION
Speech is the most efficient way to train a machine or
communicate with a machine. This work focuses on the
objective to recognize the word or the phase spoken by
human, keywords in high speed. But as speech is a bio
metric property and bio metric property of humans are
difficult to recognize. Hence the human leaded learning
processes for machines are not so successful. Although the
recognition of speech using pattern mapping algorithms are
successful, especially continuous learning algorithms like
back propagation algorithm using any combination of
feedforward and feedback networks with Gradient searching
capabilities[1-3]. The generation of voice or speech signal is
from the vocal muscles, which depends on the physical
structure of human and it varies regularly. More over the
signal is also affected by the emotions of the speakers.
Therefore it’s required to neutralize the signals to derive
emotion free inputs to detection system, analysis of
spectrogram of the speech signal and further mapping of
input spectrogram set to output spectrogram set. As after the
neutralization the signal is free from emotions and the
spectrogram analysis generates the best input sets which can
be mapped to output dataset, which is obtained during
training, hence we propose an algorithm to map input
Fig. 3. Machine Voice Spectrogram
The rest of the paper is organized in the following
manner. A brief discussion about computer human
communication techniques is presented in Section II,
Section III describes proposed methodology, Setup and
outputs are presented in Section IV, Section V presents
results are discussions and Finally Section VI concludes the
paper.
II. RELATED WORK
For a long improvisation of computer human
communication is being done. Still due to a significant
number of draw backs those prototypes leads to multiple
malfunctions. For our convenience we classify the
prototypes in five different types to justify the improvised
method of learning.
In Standard Recorded Speech Processing systems, the
captured sound signals are matched with the standard inputs.
But the captured sound signals are varies speaker to speaker
by age, sex, anatomic variation and emotion. As most of
these types of systems do not provide any training features
hence the success rate is limited.
In Natural Language Processing systems, the theoretical
algorithms are followed, where most of the times the

978-0-7695-4692-6/12 $26.00 © 2012 IEEE 307

312
311
DOI 10.1109/CSNT.2012.221
calculation of noise are not properly included. Hence it
reflects in the output error.
In Accent Neutral Recognition System systems, the
processing is done based on the processed parameters
without considering the stresses on syllables. Hence in
majority the recognition fails. More over if this types of
systems does not include training, then the failure rate is
high once again.
In Processing of continuous Speech systems, the phases
are considered to be processed. But the processing
capabilities are limited in speed of computation. More over
the speed of speakers varies, which makes the system
unstable in recognition of speech.
In Hardware dependant processing systems the variation
of hardware like quality of microphones affects results.
In this research we discuss a type of prototype where the
variation of speaker is resolved using the help of speech
training of the system. More over the neutralization process
can reduce the emotional effects of the speech signals with
noises. As the learning is also parameter based, hence the
effects of variation in speech also can be trained to the
system. The utilization of artificial neural networks
furthermore makes the processing faster.
Acquisition of
Speech Signal
data and also speaker identification using HMM. The
algorithm is divided into six parts as
• Acquisition of the speech signal
• Spectrogram analysis
• Reduction of Noise
• Normalization of signal
• Recognition using ANN
• Recognition of speaker using HMM
B. Acquisition of the speech signal: A sound signal
is a physical quantity, which can be measured and as well be
formulated. The Fig. 4 demonstrates the process flow. The
input is a noisy and destabilize speech signal which is
basically a non stationary sinusoidal signal. Mathematically
that is defined as:
x(t ) = ¦ t =0 A k ( t ) cos (ωi ( t ) t + ϕi ( t ) )
n
(1)
C. Spectrogram Analysis: The most common format
is a graph with two geometric dimensions as the horizontal
axis represents time, the vertical axis is frequency. In
spectrogram a third dimension indicating the amplitude of a
particular frequency at a particular time is represented by
the intensity or color of each point in the image. The basic
mathematics is
n
x'(t) = STFT{x(t)} ≡ X(τ, ω) = ³ x(t) ω(t −τ)e− jωt dt (2)
t =0

Spectrogram Hence, analysis of non stationary signal combining Eq. (1)

Analysis and Eq. (2) is
n

Noise
x'(t) = STFT{x(t)} ≡ X(τ, ω) = ³ x(t) ω(t −τ)e− jωt dt (3)
Reduction t =0
D. Reduction of Noise: Most of the electronic
Normalization recording equipment has an effect of noise on the recorded
sound signal. But when it is to be processed for speech
recognition, then the minimum noise also can weigh down
Recognition using ANN Speaker Recognition
the processes of neural networks during training and
processing. Hence we tried to reduce the noise before
processing the signal for ANN. The mathematical
representation is as
Output Generation

B
x '(t )
Fig. 4. Process Flow x ''(t ) = ³ log 2 (1+ )df (4)
0
n(t )
III. PROPOSED METHODOLOGY
Where, B is the bandwidth of the channel. After the
A. Strategy: The algorithm involves the acquisition of reduction of noise the signal is stable and ready to be
the speech signal, processing, matching with the training normalized, 

312
313
308
 ∝ n
The input to this model is the processed speech
³ ¦AK(t)cos(ωi ( t) t +ϕi ( t)) ω(t −τ)e dt
− jωt
B
signal and output is the decision. The intermediate
hidden layers will adjust the weight for each node.
x''(t) =³log2[1+ −∝
t=0
]df During training the network is expandable, hence not
0
n(t) only the new data, new parameters also can be stored.
(5) X’’’ is considered as the input to the neural network,
where X’’’ is defined as
E. Normalization of Signal: Normalization of the n

sound signal is really effective when the signal need to X ''' = ¦ x '''i (8)
i =1
spectrogram analysis before recognition process.
Normalization of a sound signal can be peak normalization Combining the Eq. (7) and (8)
or loudness normalization. As our process is focused on
power spectrums, hence we choose the loudness ∝ n

³ ¦A (t)cos(ω( t) t+ϕ ( t)) ω(t −τ)e

normalization as the best method. It can be achieved by −jωt
n B K i i dt
1
X'' =¦[ [³log2(1+−∝
calculating the root mean square of the power spectrum. The t=0
mathematical representation is as
)df ]2i]
i=1 n 0 nt( )
(9)
1
x '''(t ) RMS = [ x ''(t )]2 (6)
n The Weight matrix is defined in the hidden layer 1
After the normalization the speech signal is ready layer as WI and in the output layer as WJ. The weight
to get processed matrix in the hidden layers and expandable layers is
denoted as WK. Where
μ n
³ ¦ AK(t)cos¨¨©wi(t)t+ji(t) ¹¸¸w(t-t)e dt 2
§ · - jwt n n
WI = ¦ wi Wk = ¦ wk
B
x '''(t)RMS = 1n[ ³ log2(1+-μt=0 )df ] (10), (11),
0 n(t) i=0 k =0
(7) n n

F. Recognition using ANN: The most important part WJ = ¦ w j (12), w j = ¦ Wk (13),

j =0 K =0
of speech recognition is to recognize the speech using
n n
Artificial Neural Networks of the speech signal after
wJ = ¦ ¦ wk (14)
normalization. Here we propose a multilayer pattern K = 0 k =0
mapping neural network, which works on the principle of
back propagation algorithm [4-7]. The design of this Moreover we represent the composite weight matrix as
multilayer neural network is based on research on various W = WI .WJ .WK (15)
neural network systems [Fig. 5]. The specialty of this model
is the flexible and expandable hidden layer for The output of this neural network is decision, though
recognition[8-10]. we represent that as Y. Whereas Y is formulated at
Y = f ( X .W )
Y = f ( X ''' .WI .WJ .WK ) (16)
Combing the Eq. (9) and (16)
∝ n

³¦A (t)cos(ω ( t) t +ϕ ( t) ) ω(t −τ)e

− j ωt

B K i i
dt
n
1
¦ ³
t =0
Y = f ( [ [ log2 (1+ −∝
2
)df ] i ].WWW
I
. J. K)
i =1 n0 n(t)
(17)
Combing the Eq. (10), (11), (12), (13) and (17)
∝ n

³ ¦A (t)cos( ω ( t) t +ϕ ( t) ) ω(t −τ)e

−jωt
dt
n B K i i n n n
Fig. 5. Multilayer Pattern Mapping Network 1
¦ [³log (1+ −∝ t=0
¦ ¦¦w .W )
2
Y = f( [ )df ] ]. w.
2 i i k K
i=1 n 0 nt
() i=0 K=0 k=0

313
314
309
 , Which makes the final formulation for speech spectrogram analyzer fully meets these requirements with
recognition. 3.6/8/26/43/46/50/67 GHz rate, Displayed average noise
G. Recognition of Speaker using HMM: Hidden level: –152 dBm at 2 GHz; –148 dBm at 26 GHz (1 Hz
Markov Model (HMM) is a powerful statistical tool for bandwidth) and Typ. 77 dB ACLR for 3GPP, typ. 84 dB
modeling generative sequences that can be characterized by with noise correction. For the recording we have used
general purpose hardware for storage.
an underlying process generating an observable sequence.
HMMs have found application in many areas interested in
signal processing, and in particular speech processing,
phrase chunking, and extracting target information from
documents. Here we propose a HMM for recognition of
speaker [Fig 6]. The input of the Hidden Markov Model is
the data rejected by ANN.

ANN Training Fig. 7. Acquisition of speech signal (Dataset 1)

Activation Acceptance
Processing
Parameters

Frequency
Rejected

Time
Fig. 8. Analysis of Spectrogram (Dataset 2)

Table 1: Analysis of training parameter and features

Amplitude Base Data Set -1 Data Set -2 Data Set -3 Data Set -4 Data Set -5 Sample
- - - - - -
12 27.422125 27.422125 27.422125 27.422125 27.422125 27.422125
- - - - - -
Fig. 6. Proposed Hidden Markov Model 23 27.554403 27.554403 27.554403 27.554403 27.554403 27.554403
35 -26.37182 -26.37182 -26.37182 -26.37182 -26.37182 -26.37182
47 -27.20977 -27.20977 -27.20977 -27.20977 -27.20977 -27.20977
The calculation of the cut-off value is pre-decided and in - - - - - -
the processing phase, the calculation with the input data will 59 25.490559 25.490559 25.490559 25.490559 25.490559 25.490559
be done. During processing, the dataset will be justified
with cu-off value and if not rejected then the analysis of the
frequency, time and amplitude will be done and extraction
of new feature will be calculated. With the new acceptance
parameter, artificial neural network will be retrained and the
hidden layer in ANN will be expanded.

IV. SETUP AND OUTPUT

During the research we have used most eligible setups
and equipments to get best analysis results. Thought the
cheapest hardware are also compatible with the work. We Fig. 9. Analysis of Acceptance report and Error Graph
have used Earthworks SR-30 for recording, based on David
Blackmer's principle that extending a microphone's
frequency response outside the normal range of hearing will
allow for a much higher-definition sound. To handle the V. RESULT AND DISCUSSION
wide variety of measurement tasks in product development, We recognized speech and speakers using Artificial
an instrument must offer ample functionality and excellent Neural Network based on frequency of speech ranging from
performance in all areas of interest. The R&S®FSU as

314
315
310
12Hz to 586Hz with five different dataset generated from
five different speaker. The input datasets are consisting of
intensity values from spectrum analysis of the speech signal.
During the spectrogram analysis amplitude extraction of the
speech signal is done, which is considered as the input to the
neural network. Before considering the amplitudes we
processed the signals for neutralization. As the effect of
noise is reduced, we got the speech signal ranging from 0dB
to 140dB. Still this signal is not ready to be process by the
artificial neural network as the signal is not yet normalized
as contains multiple peek variation due to syllable and
accent effect. Therefore, we neutralize the speech signal
hence the long spectra got feasible to study and silent
spectra got remarkably separable from the long spectra,
which makes the speech signal ready to be processed with
artificial neural networks. The ANN will recognize the
speech signals based on the predefined acoustics
parameters. The eight acoustic features that were used
included the four formants F1 through F4, the spectral
slope, harmonic difference H1-H2 and the aperiodicity &
periodicity contents in the speech signal. Moreover during
training the nodes are expandable to accommodate the new
features. The use of hidden markov model finally decides
the rejection or addition of new parameter in hidden layers.
The calculation of the cut-off value is pre-decided and in the
processing phase, the calculation with the input data will be
done. During processing, the dataset will be justified with
cut-off value and if not rejected then the analysis of the
frequency, time and amplitude will be done and extraction
of new feature will be calculated.
In the last decade, it has continued with the EARS
project, which undertook recognition of Mandarin and
Arabic in addition to English, and the GALE project, which
focused solely on Mandarin and Arabic and required
translation simultaneously with speech recognition. Still
those systems are not free from errors as most of times
theoretical algorithms are applied. We have produced
system for testing on word “Hello” for five different
speakers [Table 1] and matched with the system voice. In
the analysis of acceptance report [Fig.9] we have noticed
that more than eighty percent of the cases, without much
training, the data are accepted. In remaining fifteen percent
cases it need to extract new parameter for training and
recognition.
VI. CONCLUSION
We presented the application of artificial neural
networks and hidden markov model for speech and speaker
reorganization. The work is majorly focused to acquisition
of speech signal, analysis of spectrogram, neutralization,
extraction of features for recognition, mapping of speech
using Artificial Neural networks. Moreover additionally
recognition of speaker using Hidden Markov Model also
made into this work, which generates the new features for
recognition. This work will be generalized in future for
human lead machine learning.
315
316
311
ACKNOWLEDGMENT
The parts of this research are supported by Prof. P.
Rammohan Rao, Mr. L. Naveen Kumar, Mr. K. Sandeep
and Prof. N. Subba Reddy.
REFERENCES
[1] C. T. Chen, W. D. Chang, «A Feedforward Neural
Network with Function Shape Autotuning», Neural
Networks, Vol.9, No 4, pp. 627-641, June 1996.
[2] F. Piazza, A. Uncini, M. Zenobi, «Artificial Neural
Networks with Adaptive Polynomial Activation Function»,
in Proceedings of IJCNN, Beijing, Cina, pp. II-343-349,
Nov. 1992.
[3] F. Piazza, A. Uncini, M. Zenobi, «Neural Networks with
Digital LUT Activation Function», in Proceedings of
IJCNN, Nagoya, Japan, pp. II-1401-1404, 1993.
[4] J-N Hwang, S-R Lay, M Maechler, R.D. Martin, J.
Schimert, «Regression Modeling in Back-Propagation and
Projection Pursuit Learning» IEEE Transactions on Neural
Networks, 5(2), 342-353.
[5] S. Guarnieri, F. Piazza, A. Uncini, «Multilayer Neural
Networks with Adaptive Spline-based Activation
Functions» In Proceedings of the WCNN 95, Washington
D.C., USA, I695-I699, 1995.
[6] L. Vecci, P. Campolucci, F. Piazza, A. Uncini,
«Approximation Capabilities of Adaptive Spline Neural
Networks» In Proceedings of ICNN’97 Huston TX 96,
,1997.
[7] L. Vecci, F. Piazza and A. Uncini, «Learning and
Approximation Capabilities of Adaptive Spline Activation
Function Neural Networks», accepted for publication in
Neural Networks.
[8] E. Catmull, R. Rom, «A Class of Local Interpolating
Splines», in R. E. Barnhill, R. F. Riesenfeld (ed.), Computer
Aided Geometric Design, Academic Press, NewYork, 1974,
pp. 317-326.
[9] N. Benvenuto, M. Marchesi, F. Piazza and A. Uncini «A
Comparison between Real and Complex valued Neural
Networks in Communication Applications», Proc. of Intern.
Conference on Neural Networks, ICANN91, Espoo,
Finland, June 1991.
[10] H. Leung, S. Haykin, «The Complex Backpropagation
Algorithm», IEEE Trans Acoust. Speech and Signal
Process, Vol.ASSP- 39, pp.2101-2104, Sept. 1991.