Measurement

ELSEVIER
Measurement 23 (1998) 93 115

Artificial neural networks in measurements

P. Daponte a,,, D. Grimaldi b
" Dip. di Ing. dell'Inf ed lngegneria Elettrica, Universit~ di Salerno, via Ponte don Melillo 1, 84084 Fisciano, (SA), Italy b Dip di Elettronica, Informatica e Sistemistica, Universit~ della Calabria. 87036 Rende, (CS). Ita(t'

Abstract

This paper, after a brief theoretical background to Artificial Neural Networks (ANNs) reviews their utilization in the field of measurements. The criteria used for choosing between the possible ANN software and hardware implementations are then discussed. The advantages and disadvantages of the possible analogue or digital hardware implementation are also detailed. Some of the applications of ANNs in the measurements field, variously utilizing software, analogue or digital hardware implementations are shown. In particular, the strategies for building suitable ANN-based software models of mixed analogue/digital measurement devices are presented. Successively, a method for designing a neural network able to pre-process signals in an analogue mode is described. Finally, the architecture of a DSP-based neural apparatus for the real-time measurement of the components of a dual-tone multi-frequency signal is given. 1998 Elsevier Science Ltd. All rights reserved.
Keywords." Neural networks; Measurement; DSP

I. Introduction

Artificial Neural Networks (ANNs) are input information processing structures which imitate human brain behaviour [1]. They closely resemble the Turing machine in which the fundamental functions of memory, processing activity and control facilities are interwoven to a single element in order to achieve fast response time. The parallel connection of such elements over the whole structure, and their ability to mutually transfer information, yields a parallel computing architecture with enhanced computational power. ANN structures have generated great interest in the last few years in different disciplines and there are many specialized applications throughout the

* Corresponding author. Tel: +39 89 964248; Fax: +39 89 964218, c-mail: daponte@unina.it 0263-2241/98/$19.0(/ 1998 Elsevier Science Ltd. All rights reserved. PIl: S0263-2241 (98)0001 3-X

fields of technology and engineering and many technical committees, special issues, special congress sections and books dealing with neural networks. Their success is directly dependent upon their ability to simplify and accomplish with speed many tasks not feasible using conventional techniques [2]. The inside benefits which derive from the processing structure of the ANNs are: (i) an adaptive structure which takes data, learns and captures quite suitable relationships also in complex interactions between input/output information: (ii) an ability to generalize and handle incomplete data or data corrupted by high noise levels: and (iii) a simultaneous and rapid execution of numerous, identical and independent, operations. In order to take greater advantage of ANN real applications, and to become competitive with the traditional ones, different, specialized implementation alternatives are available. They consist of (i)

94

P. Daponte, D. Grimaldi / Measurement 23 (1998) 93 1t5

software packages running on personal computers (PC), UNIX-based workstations or on massively parallel computers (neuro-computers); (ii) accelerator boards for PC and workstations; and (iii) dedicated analogue and digital processors designed for neural networks. Obviously, the ANNs achieve their full potential only when implemented on hardware structures characterized by parallel architecture [3]. The benefits of the A N N are advantageously utilized in many different applications in the measurement field. These are concerned with: (i) instrument and sensor calibration, (ii) development of new measurement methodologies, (iii) identification and on-line monitoring of complex dynamics, and (iv) analogue and digital signal processing. A brief description, given above, of typical applications should enable appreciation of their advantages, The effectiveness of ANNs in the field of measurements may be attributed to the fact that they are a powerful tool for non-linear approximations and can be used when all other methods have failed [4]. Many more applications can be thought up and a deep knowledge of the software and hardware characteristics of the actual realizations can increase their diffusion. This paper provides a review of the specialized software and hardware implementations of ANN application in the measurement field. ANN applications are manifold and it has become increasingly difficult to follow all those being made. Accordingly, the aim of this paper is not to provide an exhaustive survey of all the software and hardware characteristics of the actual realizations, but to provide an application-oriented reference point for those who are interested in utilizing ANNs in the field of measurements. The paper is divided as follows: firstly, a theoretical background to ANNs concerning their application in the measurement field is given. Successively, the characteristics of A N N software and hardware specialized implementations, and the results of a suitable ANN-based model of the mixed analogue/digital measurement devices are presented. Next an application of analogue neural networks able to pre-process signals in an analogue

mode is given, and finally, the hardware and software architecture of a neural apparatus based on a digital signal processor for the non-intrusive real-time monitoring of the tones of the dual-tone multi-frequency signal is described.

2. The theoretical background to ANNs

Development of the ANNs derives from a computational approach to neuroscience enabling new algorithms and architectures to be extracted from neurobiological systems. When this approach is coupled with the latest advances in VLSI technology, there is a clear opportunity for utilizing this to perform high-speed signal processing. Research into biological systems has shown the organization of the nervous system. As shown in Fig. 1, different levels of organization can be individuated in the nervous system. An analytical model corresponding to each level for determining the proper processing algorithm and the electronic architecture necessary to perform the processing objective can be obtained [5]. Both the algorithm and the electronic architecture are based on the properties and features of neurones and synapses. Neurones are the base units in which all processing activities are performed and are characterized by a summing property and a non-linear transfer characteristic, typically similar to the sigmoidal one. Synapses are the connections which transmit the information. These can be transmitted with modification by means of the synapse weights. The key feature of the ANN structure consists of a parallel distributed network, comprising many artificial neurones interconnected by weighted connections. Each artificial neurone is characterized by an input vector, a single output value and a non-linear transfer characteristic. It processes information in a predetermined manner and furnishes the results either as the ANN's output or to the input of another neurone. The weighted connections store the information and the value of the weights is pre-defined or determined by a learning algorithm. The cross-connections between the neurones form: (i) a set of input layer neurones, (ii) a set of output layer neurones, and (iii) a set

P. Daponte, D. Grimaldi / Measurement 23 (1998) 93 115

95

out

ill

........... !

Model o f the nervous system

,.

'i / \ i

Model o f the brain regions

Model o f the neural network

i

cHP l
" \i

Model of the
Ileuron@s
- x

N ....

c=.

lout

Model of the Wi (k+l)--'~ synapses

(k)+A w i (k)
EE Synapse ~'~lit OM

Fig. 1. Analytical model and electronic architecture corresponding to the different levels of organization of the nervous system.

of hidden layer neurones, The choice of the neurone number of the input and output layers is closely connected to the desired accuracy. The number and dimensions of the hidden layers depend only on the A N N ' s performance achieved in terms of model fidelity and operating speed.

Amongst the numerous A N N architectures available in the literature, the feed-forward networks with one or more hidden layers and the back-propagation learning algorithm [6] are the most c o m m o n in measurement applications. Indeed, when compared with other architectures,

96

P. Daponte, D. Grimaldi / Measurement 23 (1998) 93 l l5

they offer several successful applications in pattern classification, pattern matching, function approximation and, also, in the learning of any non-linear mapping to any desired degree of accuracy. Limitations arise from both the dimensions and variability of the input array. Reasonably small dimensions, only a few hundred input array items, and little variability in values or in the position of the input array items are required [7]. In order to circumvent one or more of these limitations a myriad of different solutions have been proposed [8]. The characteristics of the software based on these solutions are given in Tables 1 and 2. More details on ANN topologies can be found in Ref. [9]. For the sake of brevity, and with the aim of giving a deeper knowledge of particulars, only the feed-forward networks with one or more hidden layers and the back-propagation learning algorithm are considered in the following. As is well known, the use of this ANN involves three separate phases, depicted in Fig. 2, which have to be followed: (1) The learningphase, Fig. 2a, in which the ANN is forced to furnish the desired outputs in correspondence to a determined input (the learning set); in this phase, the ANN learning level is verified by means of suitable performance indices, the objective function, defined coherently by the data of the learning set. (2) The validation phase, Fig. 2b, in which the ANN's generalization capability is verified by means of data (the validation set) completely different from the data used in the previous phase. (3) The production phase, Fig. 2c, in which the ANN is capable of providing the outputs required that correspond to any input. In order to train the ANN to model physical phenomena it is necessary: (i) to specify the learning set constituted by an adequate number of input and target output vector pairs, and (ii) to minimize the objective function, defined as the relative difference, normally in the Euclidean sense, between the actual ANN output and the target outputs. It is possible to minimize the objective function by using a learning algorithm. The strat-

egy of the learning algorithm is devoted: (i) to determining the input/output transfer characteristic of each neurone by a supervised learning section, and (ii) to modifying the connection weights and the neurone bias by means of an adaptive process (error back-propagation) which minimizes the output neurone errors. This phase ends when the ANN furnishes the outputs necessary for the learning set. Progress in the ANN learning phase is monitored through a decrease in the maximum relative error. Some effort must be devoted to determining the learning set in order to obtain a high ANN accuracy. In the validation phase, the ANN accuracy is tested using the validation set. These vector pairs are different from those of the learning phase, but have similar characteristics. If the ANN's performance does not reach the desired level of accuracy a new learning phase and a modified learning set are necessary. The modified learning set must take into account additional new information obtained by the previous set. After the learning and validation phases, the production phase begins and, consequently, the ANN is used to provide the corresponding outputs required for any input. The ANN trained in this manner represents a valid model of the dynamically non-linear phenomena for which it has been trained. The main cost will be the time taken to set up the learning phase.

3. ANN applications in the measurement field

There are numerous applications of ANNs in measurements, some requiring software implementation, others requiring hardware implementation. On the basis of the task involved, all these applications can be classified as applications regarding (i) the measurement procedures, such as sensor and instrument calibration; (ii) the devices of the measurement systems, such as the analogue-to-digital converter (ADC), the digital-to-analogue converter (DAC), amplifiers and filters; and (iii) measurement signal processing as shape classification

P. Daponte, D. Grimaldi / Measurement 23 (1998) 93 115

Table 1 Characteristics of the freeware and shareware software for ANN simulation Rochester Connectionist Simulator NeurDS PlaNet 5.7 GENESIS 2.0

97

SNNS 4.1

Aspirin/MIGRAINES 6.0

Atree 3.0 Educational Kit for Windows PDP + + Uts MUlti-Module neural computing Environment NevProp

PYGMALION Matrix BackPropagation (MBP) WinNN BIOSIM The Brain FuNeGen 1.0 NeuDL

AINET DenoGNG

nn/xnn

NNDT 1.4 Trajan 2.1 Neural Networks at your Fingertips

It is for arbitrary types of neural networks and comes with a back-propagation package and X1 l/Sunview interface It is a simulator for DEC systems supporting VT100 terminal It is a conneetionist simulator with version to run only under X Windows It supports complex models of single neurones and large networks. Run on Unix platforms and is available parallel version for networks of workstations, symmetric multiprocessors and MPPs It supports back-propogation, counterpropagation, quickprop, backpercolation I, generalize Radial Basis FunctionRporp, ART1, ART2, ARTMAP, Cascade Correlation, Hopfield network, Time-Delay Neural Network. It is user-extendible and works on SunOS, Solaris, IRIX, Ultrix, OSF, AIX, HP/UX and Linux It is a C code generator for neural network simulations by reading a network description, written in Aspirin language. The MIGRAINES interface exports data to visualization tools. The software works on a large number of platforms It serves to develop simple applications using adaptive logic network (ALN). Two simple languages describe ALN and the steps of training an ALN It is written in C + + and implements feed-forward, back-propagation, Boltzman machine, Hopfield, Mean-field algorithms. Works on Unix with X-Windows The user interface is readily modifiable and scripts written in Tool Control Language (TCL) are used. Uts implements only the connectionist paradigm It provides an object-oriented facility for simulation and training. It furnishs a library of various architectures and learning algorithms The Nevada BackPropagation is a C source code implementing feed-forward backpropagation. It is a modified version of Quickprop 1.0 by adding the ability to force gradient descent and useful additional options It is a prototype that stems from an ESPRIT project. It implements back-propagation, self-organizing map and Hopfield network It is an efficient implementation of the back-propagation that includes an adaptive technique for gradient descent. It reaches peak-performance on RISC architectures It is a Neural Network package for Windows 3.1. It implements feed-forward, multilayer NN and uses modified fast back-propagation for training It is a biologically-oriented neural network simulator. It runs on Unix and PC and has a graphical user interface designed for research and teaching It is based upon the back-propagation learing algorithm and runs on a PC It is a MLP-based program to generate fuzzy rule-based classifiers The Neural-Network Description Language is an interpreted language resembling C or C + + and implements the back-propagation model. It offers the user to specify the connections between nodes and create or destroy connections as training progresses It is an application for MS-Windows 3.1 and does not require learning. It is not sensitive towards noise and provides a way to estimate the rate of error It is useful for demonstration and teaching purposes. It is written in Java and implements: Hard Competitive Learning, Neural Gas, Competitive Hebbian Learning and Growing Neural Gas It generates C code or executable programs and everything may be controlled by the graphical interface xnn. It implements: Madaline, back-propagation, counterpropagation, Elman, GRNN, Hopfield, Jordan, LVQ, Perceptron, RBFNN, Kohonen The Neural Network Development Tool runs on a PC with MS-Windows 3.1. It implements the MLP trained by the Levenberg-Marquardt method It implements the MLP with back-propagation and Kohonen network. The registered version can support up to 128 layers with up to 8192 units each It is a package for educational purposes. The supported architectures are: Adaline, back-propagation, Hopfield, Bi-directinal Associative Memory, Boltzman, counterprogagation, self-organizing map and Adaptive Resonance Theory

98

P. Daponte, D. Grimaldi / Measurement 23 (1998) 03 115

Table 2 Characteristics of the commercial software packages for AN N simulation BrainMaker SAS NeuralWorks MATLAB Neural Network Tool Propagator Neuroforecaster & VisualData NESTOR Neuroshell2 It trains back-propagation. It is available version for accelerator board, version for Intel 80170 neural chop set and multi-chip board It is oriented toward data analysis and statistical applications. Training is done by stateof-the-art numerical optimization algorithms instead of back-propagation It supports over 30 different networks: back-propagation, Art-l, Kohonen, Modula Neural Network, General Regression, Fuzzy ART-map, probabilistic networks etc. It is a tool for the design, training and simulation of neural networks. It supports a wide range of architectures with an unlimited number of neurones and interconnections It implements a back-propagation network with five layers and 32,000 neurones per layer It is specifically designed for building sophisticated and powerful forecasting and decision-support systems. It implements 12 Neuro-Fuzzy neural models It is a multilayer, feed-forward system with low connectivity within each layer. It operates in real time without special hardware. It is ideal for VLSI technology It combines neural network architectures, a Windows icon driven user interface and sophisticated utilities for MS-Windows machines. Includes a choice of 15 back-propagation, Kohonen, PNN and GRNN architectures It is a tool in dynamic link library that can create as many as 128 interactive networks, each with 32 slabs in a single network and 32K neurones in a slab It uses the back-propagation techniques to initially select fuzzy rules and membership functions. The result is a fuzzy associative memory (FAM) which implements an approximation of the training data It is an integrated environment for visual and quantitative data analysis and pattern recognition. It integrates data analysis and modelling tools for ANN, fuzzy logic genetic algorithms and statistics. Three modules are available: Analytical Visual Bas (AVB), Data Structure Analysis and Classification and Prediction It is a graphical simulation tool and supports trajectory learning with back-propagation through time. It uses a variety of unsupervised learning procedures. During learing, the parameters are dynamically adjusted It is an advanced modelling system ideal for developing and implementing ANN solutions under Windows. Complete, interactive analysis is available during training It is a block-diagram-based neural network library for extend simulation scripting language. The library consists of more than 70 functional blocks for ANN implementation and can include compiled program modules written in other languages It is a C + + class library that implements feed-forward, simple recurrent and randomorder recurrent networks trained by back-propagation. It supports all standard network parameters, includes 5 activation functions and 3 error functions It consists of six products: client/server for OS/2, Windows. The features include: data translation and scaling, application generation, multiple network models and automated network training. It support fuzzy rule systems combined with neural networks It is a neural network development tool that uses genetic algorithms to optimize the inputs and structure of a neural network. It is also equipped for predicting time-based information such as instruments and process faults It trains feed-foward networks having perceptrons in the first hidden layer and logic gates AND and OR in other hidden layers. Users can specify constraints on monotonicity, slopes and convexity of functions being learnt The Trans-Dimensional Learning allows for the fast creation of automatically constructed neural networks and help to optimize pattern recognition accuracy It is a graphical, object-oriented software product that enables users to easily build neural networks and integrate them into G2 application. It is well suited for control, data and sensor validation, fault classification and multivariable quality control applications It provides a visual, object-oriented approach to problem solving using intelligence technologies, including neural network and neuro-fuzzy techniques It provides a set of 20 neural network paradigms in the form of a programming library for C or C + +. It facilitates the construction of others by concatenation of simpler networks

NeuroWindows NeuFuz4

PARTEK

NeuroSolutions v2.0

Qnet for Windows version 2.0 NeurlLab

havBpNet + +

IBM Neural Network Utility

NeuroGenetic Optimise (NGO) version 2.0

Aree 3.0 Adaptive Logic Network Development System for Windows TDL version 1.1 NeurOn-Line

NeuFrame OWL Neural Network Library

P. Daponte, D. Grimaldi / Measurement 23 (1998) 93- 115

99

DUT in

DUT

tout

in,
...."5 ~-

a)

in J -]
c)

Fig. 2. Neural modellingphases of the deviceunder test (DUT): (a) learning phase; (b) validationphase: (c) production phases (in: input: out: output of the DUT to be modelled;y: output of the DUT neural model; E: output error).

and the detection of the signal characteristics in the time domain. In the following, a short and non-exhaustive list of ANN applications is given. (1) ANNs have recently gained more attention with regard to instrument and sensor calibration. By incorporating the ANN in the calibration process, accuracy has been increased and interference from other variations has easily been compensated [ 10,11 ]. ANNs have been used for (i) the improvement of both ultrasonic distance sensor [12] and optical fibre sensors for displacement and surface reflectivity measurements [13], (ii) the sensor compensation of full bridge strain gauges [14], and (iii) the calibration of sensor arrays, robot arms [4], artificial noses [ 15] and industrial microwave six-port instruments [16]. Moreover, integration of the neural network and the sensor systems on a single substrate, creates a more compact, less weight, self-contained sensor system. The resulting system operates with the lowest possible power dissipation, in order to avoid setting up temperature gradients which could affect the operation of the sensor. Furthermore, this integration reduces noise interference [14]. Finally, a special ANN has been utilized for control and stabilization of the frequency difference of laser sources by processing the signal generated by an interferometer [17]. (2) Some success has been had in using ANNs for the new measurement methodologies. In astronomy they have been utilized in the recon-

struction of a mathematical description of the incoming wave-front. Measurements are made by means of wave-front sensors to sample the incoming optical beam. By using the stages of the ANNs the difficulties of interpreting the noisy signals obtained from weak light sources have been overcome [4]. In mineral processing the ANNs have been demonstrated to be more effective than classical techniques for prediction of densities and distributions of hydrocyclone [18]. Moreover, the ANNs have been applied to improve the automatic time report system [19] avoiding both the time consuming and the error of the traditional time calibration, usually based on the utilization of broadcasting signals from a standard time broadcasting station. One example of a practical installation on the production line is that of measurement in real-time of the thickness of the oxide or chrome coating on aluminium sheeting. In this case, the A N N is able to interpret in real time the coating thickness from the reflected light [4]. (3) The monitoring, estimation and diagnosis of complex dynamics by means of ANNs has been usefully applied to linear, non-linear and hybrid measurement devices. ANNs have been used for (i) diagnosis and monitoring of transformer heating by also taking into account the environmental influences [20], (ii) on-line phasor detection and identification of the power system [21], (iii) historical data analysis for short time forecasting of electrical load [22], (iv) instrument fault detection [23] and (v) real-time monitoring of distribution

100

P. Daponte, D. Grimaldi / Measurement 23 (1998) 93 115

systems [24]. In this last case, the ANN allows on-line fault detection and diagnosis of real systems, with an accuracy typical of mathematical programming. Moreover, there are applications in which ANNs are used together with: (i) fuzzy logic for the diagnosis of technical failure and the reasons for mechanical failure [25], and (ii) algorithms based on wavelet transform and time series analysis for detecting both the device and process state by processing the wideband signal taken from a sensor [26]. (4) ANNs have been utilized in the recognition of the signal waveform. Due to their inherent parallelism, ANNs are able to detect signal parameters and, moreover, to classify the sampled input signal into a number of classes with an adequately quick reaction. ANNs have been used to build a DSP-based apparatus for the non-intrusive on-line and real-time monitoring of the tones of the DTMF signal [27]. Another application is the fast and contemporaneous measurement of more than one parameter of signals. Significant examples are the on-line measurement of (i) both the amplitude and time constant of the signal in nuclear fusion research [28] and (ii) both the frequency and the amplitude of harmonic signals for frequencydomain analysis [29]. Another example is ARTMAP [30], which is a new neural network architecture able to shape classification. It does not need a learning phase and is characterized by a self-organizing phase. It autonomously learns to classify arbitrarily ordered vectors into recognition categories. ARTMAP has been successfully utilized in the recognition of electrocardiogram (ECG) signals [31]. (5) The ANN-based approach has also been applied in the design of new ADCs [32,33]. In Ref. [34] a neural ADC converter has been proposed which is characterized by a very simple circuit based on a cascade of 1-bit stages including only one analogue comparator and one amplifier. Moreover, by using opportune neural blocks a full neural Gray-code based ADC has been realized [35]. The behaviour of both these converters is completely asynchronous as predicted by the theory of neural networks. Their performance includes (i) speed, (ii) simplicity, (iii) the nonnecessity of clock signals, and (iv) low power. These new neural ADCs overcome the limitations

of the traditional ADCs, which are large and complex and are not adapted to on-chip sensor signal conversion. Furthermore, ANNs have been utilized for ADC dynamic error compensation [36]. In this case, the generalization capability of the neural networks permits reduction of the huge number of experiments which would otherwise be required. (6) On the basis of the artificial neurone characteristics, a design method for transforming a signal processing scheme into an artificial neural structure has been developed [37]. This is based on the idea of performing all the processing operations by means of neural blocks. In this way, an entirely analogue processing structure can be obtained. The first step of the design method involves the transformation of basic mathematical and decision operations, included in the processing or generation of signals, into neural blocks, each one performing the basic algebraic and differential operations. The second step involves the opportune connection of the neural blocks and the third concerns the optimization of the connections between neural blocks by utilizing the multiplication and adding properties of the neurones. By following this path more complex structures for the performance of different operations, such as multiplier and analogue-to-digital conversion structures, and the thermal compensation of the sensor output with successive analogue-to-digital conversion structures, have also been implemented [ 14]. (7) The capacity of ANNs to model complex non-linear systems has been utilized in the modelling of measurement devices. In particular, two categories of components have been considered: (i) analogue measurement devices, such as sensors, transducers, filters and amplifiers; and (ii) the mixed analogue/digital measurement devices, such as ADCs and DACs. By using a suitable ANN structure, and a proper organization of the training set, it is possible, after a proper set-up phase, to obtain a neural model furnishing an output which corresponds to that which can be obtained from the real device to be modelled. The model's accuracy depends on the choice of both the ANN structure and the training set. In this way, the difficulties arising from the traditional method for modelling devices characterized by transfer func-

P. Daponte, D. Grimaldi / Measurement 23 (1998) 93 115

101

tions, which can be complex and/or hardly nonlinear or piece-wise linear, are overcome [38]. The ANN capability to model measurement devices, has been utilized to develop new architectures of ADC. In particular, in Refs. [38,39] a model of a neural ADC based on a feed-forward neural network has been proposed, in Ref. [40] a decentralized neural architecture has been proposed and in Ref. [41] a back-propagation A N N has been included in a scheme for ADC resolution enhancement.

When a very high speed response of the ANN is required, a number of alternatives are available. It is possible to utilize software simulators running on specific workstations such as DEC Alpha or Sun SPARC to obtain a response speed of up to a factor of 20 compared with the PC speed. More speed is made available by plugging a high performance accelerator board into the PC. With this aim, there are some products commercially available on the market and Table 3 shows some of their characteristics.

4. The software implementation of ANNs

As can be seen from those applications of the ANNs described above, applications exist in the measurement field which do not require specialized hardware implementations, but only competent software simulations of the ANNs. The majority of real applications end up with implementation on the PC. The PC is an excellent platform, while the Unix-based workstations or massively parallel computers are used by those interested in the theoretical aspects of the neural network. Typically, the competent software simulation works on files of data stored on disk. It may be noted that this software is hard to adapt to a situation where it necessary to use real input/ output data and to obtain a quick response. Many A N N software simulators are available free of charge to the public on the Internet [42,43]. There are, also, simulators for the massively parallel computer [44]. Generally, the simulators make the process of A N N design easier. They make more ANN architectures available and also provide graphic tools which permit the visualization in two or three-dimensions. Moreover, they are also able to show graphs of the error rate during the learning phase. Table 1 shows the characteristics of the freeware and shareware software for A N N simulation. Table 2 shows the characteristics of the commercial software packages for ANN simulation. Obviously, the tables show only that software which can be utilized in the field of measurements. More information can be obtained by visiting the web site http://www.ci.tuwien.ac.at.

5. The hardware implementation of ANNs

Unlike the previous alternatives, in any applications, such as that of monitoring real systems and detecting real signals, a hardware neural network is required. Hardware realization makes it possible to execute forward pass operations on the neural networks at high speeds, thus making ANNs possible candidates for real-time applications. There are different ways of implementing a hardware neural network [45]. Integrated realizations have been proposed using either analogue, digital or mixed-signal approaches. The characteristics and performance of those hardware products, having digital or analogue technology, commercially available on the market are shown in Table 4.

5.1. Digital implementation

The digital implementation of ANNs [46] is based on binary logic circuits, where the working operations are determined by an internal clock signal. Fig. 3a shows a general digital structure for A N N implementation. In this structure two speed memories may be noted: a memory for storing the neural network weight matrices and a memory for storing the input vector data. These memories can be taken of volatile or non-volatile technologies. If the volatile memory is accepted, the choice is between two different memory technologies. One is the dynamic random access memory, utilized if a large storage capacity, high density and fast update rates are desired. Another is the static random access memory, utilized if a faster speed is desired and density is less important. Moreover,

102

P. Daponte, D. Grimaldi / Measurement 23 (1998) 93 115

Table 3 Commerically available high performance cards for ANN implementation Description Brain Maker Accelerator Brain Maker Professional Accelerator HNC Balboa IBM ZISC/ISA Accelerator NT5000 & NT6000 NeurDynam X (NDX) Neural-Accelerators XR25 NeurDynamX (NDX) Neural-Accelerators XP50 Ni I000 Develop.System PCI 4000 Recognition Accelerator VME 4000 VME Card Rapid Imaging VME Ultima Software plus ISA-BUS 20 MHz DSP board based on TMS 320C25 chip Software plus ISA-BUS board with 4 DSP running in parallel Hardware and software system using a VME DSP-based card Card with 16 ZISC036 chips for ISA-BUS Standalone system with co-processor trained by PC and running independently ISA-BUS card with i860 XR RISC processor at 25 MHz ISA-BUS card with i860 XR R1SC processor at 25 MHz PC-based development system including ISA card with Ni 1000 chip It holds 1 to 4 Ni 1000 chips, PCI-BUS with data transfer rates of 64 and 128 MB s t It holds 1-4 Nestor Ni 1000 chips running at 25 MHz General purpose VME card-based on the lntel 80170NX ETANN chip Performance It operates 3 million connections/s It operates 40 million connections/s 80 megaflop and 20 neural networks architectures C language for training and recall 600 neurones, regular version, 4000, turbo version 22.5M connections/s with two banks of 64 MB of RAM 45M connections/s with eight banks of 65 MB of RAM Accelerates DOS and Win emulation by a factor of 25, 1500 patterns/s Delivers up to 80K patterns/s with 4 chip version Local control processor and local dual-ported memory Up to 128 inputs, on board learning, analogue-in analogue-out, real-time processing

the digital structure requires input and output interface sections constituted by an input ADC and an output DAC. The advantages of digital implementation are noise immunity and the speed with which a single operation can be performed. Indeed, the chips can have clock rates of several hundred MHz, and gallium arsenide chips can have a clock rate up to several GHz. Another advantage is that of high precision computations. The precision of a PC simulation of the A N N matches the behaviour of the digital hardware implementation. The advantages also include connection programmability, weight storage through standard memory, on-chip learning capability through the use of logic protocols, and the fact that multiple signals can be easily multiplexed and demultiplexed. In this case, the number of wires needed to provide complex network topologies is drastically reduced. A fundamental disadvantage of digital implementation, when compared with the analogue, is that of substantial power dissipation and the large

silicon area. Indeed, digital implementation requires more processing elements for the implementation of fundamental operations such as multiplication and addition. Moreover, the implementation of non-linear functions is difficult to achieve and the processing of analogue signals requires additional circuitry for the analogue-todigital and digital-to-analogue interfacing. In Table 4, the neuro chips ACC, CNAPS, IBMZISC036, MCE MT19003, Nestor NIl000, NLX420, Philips L-Neuro, Siemens MA-16, pRAM, HNC 100NAP, MD-1220 and NeuFuz are examples of digital implementation. In particular, CNAPS is one of the first commercial examples of large neuro-computers. It has 13.6 million transistors and 80 processing nodes. Each processing node is similar to a simple digital signal processor (DSP). The great advantage of the CNAPS' architecture is the versatility of the processing node and its good programmability. Moreover, this chip has an internal architecture which makes expansion easy by the simple addition of more chips. In contrast with the previous chip, the Intel Nestor Nil000 is a chip aimed at the simulation of radial

P. Daponte, D. Grimaldi / Measurement 23 (1998) 93 115 Table 4 Commercially available analogue and digital neural network chips Neural ACC CNAPS IBM ZISC036 MCE MT 19003 Nestor Nil000 NLX420 Philips L-Neuro 1.0 Philips L-Neuro 2.3 Siemens MA-16 Description Parallel processor with 8192 neurones, it uses 16 bits Based on CNAPS-1064 DSP with 64 sup-DSP with 16-bit fixed point 64 inputs with 8-bit, 36 Radical Basis Function 12-bit internal values, 16-bit multiplier, 35-bit accumulator Up to 1024 prototypes stored, 256 dimensions, 5-bit for dimension 16-bit input, emulate 1048576 neurones with 64k synaptic It contains 16 neurones with 16-bit register, 16-bit weights 12 processors in parallel or pipeline, 16-bit weights Fast matrix-matrix multiplier Learning Performance

103

On-chip with training algor- 140 MCPS single chip, 1.4GCPS l0 ithm chips Programmable on-chip 1.28 billion multiply/accumulates/s, 293 million weight updates/s On-chip 4 gs for classification at 16 MHz No on-chip On-chip No on-chip On-chip On-chip On-chip I synapse per clock cycle (40 MHz) 40k 256 element patterns per second (25 tas/pattern) Up to 300 million multiaccumulates/s 26 MCPS, for 8-bit mode 720 MCPS, 12 weights are updated in parallel with 34 ns 3.2 billion multiplications ( 16-bit x 16-bit I and additions (48-bit) per second Training of 256 neurones in less than 0.25 ms 500M CPS in feed-forward mode, 128M CPS for back-propagation 80 million conections/s, 800ns to process any synaptic input or bias At 32 MHz the chip is capable of one billion operations

pRAM-256 HNC 100 NAP MD-1220 TOTEM + +

NeuFuz/COP8

Fuzzy logic provides a very flexible and powerful method for real applications MM32k It contains 2048 neurones con- It comes with a host PC pro- On lSA-BUS PC card, it shows nected by means of switching gramming environment speedups by factors of 336 and 35, network respectively, to i486 66 MHz and 150 MHz Alpha AXP A236 Parallel DSP Module It contains the A236 paralM DSP It implements any learning 160M MAC/s in 16-bit, it impleand 1 to 8 M B RAM with 2ns algorithm ments processing and pattern recogaccess time nition N010 Parallel DSP Module It contains the N010 parallel DSP It implements any learning 640M MAC/s in 16-bit, it impleand 1 to 16MB RAM with l n s algorithm ments processing and pattern recogaccess time nition lntel 80170NX Analogue chip with 64 inputs, 16 No on-chip 8 gs propagation time for two-layer, biases, 64 hidden, 64 output equivalent to 2 billion multiply/ accumulates/s R N-200 Pulse-stream strategy, 7 bits On-chip Stand-alone

256 neurones, technology PGA On-chip learning unit with 68 pin Contains 1-D systolic array of 4 On-chip cells in SIMD, 32-bit floating point Eight internal neural circuits No on-chip CMOS VLSI, 16-bit resolution Pseudo-log representation of hum- No on-chip bers. 32 digital neurones operating in parallel Combination of neural network Software learning and fuzzy logic

b a s i s f u n c t i o n a l g o r i t h m s . I t is c o m p o s e d o f ( i ) a classifier, (ii) a t r a d i t i o n a l m i c r o c o n t r o l l e r , a n d (iii) a n e x t e r n a l i n t e r f a c e . T h e N i l 0 0 0 is p a r t i c u l a r l y a t t r a c t i v e t h a n k s t o its m i x o f h a r d w i r e d

computation in the highly parallel and time-consuming part of the process and the programmable, high versatility in the learning process. An other c h i p w i t h s i m i l a r f e a t u r e s is t h e I B M Z I S C 0 3 6 .

104

P. Daponte, D. Grimaldi / Measurement 23 (1998) 93 115

Controller

I,
tnpu~ ~

I,

'
] pr(~'c~'mllN~{

Controll~ bt~

l p/~x~c~Is~ l l p r ~

]
'~

I ,,o,,,,,,,
RAM

I I
~

"~

I no,,,,~
RAM

I
/

I~P~
~-'~,

I
7~t ~

I ~
1,1~ut section . . . . . . .

I I m~o~y

I m~m~

Proee~in8 section a)

Output ~-'etion

Host bus Controller ]

t ~ ~ ~ ~ ~
n~osuc
q'

~on~o,,erb~

Input section

Processingsection

Output section

b)
Host bus

I
mputs ....~ ~ .............. ti

Con~o,,~
~ ~

I
~ ~outputs

>
,

s~tcy

I I
~

I I
Si~nold~

r------7
r

>

I
,

IT

T~rTT

Ill

II
.

Input ~etion

Processingsection

Output section

c)
Fig. 3. Block diagram of (a) digital, (b) analogue, (c) mixed signal, implementation of ANNs.

P. Daponte, D. Grimaldi / Measurement 23 (1998) 93 115

105

An alternative solution that significantly reduces the silicon area without introducing performance degradation has been adopted to realize the architecture of the T O T E M + + chip. By using a number representation based on the pseudo-log function (PLG), in this chip the costly multipliers have been replaced by smaller adders. The resulting architecture requires less silicon area. The cost of this is (i) the need for binary to PLG and PLG to binary converters, and (ii) the reduction of computational accuracy [47]. Another strategy uses a more sophisticated tool based on an array of DSPs. The increased availability of very high speed DSP chips has suggested their utilization in the implementation of ANNs. These execute, in a parallel way, the same operation on different values and communication is through proper input and output busses [48-50]. Furthermore, in the neural network structures, two basic procedures should be performed: (i) activation of the neurones by means of the output signals from the preceding layer through the vector of synapse weights, and (ii) calculation of the neurone output signal from the current layer. Both operations may be performed recursively. In fulfilling the described procedure, based on two simple instructions with high parallelism, the DSP is a very interesting processing tool to be programmed as a neural network. In Table 4, the neuro chips MM32k, A236 Parallel DSP Module and N010 Parallel DSP Module are examples of neuro chips based on an array of DSPs. In particular, the MM32k is a nice example of good software integration of an accelerator board composed of a massively parallel array of processing nodes. Each processing node is composed of three 1-bit registers, a serial arithmetic and logic unit and 512 bits of memory. The instruction set has 17 different operations and can implement any logic function. It is a typical example of a processing system.
5.2. Analogue implementation

The analogue neural networks are the most natural images of the neural cells of the human brain. Fig. 3b shows a general analogue structure for implementation of the ANNs. The most commonly utilized device is the transistor, and parasitic capacitors and resistors can also be found. Voltage

and current are utilized in order to store the neural network weights and to carry out all the neural network operations. Their extremely small variations then produce measurable changes in the state of the device. Such a system is then continuous and strongly non-linear. A first advantage is that analogue implementation requires less silicon area than similar digital neural networks. Indeed, a single transistor can perform the multiplication and the addition can be performed by means of a simple wire, when held at a fixed potential, typically ground, in contrast to the digital adder which handles multiple inputs which simultaneously require many transistors. A second advantage is low power consumption as a consequence of the absence of switches, other than for conveying the changing inputs. Others advantages are linked to the massive parallelism that can be provided in analogue implementation and to the easy realization of non-linear functions. Moreover, ANN analogue implementations do not require an ADC section and can be directly connected to the output of the analogue sensors. The most serious disadvantage is the realization of a non-volatile memory capacity which is necessary for programmable synapse weights. This may require the use of a more expensive non-standard processing technology, e.g. floating gates. The synapse weights are either programmable by the EEPROM cells, as shown in Fig. 3b, or by the external resistor network. These latter are hard to build on chips. A problem is the long time taken in adjusting each weight. In any case, this obstacle is apprehended by the learning procedure performed on an external digital system and the successive transfer of the weights obtained into the analogue neural network. Moreover for the EEPROM memories their technology does not allow attainment of the precision and stability of digital memories. This disadvantage limits the complexity of the neural structure which could be implemented on the analogue chip [6] and the architecture's reconfigurability may also be difficult to achieve. Finally, temperature and internal and external noise are serious problems. In Table 4, the neuro chip Intel 80170NX is the only example of analogue implementation [51 ].

106

P. Daponte, D. Grimaldi / Measurement 23 (1998) 93 115

5.3. Mixed-signal implementation

To overcome the main difficulties of either a purely digital or a purely analogue realization of the ANN, several cost-effective analogue-digital circuits (mixed signal) have been proposed, aiming at taking advantage of the best features of both approaches. Fig. 3c shows a general mixed-signal structure for implementation of the ANNs. Depending on the adopted architecture, and the circuitry hardware, the proposed analogue-digital synapses possess rather different performance characteristics with respect to weight dynamic range, weight storage and retention capacity, as well as the signal range for linear operation of the multiplier [48]. For example, Fig. 3c depicts a synapse circuit in which the pulse-stream modulation strategy for weight storage is used. This system achieves higher precision but has a longer response time [52]. Contemporary development of the customer's digital VLSI circuits does not enable the making of a large network in one circuit. Moreover, multipliers require a large area and it is difficult to implement many neurones on one chip. A trade-off between neurone circuit complexity and computational accuracy, aimed at obtaining a high packing density in an integrated circuit that is well suited for implementation using the VLSI technology, has been achieved by utilizing the random-pulse machine concept [3,53]. The architecture obtained is modular and has neurones constructed by using simple digital circuits. In Table 4, the neuro chip RN-200 is an example of implementation based on pulse-stream strategy. In this way, it is possible to implement multiplication, addition and activation functions in an easy manner in the chip. The RN-200 is a chip dedicated to multilayer perceptron application on the board and implements the back-propagation learning algorithm.

6. ANN application examples in the field of measurement using software

One of the most interesting applications of the ANNs using software implementation is that

devoted to modelling the devices of measurement systems [54] such as sensors, transducers, filters, amplifiers, and ADCs and DACs. These are characterized by transfer functions which can be complex and/or hardly non-linear or piece-wise linear. By referring to the block diagram sketched in Fig. 4, these devices can be classified as: (i) analogue devices, (ii) mixed analogue/digital devices and (iii) digital devices. In order to build suitable ANN-based models, the mixed analogue/digital devices, such as the ADCs and DACs, show the most interest. As a matter of fact, in this case the neural model must represent the discontinuities and the dynamic behaviour. For this reason, attention is focused only on mixed analogue/digital device types and, in the following, the principles for building mixed measurement device neural models is reported. A very complex learning phase must be performed. In fact, even if the ANN neurone transfer function is non-linear (sigmoidal or Gaussian), it is very difficult to force the ANN into furnishing a global transfer function characterized by a "stair" shape. A suitable strategy, therefore, in performing the learning phase can be adopted, as a variation of the learning algorithm characteristic coefficients as a function of the learning progress and/or of the neurone transfer function shape [38]. Moreover, it is also possible to modify the standard backpropagation learning algorithm in order to improve its efficiency in terms of learning speed and ANN output accuracy by adopting one of the suggestions in the available literature (Momentum and adaptive learning rate, Levenberg-Marquardt or Gauss Newton approximation) [8] or by developing other specialized strategies. As far as the learning and test sets are concerned, they can be constituted by a sequence of input/output values obtained from the device to be modelled in correspondence to a ramp signal. This ramp must be chosen such that more analogue values are considered for each digital code, in order to allow the ANN to correctly reproduce the conversion process. In order to evaluate the neural model's performance by means of the usual tests for ADCs and DACs in the time and frequency domains, the

P. Daponte, D. Grimaldi / Measurement 23 (1998) 93 115

107

S I G N A L T O BE _~ MEASURED " 1

MEASUREMENT DEVICE - slmsor II - transducer

A.,,.OO

I > ~"XeD OOIOm'TAL AW' K

MEASUREMENT DEVICE - ADC - DAC

MEASURm4ENT DEVICE . D$P " P' P - FPGA

DIGITAL

MEASURE

Fig. 4. Block diagram of a measurement system.

validation set can be obtained by considering a suitable sine-wave signal. The sine-wave to be chosen must have a peak-to-peak amplitude which corresponds to the device full range, with a number of points and periods which will permit the above mentioned tests to be well applied. The ANNs can be simulated in a digital environment by means of the appropriate software, as shown in Tables 1 and 2, such as ExploreNet 3000 by HNC and the Matlab Neural Toolbox. In the following, however, the building of an ANN-based software model utilized a specific software developed in Turbo-Pascal, and running on a Pentium(~) PC. As the device under test, an ADC with 6-bit resolution, +5.12V input range and a 33 MHz maximum sampling frequency was chosen and adopted. The ANN has one input node, 6 output nodes and one hidden layer with 80 nodes. The learning phase was performed by using a ramp with +_5.12 V of amplitude as the input signal. Fig. 5a shows the output error at the end of the learning phase, after about 600,000 iterations. For the validation phase, a sine-wave, characterized by an amplitude equal to + 5.12 V, and a frequency equal to 3662.1 Hz, was used. The sampiing frequency was 1 MHz, in order to obtain 15 signal periods and 4096 signal samples. Fig. 5b shows the output signal spectra obtained from the actual device and Fig. 5c shows the output signal spectra obtained from the neural model in the validation phase. The capability of the adopted ANN to reproduce the non-linearities of the ADC to be modelled is evident. In particular, the time-domain test shows very small differences between the neural model and the ADC and in the frequency-domain test, the corresponding output signal spectra are very similar.

LSB
1

a)

lJlLIl'ulllllll,iLllLJ,
-1 4.0 -4.0 -2.0 0 2.0 4.0 Vin [V]

-40 ~ 480 -120

g . . . .

-80 -120 0 10 20 30 40 50 f[kHz Fig. 5. (a) Difference between the ADC neural model and the ADC under test transfer functions at the end of the learning phase; (b) ADC under test output signal spectra: (c) ADC neural model output signal spectrum.

7. Examples of application in the measurement field using an analogue hardware implementation of

ANNs In spite of the continuing increase in the computing power of both general purpose and specialized microprocessors, analogue measurement signal processing is required in some fields (e.g. acoustic -optic signal processing) [55]. In other cases, e.g. that of very high frequency, digital solutions cannot be developed. Furthermore, analogue processing electronics play a fundamental role in enhancing the performance of advanced sensors and transducers. To this end, a method for the design of neural

108

P. Daponte, D. Grimaldi / Measurement 23 (1998) 93 115

networks suitable for analogue processing and the generation of signals in measurement instrumentation [37] has been proposed. A wide description of the method, its applicability, examples including a library of neural blocks, some complex processing schemes, and their resulting theoretical and experimental uncertainties can be found in Refs. [35,37]. As an example, Fig. 6a shows a component of the library of neural blocks that performs the summing operation y = w l x l + w 2 x 2. Fig. 6b depicts another basic neural substructure of the library, the multiplying block. It can be implemented by a symmetrical connection of four neurones. The availability of analogue hardware solutions of ANNs has prompted testing of the applicability of the proposed design method in real cases. As is shown in Table 4, the electrical trainable analogue neural network (ETANN) Intel 80170NX [51,56] represents the only trainable neural network product available on the market with: (i) a structure resembling that of the theoretiXle X2~ ~ W 2 Yl
a)

X0

Xl; i
X2 ~

w2

-w~

~s Ws
~ Wl O ,~ ~y

N
Fig. 6. a) Neurone connection to implement a summing block; (b) neuron connection to implement a multiplying block.

cal neural network and (ii) which is customer programmable. An outline of its architecture is depicted in Fig. 7a. Each chip contains 64-buffered analogue inputs, 64 neurones and 10,240 modifiable synapses organized in two processing arrays: a 64 x 64 synapse connecting the input with the output neurones (input synapse array) and a 64 x 64 synapse connecting the neurone outputs with the input (feedback synapse array). The input and feedback arrays are accompanied by both a 16 x 64 bias array and a sample and hold block (S/H). By clocking the feedback S/H block many interesting processing options can be performed. Fig. 7b shows the ETANN structure in which the synapses are realized as a field of admittances connecting the M horizontal free available input lines (wires) with the N vertical lines representing the neurone inputs. Other K horizontal lines are connected to the bias voltage in order to shift the bias point of the neurones into the required position through use of the connecting admittances. The main ETANN characteristics are: (i) highspeed processing, 3 las for each neurone layer and 2 billion multiply-accumulate operations per second, (ii) simple circuit structure exploiting the parallelism of neural structures, and (iii) much smaller basic analogue cells than those used in a hardware realized neural network operating in the digital mode. However, the precision and stability of the chip components is only moderate. Several experiments were conducted using the ETANN chip in order to test the processing schemes obtained by the method. The example given here is devoted to the design of an analogue neural network suitable for processing the output signal of a sensor. This analogue neural network comprises an analogue pre-processing block and a neural ADC. It can be utilized to allow integration of the sensor systems and of the signal processing circuits on a single substrate. In this way, a more compact, self-contained sensor system is obtained. Moreover, it is both cheaper to manufacture and more easily integrated than the alternative multi-chip system. This approach was tested on a sensor made up

P. Dapon te, D. Grimaldi / Measurement 23 (1998) 93 I 15

109

I~t4 Bias Kbi~~

bll

Sample&Hold

~--

{:
Fee~w:isyr~se n y

>
" i
v N otnlmt ~ m

llllly

J Wll. ,.

}._
/dr.N ar~y

b'

$amplc & Hokl

Clock J

Fig. 7. (a) Block diagram of the architecture of the neuro chip Intel 80170NX able to effect different configurations of analogue neural networks; (b) the synapses are realized as a field of admittance.

of a full bridge of strain gauges, shown in Fig. 8a, to compensate for the parasitic quantity represented by influence of the temperature, which was measured by another bridge, with resistors installed in the block (dashed block), with a stable reference temperature. In the hypothesis that the output voltages (ul, u2) are independent, the correct values of both inputs e (deformation) and T (temperature), including the effects of both amplifiers A1 and A2, are [15]
T = To + d T = To + - ul dl
U2 -- a 2

8. Examples of application in the measurement field using a digital hardware implementation of ANNs

(1)
el-aid1 ca

b2 e=e o=de . . . . u~ dT bl b~ dT--bl

bl

Eq. (1) are characterized by simple algebraic operations and are easy to implement by means of ANNs utilizing the realized library. Fig. 8b shows the final pre-processing structure obtained. Temperature influence on the strain gauge bridge has been tested by using the ETANN chip and Fig. 8c shows the influence correction.

An important aspect in the monitoring of dynamic systems is the fast measurement of the actual signal parameters. The most common approach is based on the use of the proper estimating algorithm implemented into hardware architectures utilizing a DSP [57]. In this field a non-intrusive measurement apparatus for the real-time detection of the tone of the dual-tone multi-frequency signal (DTMF), by using a hardware neural network implemented into a DSP has been proposed [27]. The board architecture is shown in Fig. 9. The hardware tool utilized in implementing the neural network is an ISA-BUS compatible board on which the DSP-TMS320C30, by Texas Instruments [58], is installed. The machine cycle is 60ns. The data acquisition unit (AU) is equipped with an ADC with 14 bits and a maximum sampling frequency of 19.2 kHz. In order to accelerate the training procedure for a large training set, the neural network learning

110

P. Daponte, D. Grimaldi / Measurement 23 (1998) 93 115

. . . . . . .

a)

b)

Ul

O2

[Vl
0051 0

015

o~

4).03

4302

4301

0~I

00~

003

004

C)

~;XIO "3

Fig. 8. (a) Sensor made up of full bridge of strain gauges, preprocessing block and ADC; (b) optimized analogue neural network for executing the sensor's pre-processing operations; (c) thermal influence correction on strain gauge bridge.

procedure is performed by a software programme which runs on a personal computer (PC). This p r o g r a m m e is able to simulate a three-layer neural network: the input layer, the hidden layer and the output layer. For the input and the hidden layers, the p r o g r a m m e introduces a bias neurone, which receives a constant input equal to one. During the learning procedure it is possible to verify the value of two different types of error: (i) the maximum absolute error ( M A E ) , which corresponds to the greater error for each learning epoch, and (ii) the mean square error (MSE), which returns the greatest amongst the errors of each neurone output. The neural measurement apparatus utilizes two programmes written in C-language with some subroutines in assembly language for communication between the DSP and the PC. The first p r o g r a m m e runs on the PC, working as a slave with respect to the DSP (master). This p r o g r a m m e is dedicated to the transfer of data between the user and the DSP. The second p r o g r a m m e runs on the DSP and is dedicated (i) to behaving as a feed-forward neural network, and (ii) to controlling the AU in order to acquire real signals from the external environment. The values of the synapse weights, obtained by software simulation, are stored in a static memory available in the DSP and organized in a mono-dimensional vector, each one for a proper neurone. In order to save m e m o r y space, all these vectors are joined into one global vector. The borders between the different vectors are marked by a pointer. It is possible, in this manner, to flexibly implement all three layers of the neural network layout for the different neural network structures. This approach is faster than using the classical weights matrix. The weight values are transferred from the PC to the DSP in D M A by the ISA-BUS. All p r o g r a m m e d weights are converted into a 16-bit level for data exchange across a bus and converted back into a real number in order to compute the DSP response for the input vector. A dynamic intermediate vector is used with the aim of storing the synapse weights in a heap memory. This vector provides the serial-to-parallel conversion of the weights received from the PC by a D M A procedure. The second part of the algorithm running on

P. Daponte, D. Grimaldi / Measurement 23 (1998) 93 115

111

P|agging DSP board

Telephone network

j ooo,
j

AU

.BI i=i

Fig. 9. Block diagram of the board architecture of the DSP-based apparatus for real-time DTMF detection.

the DSP is developed to control both acquisition and the conversion hardware, to take signals from the external sources and to pass the real samples to the input of the neural network. The above described neural apparatus was utilized to carry out the tones of the DTMF signal. According to AT&T standards [58], each tone is the sum of two harmonic signals with constant amplitude and different assigned frequencies. Each frequency combination, therefore, corresponds to one phone code number and represents one tone. In order to organize the training set of the ANN, the output corresponding to each tone is constituted by a binary word with 12 bits. Table 5 shows the correspondence between the phone key, column 1, the corresponding frequencies of the
Table 5 Frequency combination defining the twelve classes of the DTM F keypad Key Signal frequency Signal frequency Expected

#1 (Hz)
1 2 3 4 5 6 7 8 9 * 0 # 697 697 697 770 770 770 852 852 941 941 941 941

#2 (Hz)
1209 1336 1477 1209 1336 1477 1209 1366 1477 1209 1366 1477 000000000001 000000000010 000000000100 000000001000 000000010000 000000100000 000001000000 000010000000 000100000000 001000000000 010000000000 100000000000

two harmonic signals, columns 2 and 3, and the associated binary word, column 4. The ANN utilized in the measurement apparatus is trained to classify the actual DTMF phone signal into 12 classes, each one characterized by a value of the binary word with 12 bits. Every input vector consists of 20 real signal samples. The neural network architecture incorporates 20 input neurones and one bias node, 80 hidden layer neurones and 12 neurones of the output layer. The starting point used in the training set was a maximum of the sum of two harmonic signals. Any decision procedure starts with the acquisition of information by the AU into its memory buffer and it then saves the starting point amongst the acquired samples and the 19 successive samples. The proposed neural measurement apparatus was tested against real signals from the communication network. The metrological characterization was performed by evaluating the sensitivity to quantization errors of the AU. To this end, the acquired signals were corrupted by reducing the conversion bit number. In Fig. 10, the complete output of the neural measurement apparatus versus different tones, for a 4-bit ADC, Fig. 10a, and a 5-bit ADC, Fig. 10b, respectively, are shown. Each bar corresponding to each key or tone represents the expected ideal output, denoted by i, and the real output, denoted by r. The neural network's working lower bound is obtained by 5-bit ADC modality. Under these operating conditions, all the real and ideal outputs coincide. The time response, which includes both

112

P. Daponte, D. Grimaldi /' Measurement23 (1998) 93 115

relative decimal value

a)
binary output

t
-0 rir l

-1 relative 0,s decimal 0,6 value

0,4 0,2

0 #
binary "~ output

b)

"" r

Fig. 10. (a) Complete binary output of the neural measurement apparatus versus different tones (keys) for a 4-bit ADC of the AU; (b) complete binary ouput of the neural measurement apparatus versus different tones (keys) for a 5-bit ADC of the AU. The bar denoted by i represents the expected ideal output, the bar denoted by r represents the real output.

the AU's time and the processing time of the neural network, is about 14.25 ms. In Fig. 11, this time response is compared with the AT&T standards [59] and the Texas Instruments (TI) proposal [59].

9. Conclusions

This paper has presented a review of the possible applications of ANNs in the field of measurements together with sufficient references to both the software and hardware A N N implementations

available to enable those interested to progress in their utilization in the field of measurements. There is no doubt that the A N N will find a use in the solving of many problems in both instrumentation and measurement applications, and also in the case of complex and noisy ill-conditioned systems. The A N N can simplify and accomplish many tasks not feasible using conventional techniques. As a matter of fact, software implementation is arriving at a level of maturity. Indeed, a number of freeware, shareware and commercial packages with different neural architectures and learning

P. Daponte, D. Grimaldi / Measurement 23 (1998) 93 115

2

I I3

lt

t' t,.ll,,li It'tIJ

)1 l[
40

J
i

a)

and more general software and hardware ANN architectures has been highlighted.

-2 ")1 I1 II I ' ) [ 0 20

i
t

Acknowledgements
The authors wish to thank Professors A. Bernieri, F. Cennamo and L. Michaeli for the collaboration given in the development of this research.

60

80

[ms]lO0

TI proposal
Neural apparatus

Acquisition time Processing time

b)
l , 0 ) 2O
i b

40

60

80

[ms]1O0

Fig. 11. (a) Time window of the signal according to the AT&T proposal; (b) comparison between the time response of the Texas Instruments (IT) proposal [59] and the proposed neural measurement apparatus.

References
[1] P.K. Simpson, Foundation of neural networks. 1EEE Technology UPDATE SERIES Neural Networks Theory, Technology, and Applications, New York, 1996, pp. 1-22. [2] W.B. Hudson, Introduction and overview of artificial neural networks. In: Instrumentation and Measurement Application, IEEE Technology UPDATE SERIES Neural Networks Theory, Technology, and Applications, New York, 1996, pp. 746-749. [3] E.M. Petriu, K. Watanabe, T.H. Yeap, Application of random-pulse machine concept to neural network design, IEEE Trans. on Instrumentation and Measurement, April 1996, pp. 665 669, [4] A.F. Armitage, Neural networks in measurement and control, Measurement and Control 28 (1995) 208-215. [5] W. Miller, T. McKenna, C. Lau, Office of Naval Research contributions to neural networks and signal processing in oceanic engineering, IEEE Journal of Oceanic Engineering 4 (1992) 299-307. [6] A. Bernieri, M. Dapuzzo, L. Sansone, M. Savastano, A neural network approach for identification and diagnosis on dynamic systems, IEEE Transaction on Instrumentation and Measurement 43 (1994) 867 873. [7] A. Maren, C. Harston, R. Pap, Handbook of Neural Computing Applications, Academic Press, 1990, New York. [8] J.A. Freeman, D.N. Skapura, Neural Networks: Algorithms, Applications and Programming Techniques, Addison Wesley, Reading, MA, 1992. [9] R.H. Nilsen, Neurocomputing, Addison Wesley, Reading, MA, 1990. [10] M. Attari, M.H. Heniche, F. Boudjema, A two dimensional intelligent calibration of an ion sensor, Proceeding of I EEE Instrumentation and Measurement and I M EKO Technical Conference, Brussels, Belgium, June 4 6, 1996, pp. 788 791. [11] M. Johnson, Neural networks in advanced instrument design, Measurement + Control 29 (1996) 101 - 105. [12] A. Carullo, F. Ferraris, S. Graziani, U. Grimaldi, M. Parvis, Ultrasonic distance sensor improvement using a two-level neural network, IEEE Transaction on Instrumentation and Measurement, April 1996, pp. 677 682.

algorithms are already available. Moreover, the software is just beginning to acquire the versatility required by users and also a complexity sufficient to justify the use of special hardware. Hardware implementation still requires some fine tuning and further researches, in particular, are needed regarding analogue implementation. Conversely, digital implementation has reached a level of versatility that makes it both useful and competitive with regard to many real problems if compared with traditional techniques. It should be noted that many ANN applications in the measurement field are oriented solely toward the use of software solutions, whereas the use of ANN hardware implementation is less common. This is mainly due to the non-availability of on-chip ANNs dedicated to the processing of measurement signals. However, the use of ANNs implemented on-chip could furnish ulterior easy and quick solutions in some fields, such as measurements in telecommunications, and the monitoring of complex non-linear systems and/or devices. Investigation of state-of-the-art software and hardware has provided information which will facilitate choosing from what is available when it comes to real applications. Examples have been given for each software and hardware implementation with an emphasis on those applications where commercially available systems may be utilized. Moreover, the actual trend towards both new

114

P. Daponte, D. Grimaldi / Measurement 23 (1998) 93 115

[13] K. Zhang, C. Butler, Q. Yang, Y. Lu, A fiber optic sensor for the measurement of surface roughness and displacement using artificial neural networks, Proceeding of IEEE Instrumentation and Measurement and IMEKO Technical Conference, Brussels, Belgium, June 4 6, 1996, pp. 917 920. [14] P. Daponte, D. Grimaldi, L. Michaeli, Analog neural networks for pre-processing in measurement sensors, XIV IMEKO World Congress, Vol. IV, Tampere, Finland, 1 6 June, 1997, pp. 56 61. [15] J.W. Gardner, E.L. Hines, C. Pang, Detection of vapours and odours from a multisensor array using pattern recognition: self-organising adaptive resonance techniques, Measurement and Control 29 (1996) 172 178. [16] Yi Liu, Calibrating an industrial microwave six-port instrument using the artificial neural network technique, IEEE Transaction on Instrumentation and Measurement, April 1996, pp. 651 656. [17] L. Ricci, G. Tecchioli, A neural system for frequency control of tunable laser source, Processing Industry, Proceedings of IEEE Instrumentation and Measurement Technical Conference, Ottawa, Canada, May 19 21, 1997, pp. 251 256. [18] H. Eren, C.C. Fung, K.W. Wong, An application of artificial neural network for prediction of densities and particle size distributions in mineral, Processing Industry, Proceedings of IEEE Instrumentation and Measurement Technical Conference, Ottawa, Canada, May 19 21, 1997, pp. 1118-1121. [19] C.S. Liao, H.T. Lin, F.D. Chu, J.L. Shi, C.L. Ho, The feasibility of applying the neural network technology to the automatic time report system, Proceedings of 1EEE Instrumentation and Measurement Technical Conference, Ottawa, Canada, May 19 21, 1997, pp. 1122 1125. [20] P. Daponte, D. Grimaldi, A. Piccolo, D. Villacci, A neural diagnostic system for the monitoring of transformer heating, Measurement 18 (1996) 35-46. [21] 1. Kamwa, R. Grondin, V.K. Sood, C. Gagnon, V.T. Nguyen, J. Mereb, Recurrent neural networks for phasor detection and adaptive identification in power system control and protection, IEEE Transaction on Instrumentation and Measurement, April 1996, 657-664. [22] M. Caciotta, R. Lamedica, E. Mocenigo, V. Orsolini Cencelli, A. Prudenzi, M. Sforna, Application of artificial neural network to historical data analysis for short time forecasting of Italian electrical load, European Transaction Electrical Power Engineering. [23] A. Bernieri, G. Betta, A. Pietrosanto, C. Sansone, A neural network approach to instrument fault detection and isolation, IEEE Transaction on Instrumentation and Measurement 44 (1995) 747 750. [24] A. Bernieri, G. Betta, C. Liguori, A. Losi, Neural networks and pseudo-measurements for real-time monitoring of distribution systems, IEEE Transaction on Instrumentation and Measurement, April 1996, pp. 645-650. [25] D. Barschdorff, Neural networks and fuzzy logic: new concepts for technical failure diagnosis, Proceedings of the

[26]

[27]

[28]

[29]

[30]

[31]

[32]

[33]

[34]

[35]

[36]

[37]

[38]

[39]

[40]

IMEKO Word Congress, Torino, Italy, September 5 9, 1994, pp. 130 137. J. Ma, J.E. Amadi-Echendu, Y. Yan, Non redundancy based approach to validate a measuring device, Proceedings of IEEE Instrumentation and Measurement Technical Conference, Ottawa, Canada, May 19 21, 1997, pp. 540 543. D. Grimaldi, L. Michaeli, A. Palumbo, A DSP based neural apparatus for signal parameters measurement, AMSE-1SIS'97, Reggio Calabria, Italy, September 11 13, 1997. D. Grimaldi, L. Michaeli, J. Saliga, Neural based twodimensional signal quantisers implemented on DSP TMS320C30, Digital Signal Processing DSP'97, Herl'any, Slovak Republic, September 3 4, 1997. L. Michaeli, J. Saliga, A two-dimensional quantiser for signal recognition in instrumentation, Technical Conference, University of Kosicc, Slovak Republic, 1996. G.A. Carpenter, S. Grossberg, D.B. Rosen, ART-2A: an adaptive resonance algorithm for rapid category learning and recognition, Neural Networks 4 (1991) 494 496. Y. Sukuki, Self-organizing QRS-wave recognition in ECG using neural networks, IEEE Transaction on Neural Networks 6 (1995) 1472 1478. D.W. Tank, J. Hopfield, Simple neural optimization networks: an A/D converter signal, decision circuit, and a linear programming circuit, IEEE Trans. CAS 33 (1986) 533 541. G. Martinelli, R. Perfetti, Synthesis of feedforward neural analogue digital converters, IEE Proc.-G 138 (1991) 567 574. F. Cennamo, P. Daponte, D. Grimaldi, E. Loizzo, A new neural A/D converter architecture, Measurement 16 (1995) 177 186. P. Daponte, D. Grimaldi, L. Michaeli, A full neural ADC Gray code based, IEEE Transaction on Instrumentation and Measurement 45 (1996) 634 639. A. Baccigalupi, A. Bernieri, C. Liguori, Error compensation of A/D converters using neural networks, 1EEE Transaction on Instrumentation and Measurement, April 1996, pp. 640 644. P. Daponte, D. Grimaldi, L. Michaeli, A design method for signal processing in measurement instrumentation by neural networks, Proc. IEEE Instrumentation and Measurement Technology Conference, Brussels, Belgium, June 4 6, 1996, pp. 1004-1009. A. Bernieri, P. Daponte, D. Grimaldi, ADC neural modeling, IEEE Transaction on Instrumentation and Measurement, April 1996, pp. 627--633. A. Bernieri, P. Daponte, D. Grimaldi, Accurate neural model identification of measurement devices, Measurement, to be published. M. Attari, F. Boudjema, M. Bouhedda, S. Bouallag, A decentralized neural architecture based A/D converter with binary code outputs, Proceedings of IEEE Instrumentation and Measurement Technical Conference, Ottawa, Canada, May 19 21, 1997, pp. 232-236.

P. Daponte, D. Grimaldi / Measurement 23 (1998) 93 115

115

[41] X.Z. Gao, X.M. Gao, S.J. Ovaska, A/D converter resolution enhancement using neural networks, Proceedings of IEEE Instrumentation and Measurement Technical Conference, Ottawa, Canada, May 19 21, 1997, pp. 1112 1117. [42] P. Wilke, C. Jacob, The NeuroGraph neural network simulator, IEEE Technology UPDATE SERIES Neural Networks Theory, Technology, and Applications, New York, 1996, pp. 931 932. [43] C. Tietz, A. Linden, T. Sudbrak, T., SESAME - an object oriented software tool for design, simulation and analysis of neural nets, IEEE Technology UPDATE SERIES Neural Networks Theory, Technology, and Applications, New York, 1996, pp. 933 934. [44] A. Zell, N. Mache, M. Vogt, M. Huttel, Problems of massive parallelism in neural network simulation, IEEE Technology UPDATE SERIES Neural Networks Theory, Technology, and Applications, New York, 1996, pp. 923 928. [45] R.P. Malhotra, R. Siferd, Evaluation of electronic artilicial neural network implementation, IEEE Technology LIPDATE SERIES Neural Networks Theory, Technology, and Applications, New York, 1996, pp. 857-863. [46] P. lenne, T. Cornu, G. Kuhn, Special-purpose digital hardware for neural networks: an architectural survey, Journal of VLSI Signal Processing Systems 13 (1996) 5 25. Kluver Academic Publishers [47] 1. kazzizzera, P. Lee, A. Sartori, G. Tecchiolli, A. Zorat, Fast and compact controllers with digital neural networks, Proceedings of IEEE Instrumentation and Measurement Technical Conference, Ottawa, Canada, May 19 21, 1997, pp. 226 231.

[48] J.M. Zurada, Artificial Neural Systems, West Publishing Company, St. Paul, MN, 1992. [49] Nestor Inc., Nestor Nil000 Data Sheet, October 1994. [50] R. Sridhar, Y.Ch. Shin, VLSI neural network architectures, IEEE Technology UPDATE SERIES Neural Networks Theory, Technology, and Applications, New York, 1996, pp. 864 873. [51] Intel Corporation, Intel 80170 NX ETANN, Data Sheet, Or. No. 290408-003, March 1993. [52] A.F. Murray, S. Churcher, A. Hamilton, A.J. Holmes, G.B. Jackson, H.M. Reekie, R.J. Woodbum Pulse stream VLSI neural networks, IEEE MICRO, June 1994, pp. 29 39. [53] A. Hamilton, A.F~ Murray, D.J. Baxter, S. Churcher, H.M, Reekie, L. Tarasenko, Integrated pulse stream neural networks: results, issues, and pointers, IEEE Transaction on Neural Networks 3 (1992) 385-393. [54] C.B. Umminger, C.G. Sodini, An integrated analog sensor for automatic alignment, IEEE J. of Solid-State Circuits 30 (1995) 1382 1390. [55] J. Brauch, S.M. Tam, M.A. Holler, A.L. Shmurun, Analog VLSI neural networks for impact signal processing, IEEE Micro, 1992, pp. 34-45. [56] A. Bernieri, G. Betta, C. Liguori, On-line detection and diagnosis obtained by implementing neural algorithms on a digital signal processor, IEEE Transaction on Instrumentation and Measurement 45 (1996) 894 899. [57] Texas Instruments, TMS320C3x User's Guide, 1994. [58] Telecom Circuits Data Book, Literature #SCTD001A, Texas Instruments Inc., 1987. [59] P. Mock, Add DTMF generation and decoding to DSP-gP designs, Digital Signal Processing Applications with the TMS320 Family, Prentice Hall, Englewood Cliffs, N J, 1987.