ARTICLE IN PRESS

Neurocomputing 69 (2006) 1860–1867

www.elsevier.com/locate/neucom

Mixed-signal neuron-synapse implementation for large-scale

neural network
Il Song Han
Department of Electronic and Electrical Engineering, University of Sheffield, Mappin Street, Sheffield, South Yorkshire S1 3JD, UK
Received 28 February 2005; received in revised form 7 August 2005; accepted 22 November 2005
Available online 14 June 2006

Abstract

This paper describes a mixed-signal neural networks VLSI for low power and asynchronous operation. The voltage-controlled
transconductance produces the synaptic function of multiplication and summation of synaptic currents for neuron, by compensating the
non-linearity of MOSFET resistance in the triode region.
The flexible configuration of synapse accommodates the spike-based neural networks, inspired by the biological plausibility and low
power requirement. The neuron with a combination of synapses demonstrates asynchronous spikes of integration-and-firing with a
refractory period. The speed of individual synapse is simulated up to 300 Mega operations/s with the power consumption of less than
33 mW, using 0.18 mm CMOS.
r 2006 Elsevier B.V. All rights reserved.

Keywords: Analogue-mixed VLSI neural network; Pulse/spike-based neural computation; Asynchronous operation; MOSFET resistance; Voltage-
controlled linear resistance

1. Introduction cognition [19]. Hence, the current demand in large capacity

VLSI neural networks have been continuously developed
either in digital or analogue, as both methods have
different advantages. The advantage of analogue VLSI is
low power consumption or larger networks integration,
though digital VLSI has advantages of design flexibility or
leading-edge technology. Issues in an analogue neural
network VLSI can be the accuracy problem, or the
complexity, in comparison with the digital. In some special
applications, the analogue utilised or developed better its
non-ideal characteristic or complex design [11,15].
Recently, biologically inspired neural networks, i.e.
spike-based operation is widely investigated in various
areas, from robots to regenerative medicine [3,5,6,16,21].
The pulse or spike based implementation of neural
networks has advantages of VLSI neural networks, which
is suitable for the large-scale real-time or embedded
requirement [4,8]. In the brain science, a large scale and
general neural network VLSI has been also expected for
Tel.: +44 (0)114 222 5810; fax: +44 114 222 5146.
E-mail address: i.s.han@sheffield.ac.uk.
or biological performance drives analogue-mixed neural
networks VLSI of pulse/spike-based operation, because of
its advantages in large-scale implementation and low
power consumption in comparison to the digital. There
exists an issue of accuracy improvement in analogue-mixed
implementation, with technology advancement towards
0.18 or 0.13 mm. An example of analogue-mixed VLSI is in
Fig. 1, which was developed for real-time packet control
[9]. Though the circuit exhibits the accuracy, it demands
the complex supply voltages which limit the level of
integration under low supply voltages of up-to-date VLSI
technology.
This paper introduces the new development in analogue-
mixed neural networks VLSI with advantages of up-to-date
advanced technology and fully asynchronous spike operation.
2. Analogue-mixed neural network synapses for low power
and asynchronous operation
The current–voltage (I–V) relationship of MOSFET in
the triode region can be adopted to implement electronic
synapses, as it provides the multiplication function.

0925-2312/$ - see front matter r 2006 Elsevier B.V. All rights reserved.
doi:10.1016/j.neucom.2005.11.013
 ARTICLE IN PRESS
I.S. Han / Neurocomputing 69 (2006) 1860–1867 1861

(a) (b)

Fig. 1. (a) Electronic synapse circuit with compensated linear MOSFET resistance in the triode region, and (b) its measured linearity of output synapse
current vs. weight voltage (710 mA with 70.5 V).

(a) (b)

Fig. 2. (a) New synapse circuit by voltage-controlled linear resistance of two MOSFETs in the triode region, and (b) its representation as differential input
and multiplying transconductor.

The equation of interest is that the drain-source current
IDS for a MOSFET in the linear or triode region:
I DS ¼ afðV GS
V T ÞV DS
V 2DS =2g.
Here, a is the MOSFET process parameter including the
geometry, VGS, VDS, VT, the transistor gate-source, drain-
source and threshold voltage, respectively. To achieve a
linear voltage (VGS)-to-voltage (VDS) multiplier, the
balanced circuit with an operational amplifier can be used
to remove the second order term of V2DS/2 in Eq. (1).
Though various methods for suppressing or controlling the
nonlinearity have been developed, there are limitations in
solving the nonlinear problems. The synapse circuit of Fig.
1 is an effective method to compensate such nonlinearity,
but it still has the drawback of demanding bipolar supply
voltages to remove the second-order term in Eq. (1).
The synapse circuit proposed in this paper makes also a
simple use of the MOSFET in the triode region. It realises
the high speed and a small size analogue multiplier without
an amplifier or dual supply voltages. The synapse circuit of
new MOSFET resistance-based analogue multiplier is
shown in Fig. 2(a). For an efficient analogue multi-
plication, two terms, VT?VDS and V2DS/2, from Eq. (1)
(1) should be eliminated from the output. Based on Eq. (1) for
the MOSFET in the triode region, the currents of transistor
M1 and transistor M2 are
I M1 ¼ afðV 2

V T ÞV 1
V 21 =2g, (2)
I M2 ¼ afðV 2þ
V T ÞV 1
V 21 =2g. (3)
Here, V2+ and V2
produce one input of two variables
for the multiplication and voltages of V2+ and V2
keep
transistors M1 and M2 in the triode region, i.e. both M1
and M2 are operated under the condition of
VGS
VT4VDS. V1 is the other input of multiplication,
which represents the drain-source voltage of M1 and M2.
The source voltages of both M1 and M2 remain in
common as M3 or M4 acts as a diode. With the help of
current mirrors (M3–M8 and M4–M7–M5–M6), the
synaptic output current is the difference of IM1 and IM2
 ARTICLE IN PRESS
1862 I.S. Han / Neurocomputing 69 (2006) 1860–1867

in Eqs. of (2) and (3):
I OUT ¼ I M1
I M2
¼ aðV 2þ
V 2
ÞV 1
¼ a V WEIGHT V 1 ,
where VWEIGHT is the difference of V2+ and V2
. One of
V2+ and V2
can be a DC reference voltage while the other
one is a synaptic weight plus DC reference. From Eq. (4),
the synaptic multiplication of synapse weight (VWEIGHT)
and effective neural input (V1) is achieved by two
MOSFETs operated in the triode region and pairs of
current mirrors.
The summation of post-synaptic current is attained by
common-output connection of synapses as each synapse
can contribute individual synaptic output current. Both of
MOSFET M6 and M8 act as current source by either
sourcing or sinking the synaptic output current, and the
summation of post-synaptic current is computed by a
integration capacitor in each neuron.
The new synapse of Fig. 2(a) designed using 0.18 mm
standard CMOS technology and evaluated by HSPICE
simulation. The power supply voltage is 3.3 V and both
accurate operation and low power consumption are design
objectives. The linearity of synapse circuit is shown in
Fig. 3(a) and demonstrates the behaviour of multiplier in
ð4Þ Eq. (4). One of inputs to M1 and M2 is applied with
sinusoidal signal with DC offset, while the other is applied
with the same DC offset. The output current illustrates the
analogue multiplication with amplitude modulation by
neural input of triangular signal, though the neural input
signal of binary state is enough for general purpose pulse/
spike neural networks. The functional description of
synapse in Fig. 2(b) presents the general output character-
istic of Fig. 3(a), as a synapse can be used as an
element for complex neural signal processing. The transis-
tor sizes used for the simulated output in Fig. 3(a) are
(W/L)M1 ¼ (W/L)M1 ¼ 0.4m/0.6m, (W/L)M3 ¼ (W/L)M4 ¼
(W/L)M7 ¼ (W/L)M8 ¼ 1.6m/0.4m, and (W/L)M5 ¼
(W/L)M6 ¼ 0.4m/0.4m.
The speed of more than 300 Mega connection per second
inputs is simulated as in Fig. 3(b). The flexibility in power
consumption can increase the operation speed further, as it
accelerates charging or discharging rate of output current

Fig. 3. (a) New synapse circuit’s characteristics as a multiplier, a neural input voltage (triangular wave) and synaptic weight voltage (sinusoidal wave) for
inputs and modulated current for synaptic output current, and (b) transient characteristics of pulses/spikes operation over 300 Mega connections per
second for synapse cell.
 ARTICLE IN PRESS
I.S. Han / Neurocomputing 69 (2006) 1860–1867 1863

faster. The overall power is less than 33 mW per activated

synapse cell and the new circuit is suitable for large scale
VLSI neural network implementation. As observed in the
circuit of Fig. 2(a), there is no current or power
consumption without any active neural input. Without
neural input, V1, IM1 and IM2 become null and there flows
no current in M1–M3 or M2–M4. Also, there flows no
current in M5–M7 or M6–M8, as the source current of
current mirror is null. Therefore, the synapse cell does not
consume any power if there is no activity. The operational
principle in Eq. (4) does not require any timing character-
istics of neural inputs, and the synapse circuit of Fig. 2(a)
operates either asynchronously or synchronously, free from
any synchronous constraints. The power consumption is
dependant on the activity of neural input or pulse/spike
firing rates of neural networks.

Fig. 5. Measured characteristics of the test circuit in Fig. 4, where two

3. The experimentation of a proposed multiplier circuit with 0.5VPP sinusoidal voltage sources are applied as synapse weight and neural
input. The output waveform is measured at the output node of the circuit
conventional CMOS devices
in Fig. 4.

The basic operation characteristics of new synapse

circuit in Fig. 2(a) is tested and measured with the discrete
CMOS transistor circuit in Fig. 4. The experimentation
circuit is implemented without any additional elements,
other than M1A and M2A. M1A and M2A are added to
improve the design constraints from the pre-determined
discrete transistors. The NMOS transistors of ALD1116
and PMOS transistors of ALD1117 are used to measure
the characteristics of the circuit in Fig. 4. For conductive
elements of (M1, M1A) and (M2, M2A) in the triode
region, PMOS transistors are used to implement the larger
dynamic range by simulating lower transconductance.
The measured characteristics in Fig. 5 demonstrated the
circuit in Fig. 2(a) or Fig. 4 as the synapse circuit or
multiplier circuit, where the synapse weight was applied
with the 0.5VPP of the sinusoidal waveform and the neural
input was applied with 0.5VPP of the sinusoidal waveform
with the offset voltage of 1.1 V. The measured character-
istics in Fig. 5 shows the 2 QAM function as in Fig. 3(a),
though the test circuit was based on only two fixed
transistors instead of three designed ones.
Fig. 4. The experimentation circuit of Fig. 2(a). PMOS transistors M1,
M1A, M2, M2A, M5 and M6 are ALD1117, and NMOS transistors M3,
M4, M7, and M8 are ALD1116.
4. Analogue-mixed and bio-inspired neurons for VLSI
implementation
There are different requirements for the VLSI imple-
mentation of neurons depending on applications, i.e.
asynchronous pulse/spike or synchronous ones. Recent
developments of biologically inspired neural networks or
neuromorphic solutions are largely based on asynchronous
operation, while certain applications like ‘real-time packet
switch controller’ are based on synchronous operation [9].
Either asynchronous or synchronous neuron is available to
the synapse of Fig. 2(a) for mixed-signal neuron-synapse
implementation of large-scale neural network.
4.1. Synchronous neuron
For analogue or analogue-mixed neural network opera-
tion, the neural state is represented as the voltage at a
neuron capacitor connected to networked synapse outputs
of Fig. 2(a). Every synapse produces the current by
multiplication of weight value and input value based on
Eq. (4), which results a voltage of neural state by the
integration of synaptic current in a neuron capacitor.
The circuit in Fig. 6(a) shows the block diagram of a
neuron with the linear ramp function and sigmoid-like
continuality. At each sampling with CLK1 ‘high’ in
Fig. 6(a), the summation of the synaptic computation as
a buffered input voltage is fed to a capacitor for copying
the neural state into the neuron. The CLK2 signal operates
on that sampled voltage, which is then transformed into the
new voltage level for a sigmoid-like function enabled by
two MOSFETs in the diode configuration. Two MOSFET
as in Fig. 6(a) are for defining the dynamic range of neural
states, where one is for the upper bound conversion and the
other is for the lower bound conversion. With the reduced
 ARTICLE IN PRESS
1864 I.S. Han / Neurocomputing 69 (2006) 1860–1867
(a)
Fig. 6. (a) Analogue-mixed neuron processing circuit and (b) its transfer characteristics.
IDISCHARGE
(a)
Fig. 7. (a) Pulse/spike based synchronous neuron, and (b) its chip photograph of a mixed-signal neuron-synapse VLSI.
gate-to-source voltage during the operation, the sub-
threshold conduction delivers the overall conversion close
to a continuous sigmoid function as shown in the transfer
curve of Fig. 6(b). The transformed voltage as a neural
state is applied to the following part of neuron circuit while
CLK3 is ‘high’.
In order to provide the pulse/spike output, the dischar-
ging current is subtracted from the neural state of a
sampling capacitor Cs when the comparator CON output
of Fig. 7(a) turns the switch MOSFET M2 ‘on’. The output
pulse from a comparator is generated when the voltage at
the sampling capacitor is higher than the provided
reference level (REF). An AND gate in Fig. 7(a) encodes
the pulse/spike output with system reference clock CLK
and PWM output of comparator CON. A MOSFET
switch M1 controls the synchronous operation of neuron
output. The neuron circuit does not always include the
transformation block in Fig. 6(a), because many applica-
tions demand only binary neuron output.
As a neuron circuit in Fig. 7(a) is not based on any
amplifying circuit but only a comparator, diodes, or
switches, the total operation speed of neural networks
does not degrade the overall performance based on
analogue-digital mixed synapse operation of Fig. 2(a).
The complexity of neuron introduces unlikely any issue in
synapse-neuron integration as illustrated in Fig. 7(b) of
neuron-synapse chip photograph, where the sampling
(b)
(b)
GK
Cmembrane Gleak GNa
Eleak ENa EK
Fig. 8. An electrical equivalent circuit of a neuron.
capacitor Cs of holding neural state occupies larger portion
of neuron area. The synchronised operation is required for
real-time synchronous applications like a high-speed
telecommunication switch controller.
4.2. Asynchronous neuron
The biologically motivated neuron has been targeted as a
neuron model for its advantages of integrating various
applications [1,2,12,13,14,18]. The feature of asynchronous
firing or spike dynamics emerges as a key aspect, because
such electrical signals seem to be essential to the neural
information processing in biological unit. The Hodgkin–
Huxley (H–H) formalism is widely adopted for its biophy-
sical characterization and dynamics. An electrical equivalent
circuit model of Fig. 8 is known as an empirical model by
the H–H formalism, which describes quantitatively the
 ARTICLE IN PRESS
I.S. Han / Neurocomputing 69 (2006) 1860–1867
KA
INTEGRATOR
+ SIGMOID
X +
EL
GL
ENA
Fig. 9. A H–H based neuron block diagram based on functions of synapse in Fig. 2.
dynamics of the voltage-dependent conductance. The
asynchronous neural spikes inspire all sorts of neuro-
morphic implementation, although most of particular
recognition tasks do not exhibit any major advantages
based on H–H formalism. Asynchronous neuron spikes or
pulses are even considered as a key element in high level
cognition [20]. Hence, asynchronous dynamics of the H–H
formalism is adopted as a reference model for asynchronous
neuron compatible to the synapse of Fig. 2(a).
The voltage–current relationship of Eq. (4) can imple-
ment the voltage-dependant conductance employed in
Fig. 7. An empirical mathematical formalism models
dynamics of each conductance element as
G ion ¼ Gion max x,
dx=dt ¼ aðb
xÞ,
iion ¼ Gion ðV m
E ion Þ,
where b is sigmoidal function of the membrane potential.
Vm is a membrane potential and the overall dynamic
modelled by an action potential and related ionic
conductance.
The block diagram of asynchronous neuron in Fig. 9 is
inspired by controlled conductance from H–H model
and sum-multiplier of spike-based synapse in Fig. 2. The
differential equation in Eq. (5) is implemented by first-
order low-pass filter, which induces a delayed response.
The neuron of Fig. 9 models three components of ionic
conductance, where KA, ENA, and EL represent EK, ENA,
Eleak respectively and GL as Gleak in Fig. 8. The capacitor of
Cmembrane is modelled as an integrator for its functional
behaviour.
The MATLAB simulation of H–H neuron block
diagram in Fig. 9 shows the result in Fig. 10(a),
which exhibits matching behaviour to H–H model in
Fig. 10(b).
The H–H formalism inspired neuron, which is based on
the conductance controlled model of Fig. 9 is used to
1865
+
LOWPASS
FILTER
X +
MEMBRANE
POTETNTAL LOWPASS
FILTER X
+
Fig. 10. (a) Simulation of action potential, and (b) an experimental action
potential from Ref [10].
implement a asynchronous silicon neuron in Fig. 11.
A further simplification is introduced by an approximation
ð5Þ of sigmoidal function and a reduction of one differentiated
conductance stage. A differential amplifier with buffer
stages (Diff amp in Fig. 11) acts as a sigmoidal transfer
function for the steady-state activation variable, where a
reference voltage is introduced for the half-activation
potential. Two synapses of SYNAPSE2 and SYNAPSE3
in Fig. 11(a) realise the approximation of two voltage-
controlled conductances for the smaller chip area without
major drawbacks.
A low-pass filter is implemented by one synapse of
SYNAPSE1 and a capacitor CL, as a synapse of Fig. 2 also
represents an operational transconductance amplifier from
Eq. (4). A transconoductance amplifier acts as a resistor
or an equivalent conductance of the RC low-pass filter,
where the resistance R can be programmed by a control
voltage (SETbias in Fig. 11). The control voltage (SETbias)
corresponding to VWEIGHT of Eq. (4) can be used to
control the bandwidth of a low-pass filter, otherwise
remaining as a fixed one. The linearity of low pass filter
is
45 dB in harmonic distortion and tuneable up to 50%
in bandwidth by a control voltage, from the simulated
result of the circuit in Fig. 2(a).

1866
Eleak
SETleak
Cm
amp
Diff
Membrane
potential Vref
parator
Com-
Vthres
(a)
Fig. 11. (a) Asynchronous spike firing neuron by three synapses of Fig. 2, inspired by H–H model, and (b) asynchronous behaviour of a neuron circuit
with synaptic spike currents as inputs: (from top to down) neuron capacitor’s potential as a membrane potential, synaptic current spikes as input, and
firing pulses with the refractory period.
The neuron of Fig. 11(a) shows a membrane dynamic
behaviour consistent to the simulated result in Fig. 10(a).
Instead of single stimulus as a single firing in Fig. 10, the
current spike stream is applied to the simulation of
asynchronous neuron as in Fig. 11(b). The 2 Mega spike
stimulus per second is simulated as in Fig. 11(b), though
elements of synapse circuit for a neuron can operate in
much higher speed as in Fig. 3. The HSPICE simulation
result demonstrates the asynchronous behaviour of inte-
gration-and-firing with a refractory period. There are
advantages of asynchronous operation, removal of refer-
ence clocks, and low voltage operation compared to
previous pulse-based analogue-mixed neural networks
VLSI [7]. Another advantage of H–H formalism inspired
silicon neuron in Fig. 11(a) is its fast operation speed in
Fig. 11(b), which may demand substantial computing
otherwise [17].
5. Conclusion
The new spike-based synapse circuit and neuron circuit
demonstrate the feasibility of large scale, low power and
asynchronous neuron-synapse VLSI implementation, for
its simpler circuit, speed and biological plausibility based
on the controlled conductance. It is flexible to adapt its
characteristics of speed, power, accuracy on demand, as
those can be tailored by controlling the operating synapse
current and parallel wired-OR expansion. For an example,
the higher operation speed, or the fast settlement is
observed with increased operational current levels in a
synapse element. The overall power consumption can be
less in practice, as individual synapse only demands the
power when there is an active neural input signal.
The core of proposed electronic synapse and neuron are
estimated, respectively, as within the size of 2 mm  2 mm
and 6 mm  6 mm in 0.18 mm CMOS VLSI technology,
though the overall neuron area is finalised by the desired
average firing frequency. The asynchronous spike based
neural networks in advanced CMOS VLSI technology
provides the advantage of analogue-digital mixed neural
ARTICLE IN PRESS
I.S. Han / Neurocomputing 69 (2006) 1860–1867
Ebias
CL
SETbias
(b)
networks VLSI with small power consumption and no need
for a synchronous operation.
References
[1] T. Asai, et al., A subthreshold MOS neuron circuit based on the
Volterra system, IEEE Trans. Neural Networks 14 (5) (2003)
1308–1312.
[2] C. Bartolozzi, G. Indiveri, A neuromorphic selective attention
architecture with dynamic synapses and integrate-and fire neurons,
in: Proceedings of BICS, Sterling, BIS2.2, 2004.
[3] G. Bugmann, Biologically plausible neural computation, Biosystems
40 (1997) 11–19.
[4] C. Christodoulou, G. Bugmann, T. Clarkson, A spiking neuron
model: applications and learning, Neural Networks 15 (2002)
891–908.
[5] R. Eckmiller, O. Baruth, D. Neumann, Neural information proces-
sing efforts to restore vision in the blind, in: Proceedings of ICONIP,
Lecture Notes in Computer Science, No. 3316, Springer, Berlin, 2004,
pp. 10–18.
[6] D. Floreano, C. Mattiussi, Evolution of spiking neural controllers for
autonomous vision-based robots, ER, Lecture Notes in Computer
Science, No. 2217, Springer, Berlin, 2001, pp. 38–61.
[7] I.S. Han, Neural network VLSI implementation and its applications,
Korea Telecom J. 2 (1) (1997) 12–20.
[8] I.S. Han, Mixed-signal neuron-synapse implementation for large
scale neural network, in: Proceedings of BICS Sterling, NC4.2, 2004.
[9] I.S. Han, R. Webb, Neural network Switch controller with analogue-
digital mixed neural network VLSI, in: Proceedings of the EANN’97,
1997, pp. 299–302.
[10] M. Hausser, The Hodgkin–Huxley theory of action potential, Nat.
Neurosci. Suppl. 3 (2000) 1165.
[11] Y. Horio, K. Aihara, O. Yamamoto, Neuron-synapse IC chip-set for
large-scale chaotic neural networks, IEEE Trans. Neural Networks
14 (5) (2003) 1393–1404.
[12] E.M. Izhikevich, Which model to use for cortical spiking neurons?
IEEE Trans. Neural Networks 15 (5) (2004) 1063–1070.
[13] B. Linares-Barranco, et al., A CMOS implementation of FizHugh–
Nagumo neuron model, IEEE JSSC 26 (7) (1991) 956–965.
[14] S. Matinoia, et al., Cultured neurons coupled to micoelectrode
arrays: circuit models, simulations and experimental data, IEEE
Trans. Biomed. Eng. 51 (5) (2004) 859–864.
[15] M. Milev, M. Hristov, Analog implementation of ANN with inherent
quadratic nonlinearity of the synapses, IEEE Trans. Neural Net-
works 14 (5) (2003) 1187–1200.
[16] T. Pearce, et al., Silicon-based neuromorphic olfactory pathway
implementation, in: Proceedings of BICS, Sterling, BIS 6.3, 2004.
 ARTICLE IN PRESS
I.S. Han / Neurocomputing 69 (2006) 1860–1867 1867

[17] U. Seiffert, Artificial neural networks on massively parallel computer developed the large scale pulse-based neural network VLSI. Since 2002,
hardware, Neurocomputing 57 (2004) 135–150. he has been with the department of Electronic and Electrical Engineering,
[18] M. Simoni, et al., A multiconductance silicon neuron with University of Sheffield, Sheffield, UK.
biologically matched dynamics, IEEE Trans. Biomed. Eng. 51 (2) His research interests include analogue-mixed CMOS VLSI systems for
(2004) 342–354. neural networks and telecommunication applications.
[19] J. Taylor, Paying attention to consciousness, Progr. Neurobiol. 71
(2003) 305–335.
[20] J. Taylor, Private communication, 2004.
[21] N. Taylor, M Hartley, J. Taylor, The learning of insertions by the
Cerebellum, in: Proceedings of BICS, Sterling, CNS 2.2, 2004.

Il Song Han received the B.Sc. degree in

Electronic Engineering from Seoul National
University in 1979, the M.Sc. degree and the
Ph.D. degree in Electrical Engineering from
Korea Advanced Institute of Science and Tech-
nology, Korea, in 1981 and 1984, respectively. He
received the MBA degree from Cranfield Uni-
versity, Cranfield, UK, in 2000.
He worked for 14 years as a researcher in
telecommunication R&D industry, where he