ANALYTICAL PROBABILITY DENSITY FUNCTIONS FOR LMS ADAPTIVE FILTERS USING THE FOKKER-PLANCK EQUATION

D10.14

S. T. Alexander Department of Electrical and Computer Engineering North Carolina State University Raleigh, NC 27695-7911.

Virginia L. Stonick Department of Electrical and Computer Engineering Carnegie Mellon University Pittsburgh, PA 15213-3890

ABSTRACT
Analytical expressions for the theoretical probability. density function (pdf) of the LMS adaptive filter weights are obtained for the steady state. The LMS update is formulated as a stochastic differential equation and the weight pdf is next derived as the solution to a second order partial differential equation known as the Fokker-Planck equation. The application of systems identification is then examined and closed form solutions are obtained for the steady-state probability density function for the LMS weights.

are found for a systems identification problem, although the theory developed can also be applied to finding the time-varying pdfs during the adaptation phase. Many continuous domain optimization problems can be solved by the continuous domain LMS algorithm. The LMS update for continuous time is the first order vector differential equation [3,4]:

w(t) = a e ( t ) x ( t )

(1)

I. INTRODUCTION
The Least Mean Squares (LMS) technique has received substantial investigation due to its simplicity, both in discrete-time implementations [1,2] and analog or continuous-time implementations [3-51. One of the reasons for the simplicity of LMS is that it uses a simple approximation to the gradient to update its filter weights. This approximation, sometimes called the noisy gradient, thus introduces a variance or jitter into the LMS weight adaptation process and this jitter is present even at convergence. Consequently, the LMS filter weights are a stochastic process, having a timevarying probability density function (pdf) during the adaptation phase and a stationary pdf after the filter has converged. Many applications using adaptive filters require a knowledge of this weight pdf in order to compute decision boundaries and probabilities of errors in forming these decisions. Previous work in [SI has determined accurate mathematical forms for converged LMS weight pdfs by working entirely within the discrete time domain and using a characteristic function approach to the stochastic problem. The current paper develops an alternative method for obtaining the LMS weight pdfs by using a continuous time domain method to find analytical solutions for the weight pdfs. The analytical pdfs are found by solving the partial differential equation known as Fokker-Planck equation. In this initial work, only the steady state pdfs This research funded in part by the National Science Foundation, Grant No. MIP-8552571.

where the dot notation signifies the first derivative with respect to time. In (l),a is the feedback gain parameter, e ( t ) is the error signal, x ( t ) is the N-length vector of input signals
x ( t ) = [Zl(t),
**e,

zlV(t)lT

(2)

and w(t) is the N-length vector of continuous weights

w(t) = [ W l ( t ) ,

a..,

WN(t)lT

(3)

In ( l ) , the error signal e(t) is computed by

e(t) = T(t)

- xT(t)w(t)

(4)

where ~ ( t is)denbted as the reference signal. Substituting (4) into (1) gives the update differential equation

W ( t )= - a x ( t ) x T ( t ) w ( t ) + ( Y T ( t ) X ( t )

(5)

Equation (5) is a vector stochastic differential equation. Previous analysis of (5) has usually assumed w(t) and z ( t ) to be uncorrelated so that taking expectations in (5) allows the expected value of w(t) to be obtained from an ordinary differential equation with constant coefficients [3,4]. While this approach provides mathematical simplicity, information only about first and second order statistics results. However, retaining the stochastic form of (5) allows theoretical solution for the entire pdf of the weights by considering a partial differential equation associated with (5). This approach was taken in [5] in the context of an adaptive array analysis of steady-state antenna properties. This partial differential equation is known as the Fokker-Planck equation, and it is described next.

- 2117 -

CH2977-719110000-2117 $1.00 1991 IEEE

@

M, =

- 2118 -

without introducing a large amount of error. Therefore, as in [5], let

z Z ( t ) M U: g(t) (18) To accurately utilize the model in equation (18) the standard deviation, u g , of g ( t ) should be small enough so that the right hand side of (18) is rarely negative, but large enough so that the approximation (18) approaches zero from time to time, which corresponds to the zero crossings of the original z ( t ) . A value of

where 6 ( . ) is the dirac delta. Integrating (24) over T and X over the range specified in (23) and using (19) thus evaluates the first double integral in (23):

(19)
achieves this trade-off well. Substituting (18) into (17) and taking the required expectations then gives

Next, the second double integral in (23) evaluates to zero since n(t) is zero mean and uncorrelated with z ( t ) . The third double integral in (23) is evaluated by recognizing that, since z ( t ) and n(t) are uncorrelated, the expectation gives

E{n(X)n(T)z(X)z(T)dTdX =u;~(T - A)

+u ; ~ ( T - A)

E{Awlw} = a [ A - w ( t ) ] u f A t

(20)

where the t dependence on the left hand side has been surpressed for simplicity. Finally, performing the limiting operation required in the definition (14) then gives

(26) Using (26) the third double integration in (23) is then evaluated as

Mi

zz

(YuZ[A -~ ( t ) ]

(21)

(27) Finally, substituting (25) and (27) into (23) then gives

Lt I=,

t+At

t+At

E{n(A)n(T)z(X)z(T)dTdX = u;u;At

The second order conditional moment is computed similarly. Using (16) and the method for computing the square of an integral first gives

E { [ A w ] ~ w= } ( a [ A- w(t)]u:At)

+ l4a [ A - w(t)]u:At + au;u:At

Mi1 = 1 a 2 [ A- ~(t)]01+ CYU~U;
4

(28)

[Aw(t)] =
CY

where, again, the t dependence of the left hand side has been surpressed. Substituting (28) into the definition (9) / / [ A - w(X)][A - w ( ~ ) ] z ~ ( X ) z ( ~ ) d ~ d X for the second order conditional moment then gives
(29)

+ CY J J [ A - w ( A ) ] ~ ( A ) ~ ( T ) z ( T ) ~ T ~ X + a

Jn(X)n(r)z(X)z(T)dTdX

(22)

where in (22) and remaining integrals

J J = L I=,
As in (17), next take expectations of both sides of (22) while holding w(t) fixed, giving

t+At

t+At

In general, the expressions (21) and (29) may be substituted into (13) and the time-varying weight pdf s the solution to (13). However, an analytical found a l l t is beyond the scope of solution to (13) for p ( w ,t ) for a the current paper. For this initial investigation a great deal of information may be obtained by examining the steady state pdf for the converged weight as t + W.
Steady State

PDF

E{[Aw(t)llw(t)}=
a 2 [ A- w(t)] //E{z(A)z(r)}drdX

In steady state the partial derivative of p(w,t)with respect to t is zero and (13) becomes an ordinary differential equation:

+ a 2 [-~ w(t)j J J E{zZ(X)n(T)z(T)dTdX + a 2 J J E { n ( X ) n ( r ) 5 ( X ) z ( T ) d 7 d x (23)

The first double integral in (23) is evaluated by substituting (18) for z(t). Taking the resulting expectation and using the property that g ( t ) is assumed uncorrelated gives E{~(A)Z(T)) = u;6(r - A) (24)

where p ( w ) in (30) is the pdf of the steady state weight and M I and A411 are, in general, functions of w. Equation (30) is equivalent to

( ~ 2+ )~

where c1 is a constant of integration, as can be verified by differentiating (31) with respect to w. Taking the

2119

2f29