Digital Signal Processing: Mathematical and Computational Methods, Software Development and Applications

Digital signal processing

Mathematical and Computational Methods, Software Development and Applications

Second Edition

Jonathan M. Blackledget†

Professor of Information and Communications Technology, Department of Electronic and Electrical Engineering, Loughbourough University, England.

Horwood Publishing, Chichester,

West Sussex, England

†Professor of Computer Science, Department of Computer Science, University of the Western Cape, Cape Town, South Africa.

Table of Contents

Cover image

Title page

Copyright page

Digital Signal Processing

Dedication

About the Author

Foreword to the Second Edition

Preface to the Second Edition

Acknowledgements

Notation

Glossary of Terms

Introduction

Part I: Signal Analysis

Chapter 1: Complex Analysis

1.1 Introduction

1.2 Complex Numbers

1.3 Complex Functions

1.4 Complex Integration

1.5 Cauchy’s Residue Theorem

1.6 Summary of Important Results

1.8 Problems

Chapter 2: The Delta Function

2.1 Some Important Generalized Function

2.2 The Delta Function

2.3 Properties of the δ-function

2.4 Derivatives of the δ-function

2.5 Integration Involving Delta Functions

2.6 Convolution

2.7 The Green’s Function

2.8 Summary of Important Results

2.10 Problems

Chapter 3: The Fourier Series

3.1 Derivation of the Fourier Series

3.2 The Half Range Fourier Series

3.3 Fourier Series for an Arbitrary Period

3.4 Applications of the Fourier Series to Circuit Theory

3.5 The Complex Fourier Series

3.6 The Fourier Transform Pair

3.7 The Discrete Fourier Transform

3.8 Relationship between the DFT and the Fourier Transform

3.9 ‘Standard’ and ‘Optical’ Forms of the DFT

3.10 Summary of Important Results

3.12 Problems

Chapter 4: The Fourier Transform

4.1 Introduction

4.2 Selected but Important Functions

4.3 Selected but Important Theorems

4.4 Convolution and Correlation

4.5 The Sampling Theorem

4.6 Fourier Filters

4.7 The Kronecker Delta Function and the DFT

4.8 Deriving Filters using the DFT

4.9 Case Study: Scattering from Layered Media

4.10 Summary of Important Results

4.12 Problems

Chapter 5: Other Integral Transforms

5.1 The Laplace Transform

5.2 The Sine and Cosine Transforms

5.3 The Walsh Transform

5.4 The Cepstral Transform

5.5 The Hilbert Transform

5.6 Case Study: FM Imaging by Real Zero Conversion

5.7 STFT and The Gabor Transform

5.8 The Wigner and the Wigner-Ville Transforms

5.9 The Riemann-Liouville and the Wyle Transforms

5.10 The z-Transform

5.11 The Wavelet Transform

5.12 Discussion

5.13 Summary of Important Results

5.15 Problems

Part II: Computational Linear Algebra

Chapter 6: Matrices and Matrix Algebra

6.1 Matrices and Matrix Algebra

6.2 Systems of Linear Algebraic Equations

6.3 Linear Systems

6.4 Summary of Important Results

6.6 Problems

Chapter 7: Direct Methods of Solution

7.1 Gauss’ Method

7.2 Jordan’s Method

7.3 LU Factorization

7.4 Banded Systems

7.5 Computational Considerations

7.6 Solution to Complex Systems of Equations

7.7 Summary of Important Results

7.9 Problems

Chapter 8: Vector and Matrix Norms

8.1 Vector Norms

8.2 The Triangle Inequality for the Euclidean Norm

8.3 Matrix Narms

8.4 The Conditioning of Linear Equations

8.5 Iterative Improvement

8.6 The Least Squares Method

8.7 Summary of Important Results

8.9 Problems

Chapter 9: Iterative Methods of Solution

9.1 Basic Ideas

9.2 Iterative Methods

9.3 Example of the Iteration Method

9.4 General Formalism

9.5 Criterion for Convergence of Iterative Methods

9.6 The Conjugate Gradient Method

9.7 Summary of Important Results

9.9 Problems

Chapter 10: Eigenvalues and Eigenvectors

10.1 Formal Methods of Solution

10.2 Properties of Eigenvalues

10.3 The Cayley-Hamilton Theorem

10.4 The Power Method

10.5 Jacobi’s Method for Symmetric Matrices

10.6 Sturm Sequence Iteration

10.7 LR and QR Methods

10.8 Inverse Iteration

10.9 Special Types of Matrices

10.12 Problems

Part III: Programming and Software Engineering

Chapter 11: Principles of Software Engineering

11.1 Introduction

11.2 Decimal and Binary Number Systems

11.3 Binary Number Systems

11.4 Fixed Point Storage and Overflow

11.5 Floating Point Representation of Binary Numbers

11.6 Numerical Error and Accuracy

11.7 Methods that Reduce Errors and Maintain Accuracy

11.8 Program Structures

11.9 Procedures

11.10 Processes Specification

11.11 Program Specification

11.12 System Design

11.13 Requirement Specifications

11.14 Modularity

11.15 The Jackson Method

11.16 Data Analysis

11.17 Data Flow Design

11.18 Testing and Implementation

11.19 Stages of System Design

11.20 Computer Aided Software Engineering Tools

11.21 Operating Systems and Languages

11.22 Programming Languages

11.23 Object Oriented Programming

11.24 Discussion

11.25 Summary of Important Results

11.27 Problems

Chapter 12: Modular Programming in C

12.1 About C

12.2 Modular and Structured Programming in C

12.3 Modularity

12.4 Module Size

12.5 Structured Programming

12.6 Modular Programming using Borland Turbo C++

12.7 On Style and Presentation

Discussion

12.8 Summary of Important Results

12.10 Problems

Part IV: DSP: Methods, Algorithms and Building a Library

Chapter 13: Digital Filters and the FFT

13.1 Digital Filters

13.2 The Fast Fourier Transform

13.3 Data Windowing

13.4 Computing with the FFT

13.5 Discrete Convolution and Correlation

13.6 Computing the Analytic Signal

13.7 Summary of Important Results

13.9 Programming Problems

Chapter 14: Frequency Domain Filtering with Noise

14.1 Highpass, Lowpass and Bandpass Filters

14.2 The Inverse Filter

14.3 The Wiener Filter

14.4 Power Spectrum Equalization

14.5 The Matched Filter

14.6 Case Study: Watermarking using Chirp Coding

14.7 Constrained Deconvolution

14.8 Homomorphic Filtering

14.9 Noise

14.10 Noise Types

14.11 Pseudo Random Number Generation

14.12 Additive Generators

14.13 Chaos

14.14 Case Study: Cryptography using Chaos

14.15 Summary of Important Results

14.17 Programming Problems

Chapter 15: Statistics, Entropy and Extrapolation

15.1 Bayes Rule

15.2 The Maximum Likelihood Method

15.3 Maximum a Posteriori Method

15.4 The Maximum Entropy Method

15.5 Spectral Extrapolation

15.6 The Gerchberg-Papoulis Method

15.7 Application of Weighting Functions

15.8 Burg’s Maximum Entropy Method

15.9 Summary of Important Results

15.11 Problems

Chapter 16: Digital Filtering in the Time Domain

16.1 The FIR Filter

16.2 The FIR Filter and Discrete Correlation

16.3 Computing the FIR filter

16.4 The IIR Filter

16.5 Non-Stationary Problems

16.6 Summary of Important Results

16.8 Programming Problems

Chapter 17: Random Fractal Signals

17.1 Introduction

17.2 Stochastic Modelling Revisited

17.3 Fractional Calculus

17.4 Non-stationary Fractional Dynamic Model

17.5 Green’s Function Solution

17.6 Digital Algorithms

17.7 Non-stationary Algorithm

17.8 General Stochastic Model

17.9 Case Study: Fractal Modulation

17.10 Case Study: Financial Signal Processing

17.11 Introduction

17.12 The Efficient Market Hypothesis

17.13 Market Analysis

17.14 Modelling Financial Data

17.15 Summary of Important Results

17.17 Problems

Summary

Appendix A: Solutions to Problems

A.1 Part I

A.2 Part II

A.3 Part III

A.4 Part IV

Appendix B: Graphics Utility

Index

Digital Signal Processing

Talking of education, people have now a-days (he said) got a strange opinion that every thing should be taught by lectures. Now, I cannot see that lectures can do so much good as reading the books from which the lectures are taken. I know nothing that can be best taught by lectures, except where experiments are to be shown. You may teach chymestry by lectures - You might teach making of shoes by lectures!

James Boswell: Life of Dr Samuel Johnson, 1766

Dedication

To all those students with whom I had the good fortune to work and, in using the material herein, taught me how to teach it.

About the Author

Jonathan Blackledge graduated in physics from Imperial College and music from the Royal College of Music, London, in 1980 and obtained a Doctorate in theoretical physics from the same university in 1983. He was appointed as Research Fellow of Physics at Kings College, London from 1983 to 1988 specializing in inverse problems in electromagnetism and acoustics. During this period, he worked on a number of industrial research contracts undertaking theoretical and computational work on the applications of inverse scattering theory for the analysis of signals and images.

In 1988, he joined the Applied Mathematics and Computing Group at Cranfield University as Lecturer and later, as Senior Lecturer and Head of Group where he promoted postgraduate teaching and research in applied, engineering and industrial mathematics in areas which included computer aided engineering, digital signal processing and computer graphics. In 1994, he was appointed Professor of Applied Mathematics and Computing and Head of the Department of Mathematical Sciences at De Montfort University where he established the Institute of Simulation Sciences. He is currently Professor of Computer Science in the Department of Computer Science at the University of the Western Cape, South Africa nd Professor of of Information and Communications Technology in the Department of Electronics and Electrical Engineering at Loughborough University, England. He is also a co-founder and Director of a group of companies specializing in communications technology and financial analysis based in London and New York.

Professor Blackledge has published over one hundred scientific and engineering research papers and technical reports for industry, six industrial software systems, fifteen patents, ten books and has been supervisor to sixty research (PhD) graduates. He lectures widely to a variety of audiences composed of mathematicians, computer scientists, engineers and technologists in areas that include cryptology, communications technology and the use of artificial intelligence in process engineering, financial analysis and risk management. His current research interests include computational geometry and computer graphics, image analysis, nonlinear dynamical systems modelling and computer network security, working in both an academic and commercial context. He holds Fellowships with England's leading scientific and engineering Institutes and Societies including the Institute of Physics, the Institute of Mathematics and its Applications, the Institution of Electrical Engineers, the Institution of Mechanical Engineers, the British Computer Society, the Royal Statistical Society and the Institute of Directors. He is a Chartered Physicist, Chartered Mathematician, Chartered Electrical Engineer, Chartered Mechanical Engineer, Chartered Statistician and a Chartered Information Technology Professional. He has an additional interest in music for which he holds a Fellowship of the Royal Schools of Music, London.

Copyright

HORWOOD PUBLISHING LIMITED

Coll House, Westergate, Chichester, West Sussex, P020 3QL, England.

First published in 2003 and re-printed in 2006 with corrections and additions.

COPYRIGHT NOTICE

All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopy, recording, or otherwise, without the permission of Horwood Publishing Limited, Coll House, Westergate, Chichester, West Sussex, P020 3QL, England.

©J. M. Blackledge, 2006

British Library Cataloguing in Publishing Data

A catalogue record of this book is available from the British Library.

ISBN 1-904275-26-5

Typeset and produced by the author using LaTeX, the TeXnicCenter graphical user interface and the stylefile of the Institute of Mathematics and its Applications.

Printed and bound by CPI Antony Rowe, Eastbourne

Foreword to the Second Edition

I was flattered when the publisher, Ellis Horwood asked me to write this Foreword, because my introduction to signal processing began in the Second World War when, as a communications officer in the Royal Corps of Signals, I worked with a war-time teleprinter. We used a system based on 5-bit letters and a pair of copper wires. It provided a bandwidth of about 200Hz that could be separated by filters from the rest of a voice channel without noticeably distorting the speech. Today the bandwidth available is huge by comparison and information can be conveyed through a multitude of channels using tiny glass fibres. However, although the engineering associated with information and communications technology in general has and continues to undergo radical change, many of the underlying mathematical principles remain the same.

G H Hardy in his book A Mathematician’s Apology wrote that there were ‘no interesting applications of pure mathematics’. This is no longer true and Professor Blackledge's book Digital Signal Processing will enable many people to make use of their interest in, and perhaps fascination with, mathematics in such a way, and through a field of study, that will help us all communicate our ideas more quickly and conveniently through the digital world of today.

The Earl Kitchener of Khartoum

Preface to the Second Edition

J.M. Blackledge

This book provides an account of the mathematical background, computational methods and software engineering associated with digital signal processing. The aim has been to provide the reader with the mathematical methods required for signal analysis which are then used to develop models and algorithms for processing digital signals and finally to encourage the reader to design software solutions for Digital Signal Processing (DSP). In this way, the reader is invited to develop a small DSP library that can then be expanded further with a focus on his/her research interests and applications.

There are of course many excellent books and software systems available on this subject area. However, in many of these publications, the relationship between the mathematical methods associated with signal analysis and the software available for processing data is not always clear. Either the publications concentrate on mathematical aspects that are not focused on practical programming solutions or elaborate on the software development of solutions in terms of working 'black-boxes' without covering the mathematical background and analysis associated with the design of these software solutions. Thus, this book has been written with the aim of giving the reader a technical overview of the mathematics and software associated with the 'art' of developing numerical algorithms and designing software solutions for DSP, all of which is built on firm mathematical foundations. For this reason, the work is, by necessity, rather lengthy and covers a wide range of subjects compounded in four principal parts. Part I provides the mathematical background for the analysis of signals, Part II considers the computational techniques (principally those associated with linear algebra and the linear eigenvalue problem) required for array processing and associated analysis (error analysis for example). Part III introduces the reader to the essential elements of software engineering using the C programming language, tailored to those features that are used for developing C functions or modules for building a DSP library.

The material associated with parts I, II and III is then used to build up a DSP system by defining a number of 'problems' and then addressing the solutions in terms of presenting an appropriate mathematical model, undertaking the necessary analysis, developing an appropriate algorithm and then coding the solution in C. This material forms the basis for part IV of this work.

In most chapters, a series of tutorial problems is given for the reader to attempt with answers provided in Appendix A. These problems include theoretical, computational and programming exercises. Part II of this work is relatively long and arguably contains too much material on the computational methods for linear algebra. However, this material and the complementary material on vector and matrix norms forms the computational basis for many methods of digital signal processing. Moreover, this important and widely researched subject area forms the foundations, not only of digital signal processing and control engineering for example, but also of numerical analysis in general.

The material presented in this book is based on the lecture notes and supplementary material developed by the author for an advanced Masters course 'Digital Signal Processing' which was first established at Cranfield University, Bedford in 1990 and modified when the author moved to De Montfort University, Leicester in 1994.

The programmes are still operating at these universities and the material has been used by some 700++ graduates since its establishment and development in the early 1990s. The material was enhanced and developed further when the author moved to the Department of Electronic and Electrical Engineering at Loughborough University in 2003 and now forms part of the Department’s post-graduate programmes in Communication Systems Engineering. The original Masters programme included a taught component covering a period of six months based on two semesters, each Semester being composed of four modules. The material in this work covers the first Semester and its four parts reflect the four modules delivered. The material delivered in the second Semester is published as a companion volume to this work entitled Digital Image Processing, Horwood Publishing, 2005 which covers the mathematical modelling of imaging systems and the techniques that have been developed to process and analyse the data such systems provide.

Since the publication of the first edition of this work in 2003, a number of minor changes and some additions have been made. The material on programming and software engineering in Chapters 11 and 12 has been extended. This includes some additions and further solved and supplementary questions which are included throughout the text. Nevertheless, it is worth pointing out, that while every effort has been made by the author and publisher to provide a work that is error free, it is inevitable that typing errors and various 'bugs' will occur. If so, and in particular, if the reader starts to suffer from a lack of comprehension over certain aspects of the material (due to errors or otherwise) then he/she should not assume that there is something wrong with themselves, but with the author!

January 2006

Acknowledgements

The material developed in this book has been helped and encouraged by numerous colleagues of the author over the years. The author would like to thank all of his fellow colleagues, but particular thanks go to Prof Roy Hoskins who, for many years, has played a central role in teaching aspects of signal analysis coupled with the mathematical rigour required to come to terms with such entities as the Delta function. Thanks also go to Dr Peter Sherar at Cranfield University who helped the author establish the MSc programme in ‘Software Solutions for Digital Signal Processing’ from which much of the material presented in this work has been derived. Thanks also go to Dr Martin Turner, Dr Mohammed Jaffar, Dr Martin Crane and Prof Gwynne Evans at De Montfort University who have worked with the author for many years, and as module leaders for De Montfort University's advanced MSc programme in digital signal processing, developed valuable additions to the teaching and learning materials first established by the author. Further, many students of this MSc course were the result of research programmes established by Prof B Foxon at De Montfort University with links to universities and research institutes in Russia, Poland, South Africa and the USA, to whom the author (and the students) are very grateful. The author would also like to thank Prof Peter Smith who as Head of the Department of Electronic and Electrical Engineering at Loughborough University has provided the infrastructure for the author to operate in an internationally acknowledge center of engineering excellence. Thanks also go to Dr S Datta of Loughborough University who has worked with the author for many years and provided him with many valuable recommendations, ideas and research themes. In addition, the author would like to thank all those organizations and industries that have provided funding for the development of his teaching and research activities over the years including: the Engineering and Physical Sciences Research Council, the Defense Evaluation and Research Agency, the International Science and Technology Council, British Coal, British Gas, British Petroleum, British Aerospace, Oxford Instruments, Microsharp, Marconi, Microsoft and British Intelligence. Finally, the author would like to thank all those postgraduate students (both MSc and PhD) who, over the years, have used the material in this book as part of their technical training and educational development and have provided critical and constructive appraisal from one year to the next. The material was written by the author for students undertaking advanced MSc programmes at the universities of Cranfield, De Montfort and more recently, at Loughborough University and has derived great pleasure from presenting it and developing it further as part of his teaching portfolio. In addition to the taught postgraduate programmes from which the material herein has been derived, numerous research students have worked closely with the author providing valuable insights and comments that have helped to enhance and strengthen the material.

Notation

Alphabetic

αn   Real coefficients of a Fourier cosine series

adjA   Adjoint of matrix A

A ≡ (aij)   Matrix with element at ith row and jth column

AT   Transpose of A

A−1   Inverse of A

[A|B]   Augmented matrix formed from matrices A and B

|A|  

Determinant of A

A(t)   Amplitude modulation (amplitude envelope)

A(ω)   Amplitude spectrum

bn   Real coefficients of a Fourier sine series

b  

Data vector of linear system Ax = b

chirp (t)   Unit chirp function [with complex form exp(–iαt²)]

comb(t)

cond(A)   Condition number of matrix A (=||A|| × ||A−1||)

cn   Complex coefficients of a complex Fourier series for example

C  

Capacitance or the contour followed by a path of integration in the z-plane

c  

Data vector associated with linear system x = Mx + c

D  

Fractal dimension or diagonal matrix

detA   Determinant of A (also denoted by | A |)

e  

Error vector

f(t)   Arbitrary real function - typically object function or system input

f(z)   Function of a complex variable

| f |   modulus of complex variable or function f

||f(t)||   Norm (e.g. a Euclidean norm) of a function f(t)

||fi||   Norm of an array or vector fi

||fi||2   Euclidean norm of array or vector fi

||x||p   p-norm of vector x

||x||∞   ‘Infinity’ or uniform norm of a vector x

||A||   Norm of matrix A

F(ω)   Complex spectrum of function f(t)

Fr(ω)   Real component of spectrum

Fi(ω)   Imaginary component of spectrum

Fi   Discrete complex spectrum of discrete function fi

g(t)   Arbitrary function

g(t|t0,ω)   Green’s function

H(t)   Tophat function

I  

Unit or identity matrix

Im[f]   Imaginary part of complex variable or function f

k  

Wavenumber (= 2π/λ)

L  

Lower triangular matrix

L1   Lower triangular matrix with l’s along the leading diagonal

M  

Iteration matrix associated with linear system x = Mx + c

n(t)   Noise function

ni   Discrete noise function

Ni   Noise spectrum

p(t)   Instrument function or Impulse Response Function

pi   Discrete Impulse Response Function

P(ω)   Transfer Function (Fourier transform of pi)

Pi   Discrete Transfer Function (DFT of pi

P(x)   Probability density function also denoted by Pr[x(t)]

P(a|b)   Conditional probability of obtaining a given b

P(ω)   Power spectrum (=| F(ω) |²) where F(ω) is the Fourier transform of f(t)

Pi   Discrete power spectrum

Pr(x)   Probability occurrence of x

q  

Fourier dimension

q(t)   Quadrature signal

R  

Resistance

Ri   Denotes the ith row of a matrix

Re[f]   Real part of complex variable or function f

s(t)   Real or complex (analytic) signal

si   Discrete real or complex signal

sgn(t)  

Sign function

sinc(t)   Sinc function (= sin(t) / t)

t  

Time

U  

Upper triangular matrix

U1   Upper triangular matrix with l’s along the leading diagonal

U(t)   Unit step function

u(x,t)   Solution to a partial differential equation (e.g. wavefield)

vi   Eigenvectors of linear system Avi = λivi

WAL(n, t)   Walsh function

x  

Solution vector of linear system Ax = b

xi   Eigenvectors of linear system Axi = λixi

xT   Transpose of vector x

xi   ith element of vector x

x0   Initial value

z  

Complex number of the form a + ib

z*  

Complex conjugate a – ib of a complex number a + ib

∈  

In (e.g. x ∈ [a, b) is equivalent to a ≤ x < b)

∀  

Forall (e.g. f (t)= 0, ∀t ∈ (a,b])

Greek

α  

Chirping parameter

Γ(q)   Gamma function

δ(t)   Dirac delta function

δij   Kronecker delta