Borrowed from UMD ENEE631

Spring’04

Unitary Transforms
UMCP ENEE631 Slides (created by M.Wu © 2004)

Image Transform: A Revisit

With A Coding Perspective
 Why Do Transforms?

 Fast computation
UMCP ENEE631 Slides (created by M.Wu © 2001)

– E.g., convolution vs. multiplication for filter with wide support

 Conceptual insights for various image processing
– E.g., spatial frequency info. (smooth, moderate change, fast change, etc.)
 Obtain transformed data as measurement
– E.g., blurred images, radiology images (medical and astrophysics)
– Often need inverse transform
– May need to get assistance from other transforms

 For efficient storage and transmission

– Pick a few “representatives” (basis)
– Just store/send the “contribution” from each basis
 Basic Process of Transform Coding
UMCP ENEE631 Slides (created by M.Wu © 2004)

Figure is from slides at Gonzalez/ Woods DIP book website (Chapter 8)

 1-D DFT and Vector Form
 1 N 1


Z ( k )  
N n 0
z ( n )  W nk
N

{ z(n) }  { Z(k) } 
UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

 N 1
 z ( n)  1
n, k = 0, 1, …, N-1 


N k 0
Z ( k )  W  nk
N

WN = exp{ - j2 / N }
~ complex conjugate of primitive Nth root of unity
 Vector form and interpretation for inverse transform
z = k Z(k) ak
ak = [ 1 WN-k WN-2k … WN-(N-1)k ]T /  N
 Basis vectors 
– akH = ak* T = [ 1 WNk WN2k … WN(N-1)k ] /  N

– Use akH as row vectors to construct a matrix F

– Z = F z  z = F*T Z = F* Z

– F is symmetric (FT=F) and unitary (F-1 = FH where FH = F*T ) 

 A vector space consists of a set of vectors, a
field of scalars, a vector addition operation,
Basis Vectors and Basis Images and a scalar multiplication operation.

 A basis for a vector space ~ a set of vectors

– Linearly independent ~  ai vi = 0 if and only if all ai=0
UMCP ENEE631 Slides (created by M.Wu © 2001)

– Uniquely represent every vector in the space by their linear combination

~  bi vi ( “spanning set” {vi} )
 Orthonormal basis
– Orthogonality ~ inner product <x, y> = y*T x= 0
– Normalized length ~ || x ||2 = <x, x> = x*T x= 1

 Inner product for 2-D arrays

– <F, G> = m n f(m,n) g*(m,n) = G1*T F1 (rewrite matrix into vector)
 !! Don’t do FG ~ may not even be a valid operation for MxN matrices!
 2D Basis Matrices (Basis Images) 
– Represent any images of the same size as a linear combination of basis
images
 1-D Unitary Transform

 Linear invertible transform

UMCP ENEE631 Slides (created by M.Wu © 2001)

– 1-D sequence { x(0), x(1), …, x(N-1) } as a vector

– y = A x and A is invertible

 Unitary matrix ~ A-1 = A*T

– Denote A*T as AH ~ “Hermitian”
– x = A-1 y = A*T y =  ai*T y(i)
– Hermitian of row vectors of A form a set of orthonormal basis vectors
ai*T = [a*(i,0), …, a*(i,N-1)] T

 Orthogonal matrix ~ A-1 = AT

– Real-valued unitary matrix is also an orthogonal matrix
– Row vectors of real orthogonal matrix A form orthonormal basis vectors
 Properties of 1-D Unitary Transform y = A x

 Energy Conservation
UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

– || y ||2 = || x ||2

 || y ||2 = || Ax ||2= (Ax)*T (Ax)= x*T A*T A x = x*T x = || x ||2

 Rotation
– The angles between vectors are preserved

– A unitary transformation is a rotation of a vector in an

N-dimension space, i.e., a rotation of basis coordinates
 Properties of 1-D Unitary Transform
(cont’d)
 Energy Compaction
UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

– Many common unitary transforms tend to pack a large fraction of signal

energy into just a few transform coefficients

 Decorrelation
– Highly correlated input elements  quite uncorrelated output coefficients
– Covariance matrix E[ ( y – E(y) ) ( y – E(y) )*T ]
 small correlation implies small off-diagonal terms

Example: recall the effect of DFT

Question: What unitary transform gives the best compaction and decorrelation?

=> Will revisit this issue in a few lectures

 Review: 1-D Discrete Cosine Transform
(DCT)
 N 1
 ( 2n  1)k 
 
UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

 Z ( k )  z ( n )  ( k ) cos  
 n 0 2N

 z ( n) 
N 1
 ( 2n  1)k 
  Z ( k )   ( k ) cos 
 
 k 0 2N
1 2
 (0)  ,  (k ) 
N N

 Transform matrix C
– c(k,n) = (0) for k=0
– c(k,n) = (k) cos[(2n+1)/2N] for k>0
 C is real and orthogonal
– rows of C form orthonormal basis
– C is not symmetric!
– DCT is not the real part of unitary DFT!  See Assignment#3
 related to DFT of a symmetrically extended signal
 Periodicity Implied by DFT and DCT
UMCP ENEE631 Slides (created by M.Wu © 2004)

Figure is from slides at Gonzalez/ Woods DIP book website (Chapter 8)

 From Ken Lam’s DCT talk 2001 (HK Polytech)

Example of 1-D DCT

UMCP ENEE631 Slides (created by M.Wu © 2001)

100 100

50 50
z(n) Z(k)
0 0
DCT
-50 -50

-100 -100
0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7
n k
 From Ken Lam’s DCT talk 2001 (HK Polytech)

Example of 1-D DCT (cont’d)

1.0 1.0 100 100
UMCP ENEE631 Slides (created by M.Wu © 2001)

0.0 0.0 0 0

-1.0 -1.0 -100 u=0 -100 u=0 to 4

z(n)
1.0 1.0 100 100

n 0.0 0.0 0 0

Original signal -1.0 -1.0 -100 u=0 to 1 -100 u=0 to 5

1.0 1.0 100 100

0.0 0.0 0 0

-1.0 -1.0 -100 u=0 to 2 -100 u=0 to 6

Z(k)
1.0 1.0 100 100

k 0.0 0.0 0 0

-1.0 -1.0 -100 u=0 to 3 -100 u=0 to 7

Transform coeff.
Basis vectors Reconstructions
 2-D DCT
 Separable orthogonal transform
– Apply 1-D DCT to each row, then to each column
UMCP ENEE631 Slides (created by M.Wu © 2001)

Y = C X CT  X = CT Y C = mn y(m,n) Bm,n

 DCT basis images:

– Equivalent to represent
an NxN image with a set
of orthonormal NxN
“basis images”
– Each DCT coefficient
indicates the contribution
from (or similarity to) the
corresponding basis image
 2-D Transform: General Case

 N 1 N 1

 y ( k , l )    x ( m, n) ak ,l (m, n)
UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

 m 0 n 0
 N 1 N 1
 x( m, n) 


 
k 0 l 0
y ( k , l ) hk ,l (m, n)

 A general 2-D linear transform {ak,l(m,n)}

– y(k,l) is a transform coefficient for Image {x(m,n)}
– {y(k,l)} is “Transformed Image”
– Equiv to rewriting all from 2-D to 1-D and applying 1-D transform

 Computational complexity
– N2 values to compute
– N2 terms in summation per output coefficient
– O(N4) for transforming an NxN image!
 2-D Separable Unitary Transforms

 Restrict to separable transform

UMCP ENEE631 Slides (created by M.Wu © 2001)

– ak,l(m,n) = ak(m) bl(n) , denote this as a(k,m) b(l,n)

 Use 1-D unitary transform as building block

– {ak(m)}k and {bl(n)}l are 1-D complete orthonormal sets of basis vectors
 use as row vectors to obtain unitary matrices A={a(k,m)} & B={b(l,n)}
– Apply to columns and rows Y = A X BT
 often choose same unitary matrix as A and B (e.g., 2-D DFT)

 For square NxN image A: Y = A X AT  X = AH Y A*

– For rectangular MxN image A: Y = AM X AN T  X = AMH Y AN*

 Complexity ~ O(N3)
– May further reduce complexity if unitary transf. has fast implementation
 Basis Images

 X = AH Y A* => x(m,n) = k l a*(k,m)a*(l,n) y(k,l)

UMCP ENEE631 Slides (created by M.Wu © 2001)

– Represent X with NxN basis images weighted by coeff. Y

– Obtain basis image by setting Y={(k-k0, l-l0)} & getting X
 { a*(k0 ,m)a*(l0 ,n) }m,n
 in matrix form A*k,l = a*k al
*T ~ a* is kth column vector of
k
AH
 trasnf. coeff. y(k,l) is the inner product of A*k,l with

the image  1 1   
1 2
 2 2  X  
A
 1 1  3 4 

 2 2 

(Jain’s e.g.5.1, pp137)

 Common Unitary Transforms and Basis
Images
DFT DCT Haar transform K-L transform
UMCP ENEE631 Slides (created by M.Wu © 2001)

See also: Jain’s Fig.5.2 pp136

 2-D DFT
 2-D DFT is Separable
UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

– Y = F X F  X = F* Y F*

– Basis images Bk,l = (ak ) (al )T

where ak = [ 1 WN-k WN-2k … WN-(N-1)k ]T /  N

 1 N 1 N 1


Y ( k , l ) 
N

m 0 n 0
X ( m , n )  W N
nl
 W mk
N

 N 1 N 1
 X (m, n)  1

 N

k 0 l 0
Y ( k , l )  W  nl
N  W N
 mk
 Summary and Review (1)

 1-D transform of a vector

UMCP ENEE631 Slides (created by M.Wu © 2001)

– Represent an N-sample sequence as a vector in N-dimension vector space

– Transform
 Different representation of this vector in the space via different basis
 e.g., 1-D DFT from time domain to frequency domain

– Forward transform
 In the form of inner product
 Project a vector onto a new set of basis to obtain N “coefficients”
– Inverse transform
 Use linear combination of basis vectors weighted by transform coeff.
to represent the original signal

 2-D transform of a matrix

– Rewrite the matrix into a vector and apply 1-D transform
– Separable transform allows applying transform to rows then columns
 Summary and Review (1) cont’d
 Vector/matrix representation of 1-D & 2-D sampled signal
– Representing an image as a matrix or sometimes as a long vector
UMCP ENEE631 Slides (created by M.Wu © 2001)

 Basis functions/vectors and orthonomal basis

– Used for representing the space via their linear combinations
– Many possible sets of basis and orthonomal basis

 Unitary transform on input x ~ A-1 = A*T

– y = A x  x = A-1 y = A*T y =  ai*T y(i) ~ represented by basis vectors {ai*T}
– Rows (and columns) of a unitary matrix form an orthonormal basis

 General 2-D transform and separable unitary 2-D transform

– 2-D transform involves O(N4) computation
– Separable: Y = A X AT = (A X) AT ~ O(N3) computation
 Apply 1-D transform to all columns, then apply 1-D transform to rows
UMCP ENEE631 Slides (created by M.Wu © 2004)

Optimal Transform
 Optimal Transform
 Recall: Why use transform in coding/compression?
– Decorrelate the correlated data
UMCP ENEE631 Slides (created by M.Wu © 2004)

– Pack energy into a small number of coefficients

– Interested in unitary/orthogonal or approximate orthogonal transforms

 Energy preservation s.t. quantization effects can be better understood
and controlled
 Unitary transforms we’ve dealt so far are data independent
– Transform basis/filters are not depending on the signals we are processing

 What unitary transform gives the best energy compaction and decorrelation?
– “Optimal” in a statistical sense to allow the codec works well with
many images
 Signal statistics would play an important role
 Review: Correlation After a Linear
Transform
 Consider an Nx1 zero-mean random vector x
– Covariance (autocorrelation) matrix Rx = E[ x xH ]
UMCP ENEE631 Slides (created by M.Wu © 2004)

 give ideas of correlation between elements

 Rx is a diagonal matrix for if all N r.v.’s are uncorrelated

 Apply a linear transform to x: y = A x

 What is the correlation matrix for y ?
Ry = E[ y yH ] = E[ (Ax) (Ax)H ] = E[ A x xH AH ]
= A E[ x xH ] AH = A Rx AH

 Decorrelation: try to search for A that can produce a decorrelated y (equiv. a

diagonal correlation matrix Ry )
 K-L Transform (Principal Component
Analysis)
 Eigen decomposition of Rx: Rx uk = k uk
UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

– Recall the properties of Rx

 Hermitian (conjugate symmetric RH = R);
 Nonnegative definite (real non-negative eigen values)

 Karhunen-Loeve Transform (KLT)

y = UH x  x = U y with U = [ u1, … uN ]
– KLT is a unitary transform with basis vectors in U being the
orthonormalized eigenvectors of Rx

– UH Rx U = diag{1, 2, … , N} i.e. KLT performs decorrelation

– Often order {ui} so that 1  2  …  N

– Also known as the Hotelling transform or

the Principle Component Analysis (PCA)
 Properties of K-L Transform

 Decorrelation
UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

– E[ y yH ]= E[ (UH x) (UH x)H ]= UH E[ x xH ] U = diag{1, 2, … , N}

– Note: Other matrices (unitary or nonunitary) may also decorrelate

the transformed sequence [Jain’s e.g.5.7 pp166]

 Minimizing MSE under basis restriction

– If only allow to keep m coefficients for any 1 m N, what’s the
best way to minimize reconstruction error?
 Keep the coefficients w.r.t. the eigenvectors of the first m largest
eigenvalues

 Theorem 5.1 and Proof in Jain’s Book (pp166)

 KLT Basis Restriction
 Basis restriction
– Keep only a subset of m transform coefficients and then perform
UMCP ENEE631 Slides (created by M.Wu © 2004)

inverse transform (1 m  N)

– Basis restriction error: MSE between original & new sequences

 Goal: to find the forward and backward transform matrices to minimize the
restriction error for each and every m
– The minimum is achieved by KLT arranged according to the
decreasing order of the eigenvalues of R
 K-L Transform for Images

 Work with 2-D autocorrelation function

UMCP ENEE631 Slides (created by M.Wu © 2001)

– R(m,n; m’,n’)= E[ x(m, n) x(m’, n’) ] for all 0 m, m’, n, n’  N-1

– K-L Basis images is the orthonormalized eigenfunctions of R

 Rewrite images into vector form (N2x1)

– Need solve the eigen problem for N2xN2 matrix! ~ O(N 6)

 Reduced computation for separable R

– R(m,n; m’,n’)= r1(m,m’) r2(n,n’)
– Only need solve the eigen problem for two NxN matrices ~ O(N3)
– KLT can now be performed separably on rows and columns
 Reducing the transform complexity from O(N4) to O(N3)
 Pros and Cons of K-L Transform

 Optimality
UMCP ENEE631 Slides (created by M.Wu © 2001)

– Decorrelation and MMSE for the same# of partial coeff.

 Data dependent
– Have to estimate the 2nd-order statistics to determine the transform
– Can we get data-independent transform with similar performance?
 DCT

 Applications
– (non-universal) compression
– pattern recognition: e.g., eigen faces
– analyze the principal (“dominating”) components
 Energy Compaction of DCT vs. KLT
 DCT has excellent energy compaction for highly
correlated data
UMCP ENEE631 Slides (created by M.Wu © 2004)

 DCT is a good replacement for K-L

– Close to optimal for highly correlated data
– Not depend on specific data like K-L does
– Fast algorithm available

[ref and statistics: Jain’s pp153, 168-175]

 Energy Compaction of DCT vs. KLT (cont’d)
 Preliminaries
– The matrices R, R-1, and R-1 share the same eigen vectors
UMCP ENEE631 Slides (created by M.Wu © 2004)

– DCT basis vectors are eigenvectors of a symmetric tri-diagonal matrix Qc

– Covariance matrix R of 1st-order stationary Markov sequence with has an

inverse in the form of symmetric tri-diagonal matrix
 DCT is close to KLT on 1st-order stationary Markov
– For highly correlated sequence, a scaled version of R-1 approx. Qc
 Summary and Review on Unitary Transform
 Representation with orthonormal basis  Unitary transform
– Preserve energy
UMCP ENEE631 Slides (created by M.Wu © 2001)

 Common unitary transforms

– DFT, DCT, Haar, KLT

 Which transform to choose?

– Depend on need in particular task/application
– DFT ~ reflect physical meaning of frequency or spatial
frequency
– KLT ~ optimal in energy compaction
– DCT ~ real-to-real, and close to KLT’s energy compaction