M1 Numerical Analysis Course

M1 SMA International Ecole Centrale de Nantes
Numerical Analysis
Anthony NOUY
anthony.nouy@ec-nantes.fr
Oce : F231
Origin of problems in numerical analysis References
Part I Introduction
Origin of problems in numerical analysis
References
Part I Introduction
References
Origin of problems in numerical analysis I
How to interpret the reality with a computer language: from a continuous world to a discrete world. Numerical solution of a dierential equation Find u : x u (x ) such that
A(u ) = b
Example 1 (1D diusion equation, beam in traction, ...)
d du ( ) = b (x ) for x = (0, 1), u (0) = u (1) = 0 dx dx
Origin of problems in numerical analysis II

Approximation (from a continuous to a discrete representation) Represent a function u on a (nite-dimensional) approximation space : u (x ) =
n i =1
ui hi (x )
The solution is then represented by u = (u1 , . . . , un ) Rn . For the denition of the expansion, dierent alternatives such as methods based on a weak formulation of the problem. Example 2 (Galerkin approximation) Find u V = {v : (0, 1) R; v (0) = v (1) = 0} such that
dv du dx = dx dx
v b dx v V
and replace function space V by approximation space V n = {v (x ) = n i =1 vi hi (x )} V
Origin of problems in numerical analysis III

If A is a linear operator, the initial continuous equation is then transformed into Linear systems of equations Find u Rn such that where A Rnn is a matrix and Numerical Integration f (x ) dx
=b n b R a vector.
Au
In order to construct the system of equation (matrix A and right-hand-side b):

K k =1
k f (xk )
If A is a nonlinear operator:
Origin of problems in numerical analysis IV

Nonlinear system of equations Find u Rn such that
A(u)
=b
where A : u Rn A(u) Rn . Remedy: iterative solution techniques which transform the solution of a nonlinear equation into solution of linear equations. Example 3
d du ((x , u ) ) = b (x , u ) for x = (0, 1), u (0) = u (1) = 0 dx dx
Eigenproblems Find (u, ) Cn C such that

Au
= u
or
Au
= Bu
where A, B Cnn are matrices.
Origin of problems in numerical analysis V

Example 4 (Eigenmodes of a beam) Wave equation: solution u (x , t ) such that
u 2u ( ) + 2 = 0 x x t
for x = (0, 1), u (0, t ) = u (1, t ) = 0
for which we search solutions of the form u (x , t ) = w (x ) cos(t ): w Vn ,
v w dx = 2 x x
v w dx
v V n
Ordinary dierential equations in time d u(t ) + A(u(t ); t ) = b(t ) dt
Part I Introduction
References
References for the course
G. Allaire and S. M. Kaber. Numerical linear algebra. Springer, 2007. materials for chapters 1 (Linear Algebra), 2 (Linear
systems), 3(Eigenvalues)
K. Atkinson and W. Han. Springer, 2009.

materials for chapters 4 (Nonlinear equations), 5
Theoretical Numerical Analysis: A Functional Analysis Framework
(Approximation/Interpolation) a quite abstract introduction to numerical analysis (very instructive), with an introduction to functional analysis
E. Suli and D. Mayers. Cambridge University Press, 2003. G. Allaire. Cambridge University Press, 2007.
An Introduction to Numerical Analysis
a clear and simple presentation of all the ingredients of the course

Numerical Analysis and Optimization
additional material for numerical solution of PDE and optimization
problems a natural continuation of the course
Matrices Reduction of matrices Vector and matrix norms
Part II Linear algebra
Vector space
Let V be a vector space with nite dimension n, on the eld K (R or C). Let E = {e1 , . . . , en } be a basis of V . A vector v V admits a unique decomposition v=
n
where the (vi )n i =1 are the components of v on the basis E . When a basis is chosen and when there is no ambiguity, we can identify V to Kn (Rn or Cn ) and let v = (vi )n i =1 , represented by the column vector

i =1
vi e i
. v = . . vn We denote respectively by v T and v H the transpose and conjugate transpose of v , which are the following row vectors v T = v1 . . . vn
1 . . . v n vH = v
v1
where a denotes the complex conjugate of a.
Canonical inner product
We denote by (, ) : V V K the canonical inner product dened for all u , v V by

(u , v ) = u T v = v T u = (u , v ) = u H v = v H u =
n i =1 n i =1
ui vi u i vi
if if
K=R K=C
K = C.
It is called euclidian inner product if K = R and hermitian inner product if
Orthogonality
Orthogonality on a vector space V must be thought with respect to an inner product (, ). If not mentioned, we classically consider the canonical inner product. Two vectors u , v V are said orthogonal with respect to inner product (, ) if and only if (u , v ) = 0. A vector v is said orthogonal to a linear subspace U V , which is denoted v U , if and only if (v , u ) = 0 for all u U . Two linear subspaces U V and U V are said orthogonal, and it is denoted U U , if
(u , u ) = 0 u U , u U
For a given subspace U V , we denote by U its orthogonal complement, which is the largest subspace orthogonal to U . The orthogonal complement of a vector v V is denoted by v .
Matrices
Let V and W be two vector spaces with dimension n and m respectively, with m bases E = (ei )n i =1 and F = (fi )i =1 . A linear map A : V W , relatively to those bases, is represented by a matrix A with m rows and n columns a11 a12 . . . a1n a21 a22 . . . a2n A= . . . . . . . . . am1 am2 . . . amn

where the coecients aij are such that

Aej =
m i =1
aij fi , 1
We denote (A)ij = aij . The j -th column of A represents the vector Aej in the basis F . Denition 5 The set of matrices with m rows and n columns with entries in the eld K is a vector space denoted Mm,n (K) or Kmn .
Transpose
We denote AH the adjoint (or conjugate transpose) matrix of a complex matrix A = (aij ) Cmn , dened by
(AH )ij = aji
We denote AT the transpose of a real matrix A = (aij ) Rnm , dened by

(AT )ij = aji
We have the following characterization of AH and AT :

(Au , v ) = (u , AH v ) (Au , v ) = (u , AT v ) u C n , v C m u R n , v R m
Product
To the composition of two linear maps corresponds the multiplication of the associated matrices. If A = (aik ) Kmq and B = (bkj ) Kqn , the product AB Kmn is dened by
(AB )ij =
q k =1
aik bkj
We have
The set of square matrices Mn,n (K) is simply denoted Mn (K) = Knn . In the following, unless it is mentioned, we only consider square matrices.
(AB )T = B T AT ,
(AB )H = B H AH
Inverse
We denote by In the identity matrix on Knn , associated with the identity map from V to V . If there is no ambiguity, we simply denote In = I and
(I )ij = ij
where ij is the Knonecker delta. A matrix is invertible if there exists a matrix denoted A1 (unique if it exists) and called the inverse matrix of A, such that AA1 = A1 A = I . A matrix which is not invertible is said singular. If A and B are invertible, we have
(AB )1 = B 1 A1 , (AT )1 = (A1 )T AT , (AH )1 = (A1 )H AH
Particular matrices
Denition 6 A matrix A Cnn is said Hermitian if A = AH Normal if AAH = AH A Unitary if AAH = AH A = I Denition 7 A matrix A Rnn is said Symmetric if A = AT Orthogonal if AAT = AT A = I
Particular matrices
A matrix A Knn is said diagonal if aij = 0 for i = j and we denote a11 0 . . . 0 . .. .. 0 . . . . A = diag (aii ) = diag (a11 , . . . , ann ) = . . . . . . . . 0 . 0 . . . 0 ann A matrix A is said upper triangular if aij = 0 for i > j : a11 a12 . . . a1n 0 a22 . . . a2n A= . . .. .. . . . . . . 0 . . . 0 ann A matrix A is said lower triangular if aij = 0 for j > i : a11 0 . . . 0 . .. a21 a22 . . . A= . . . . . . . . . 0 an1 an2 . . . ann
Properties of triangular matrices
Let Ln Knn be the set of lower triangular matrices, and Un Knn be the set of upper triangular matrices. Theorem 8 If A, B Ln , then AB Ln If A, B Un , then AB Un A Ln (or Un ) is invertible if and only if all its diagonal terms are nonzero. If A Ln , A1 Ln (if it exists) If A Un , A1 Un (if it exists)
Trace
Denition 9 The trace of a matrix A Knn is dened as tr (A) = Property 10 tr (A + B ) = tr (A) + tr (B ), tr (AB ) = tr (BA)
n i =1
aii
Determinant
Let Sn denote the set of permutations of {1, . . . , n}. For Sn , we denote by sign() the signature of the permutation, with sign() = +1 (resp. 1) if is an even (resp. odd) permutation of {1, . . . , n}. Denition 11 The determinant of a matrix A Knn is dened as det (A) = Property 12 det (AB ) = det (BA) = det (A)det (B )
Sn
sign()a(1)1 . . . a(n)n
Image, Kernel I
Denition 13 The image of A Kmn is a linear subspace of Km dened by Im(A) = {Av ; v Kn } The rank of a matrix A, denoted rank (A), is the dimension of Im(A): rank (A) = dim(Im(A)) min(m, n)
Denition 14 The kernel of A Kmn is a linear subspace of Kn dened by Ker (A) = {v Kn ; Av = 0} The dimension of Ker (A) is called the nullity of A. Property 15 dim(Im(A)) + dim(Ker (A)) = n
Image, Kernel II
Property 16 For A Rmn , Ker (AT ) + Im(A) = Rm , Ker (AT ) = Im(A) Ker (A) + Im(AT ) = Rn , Ker (A) = Im(AT ) Proof. Let us prove that Ker (AT ) = Im(A) , which implies Ker (AT ) + Im(A) = Rm . First, u Ker (AT ) AT u = 0 v T AT u = 0 v u T y = 0 y Im(A) Ker (AT ) Im(A) . Secondly, u Im(A) u T Av = 0 v v T (AT u ) = 0 v AT u = 0 Im(A) Ker (AT ). Exercice. Finish the proof.
Eigenvalues and eigenvectors I
Denition 17 Eigenvalues i = i (A), 1 characteristic polynomial
n, of a matrix A Knn are the n roots of its
pA : C pA () = det (A I ) The eigenvalues may be real or complex. An eigenvalue is said of multiplicity k if it is a root of pA with multiplicity k . The spectrum of matrix A is the following subset of the complex plane sp (A) = {i (A)}n i =1 We have tr (A) =
n i =1
i (A ),
det (A) =
n i =1
i (A)
Eigenvalues and eigenvectors II
Denition 18 The spectral radius (A) of a matrix A is dened by

(A) = max |i (A)| 1 i n
Property 19 sp (A) if and only if the following equation has (at least) a nontrivial solution v Cn \{0}: Av = v Denition 20 For sp (A), a vector v satisfying Av = v is called an eigenvector of A associated with . The linear subspace {v Kn ; Av = v } (with dimension at least one) is called the eigenspace associated with .
Reduction of matrices
Let V be a vector space with dimension n and A : V V a linear map on V . Let A be the matrix associated with A, relatively to the basis E = (ei )n i =1 of V . Relatively to another basis F = (fi )n of V , the application A is associated i =1 with another matrix B such that B = P 1 AP where P is an invertible matrix whose j -th column is composed by the components of fj on the basis E . Denition 21 Matrices A and B are said similar when they represent the same linear map in two dierent basis, i.e. when there exists an invertible matrix P such that B = P 1 AP .
Theorem 22 (Triangularization)
For A Cnn , there exists a unitary matrix U such that U 1 AU is a triangular matrix, called the Schur form of A (if upper triangular). Remark. The previous theorem says that there exists a nested sequence of A-invariant subspaces {0} = V0 V1 . . . Vn = Cn and there exists an orthonormal basis of Cn such that Vi is the span of the rst i basis vectors. Theorem 23 (Diagonalization) For a normal matrix A Cnn , i.e. such that AH A = AAH , there exists a unitary matrix U such that U 1 AU is diagonal. For a symmetric matrix A Rnn , there exists an orthogonal matrix O such that O 1 AO is diagonal.
Singular values and vectors
Denition 24 The singular values of A Kmn are the eigenvalues of Singular values of A are real non-negative numbers.
AH A Knn .
Denition 25 R+ is a singular value of A if and only if there exists normalized vectors u Km and v Kn such that we have simultaneously Av = u and AH u = v u and v are respectively called the left and right singular vectors of A associated with singular value .
Singular value decomposition (SVD) I

Theorem 26 For A Kmn , there exist two orthogonal (if K = R) or unitary (if K = C) matrices U Kmm and V Knn such that A = USV H where S = diag (i ) Rmn is a diagonal matrix, with i the singular values of A. The columns of U are the left singular vectors of A, and the columns of V are the right singular vectors of A.
1
If n = m, S = diag (i ) =
..
.
m
. If n = m,
S = diag (i ) Rmn must be interpreted as follows (0kl is a k l matrix with zero entries):
1 1
..
.
n
0m(nm) if n > m,
..
0(mn)n
if n < m, n
Truncated Singular Value Decomposition (SVD)

The SVD of A can be written A = USV H =
min(n,m)
i =1
i ui viH
After ordering the singular values by decreasing values (1 2 . . .), matrix A can be approximated by a rank-K matrix AK obtained by a truncation of the SVD: AK = We have the following error estimate:
K i =1
i ui viH
min(n,m) A AK F 2 = i2 A F i =K +1
Illustration: SVD for data compression
Initial image (778 643)
Singular values
Rank-10 SVD
Singular values
Rank-20 SVD
Singular values
Rank-30 SVD
Singular values
Rank-40 SVD
Singular values
Rank-50 SVD
Singular values
Rank-100 SVD
Vector norms
Denition 27 A norm on vector space V is an application : V R+ verifying v = 0 if and only if v = 0 v = || v for all v V and K u+v u + v for all u , v V (triangle inequality) Example 28 (For V = Kn )
n |v |2 1/2 (2-norm) v 2 = i =1 i n (1-norm) v 1 = i =1 |vi | (-norm) v = maxi {1,...,n} |vi |
(p -norm) v p =
n |v |p i =1 i
1/p
for p
1.
Useful inequalities
(, ) denote the canonical inner product.
Theorem 29 (Cauchy-Schwartz inequality)

|(u , v )|
Theorem 30 (Hlder's inequality)

1 1 +q = 1, then Let 1 p , q such that p
|( u , v ) |
u p v q
Theorem 31 (Minkowski inequality) Let 1 p , then u+v p
u p+ v p
Minkowski inequality is in fact the triangular inequality for the norm p .
Matrix norms I
Denition 32 A norm on Kmn is a map : Kmn R+ which veries A = 0 is and only if A = 0 A = || A for all A Kmn and K A+B A + B for all A, B Kmn (triangle inequality) For square matrices (n = m), a matrix norm is a norm which satises the following additional inequality AB A B for all A Knn , B Knn An important class of matrix norms is the class of subordinate matrix norms. Denition 33 (subordinate matrix norm) Given norms on Kn and Km , we can dene a natural norm on Kmn , subordinate to the vectors norms, and dened by A = max n
v C :v =0
Av = max v Cn : v v
Av =
v Cn : v
max
=1
Av
Matrix norms II
Example 34 When considering classical vector norms on Kn , we have the following characterization of the subordinate norms of a square matrix A Knn : 1 A 1 = maxv Av v 1 = maxj i |aij | A
A 2 Note that A Property 35
= maxv Av v = maxi j |aij | H 2 H H = maxv Av v 2 = (A A) = (AA ) = A
2.
corresponds to the dominant singular value of A.
For all unitary matrix U (i.e. UU H = I ), we have A

2
= AU
= UA
2 2
= U H AU
If A is normal (i.e. AAH = AH A), then A
= (A ).
Matrix norms III
Theorem 36 Let A be a square matrix and an arbitrary matrix norm. Then

(A)
A
(A ) +
For > 0, there exists at least one subordinate matrix norm such that A
Conditioning Direct methods Iterative methods
Part III Systems of linear equations

6 7
Conditioning Direct methods Triangular systems Gauss elimination LU factorization Cholesky factorization Householder method and QR factorization Computational work Iterative methods Generalities Jacobi, Gauss-Seidel, Relaxation Projection methods Krylov subspace methods
The aim is to introduce dierent strategies for the solution of a system of linear equations Ax = b n n n with A R , b R .

6 7
Condition number
Let consider the following two systems of equations 10 7 8 7 10 7 8 7
7 5 6 5 7 5 6 5
8 6 10 9 8 6 10 9
32 7 23 5 x = 33 9 31 10 7 32.1 5 x = 22.9 9 33.1 10 30.9

1 x = 1
1 1
9 .2 12.6 x = 4 .5 1.1

We observe that a little modication of the right-hand side leads a large modication in the solution. If an error is made on the input data (here the right-hand side), the error on the solution may be drastically amplied. This phenomenon is due to a bad conditioning of the matrix A. It reveals that for badly conditioned matrices, the solution of systems of equations obtained with nite precision computers has to be considered carefully or even not considered as a good solution.
Denition 37 Let A Knn be an invertible matrix and let be a matrix norm subordinate to the vector norm . The condition number of A is dened as cond (A) = A A1 Let b Kn be the right-hand side of a system and let A Knn and b Kn be perturbations of matrix A and vector b. Property 38 If x and x are solutions of the following systems Ax = b, A x = b , with A A
= O ( ) and b b = O ( ), then
x x x
cond (A)
AA A
bb b
+ O ( 2)
Property 39 For every matrix A and every matrix norm, cond (A) 1, cond (A) = cond (A1 ), cond (A) = cond (A), = 0. For every matrix A, the condition number cond2 (A) = A associated with the 2-norm veries maxi i (A) cond2 (A) = mini i (A) where the i (A) are the singular values of A. For a normal matrix A, cond2 (A) = maxi |i (A)| mini |i (A)|
A1
where the i (A) are the eigenvalues of A. For unitary or orthogonal matrix A, the condition number cond2 (A) = 1. The condition number cond2 (A) is invariant trough unitary transformation: cond2 (A) = cond2 (AU ) = cond2 (UA) = cond2 (U H AU ) for every unitary matrix U.
Triangular systems Gauss elimination LU factorization Cholesky

6 7
Principle of direct methods I

For solving
Ax = b, MAx = Mb
direct methods consist in determining an invertible matrix M such that is an upper triangular system. This is called the elimination step. Then, a simple backward substitution can be performed to solve this triangular system. Do not compute the inverse !!! In practice, the solution x of Ax = b is not obtained by rst computing the inverse A1 and then computing the matrix-vector product A1 b. Indeed, it would be equivalent to solving n systems of linear equations. For simplicity, we use sometimes the notation M 1 x but the inverse is never computed in practise. This operation corresponds to the solution of a system of equations (generally easy due to properties of M : diagonal, triangular).

6 7
Triangular systems of equations I

If A is lower triangular, the system
a11
0 a22 . . . an 2
a21 . . .
an 1
0 b1 . x1 .. . . . . . . .=. . .. . 0 xn bn . . . ann
...
is solved by a forward substitution Algorithm 40 (Forward substitution for lower triangular system) Step 1. a11 x1 = b1 Step 2. a22 x2 = a21 x1 ... 1 Step n. ann xn = bn n j =1 anj bj
Triangular systems of equations II

If A is upper triangular, the system
a11 a12 . . . a1n x1 b1 0 a22 . . . a2n . . . = . . . . .. .. . . . . . . . xn bn 0 . . . 0 ann

is solved by a backward substitution Algorithm 41 (Backward substitution for upper triangular system) Step 1. Step 2. ... Step n. ann xn = bn an1,n1 xn1 = an1,n xn a11 x1 = b1
n a b j =2 1j j

6 7
Gauss elimination I
Denition 42 (Pivoting matrix) A pivoting matrix P (i , j ), associated with a linear mapping written in a basis E = (ei )n i =1 , is dened as follows P (i , j ) = I (ei ej )(ei ej )H For A Knn , P (i , j )A is the matrix A with permuted lines i and j , and AP (i , j ) is the matrix A with permuted columns i and j . Let us note that P (i , i ) = I . We now describe the Gauss elimination procedure
1 Let A = A1 = (aij ). Select a nonzero element ai1 1 of the rst column and permute the lines 1 and i . Let P1 = P (1, i ) and set 1 1 = P1 A1 = ( A aij )
Step 1.
Gauss elimination II
Let introduce the matrix

a 1 21 1 a 11 E1 = . . .1 a n 1 a 1
11
..
. .. . 1
1 a12 2 a22
such that
1 a11 0 1 = A2 = E 1 A . . .
2 2 an 2 . . . ann
. . .
... ...
1 a1 n 2 a2n . . .
Step 2.
We have det (A2 ) = det (E1 P1 A1 ) = det (E1 )det (P1 )det (A) = det (A) (det (A) if a line permutation has been made, +det (A) if not). Therefore A2 is invertible, and so is the submatrix (A2 )ij , 2 i , j n. We can then operate as in step 1 for this submatrix for eliminating the subdiagonal elements of
Gauss elimination III
column 2: introduce a permutation matrix P2 = P (2, i ), with i 2 = P3 A2 and A3 = E3 A 2. line operation matrix E2 , and let A Step k 1. After k 1 steps, we have the matrix
k a11 Ak = Ek 1 Pk 1 . . . E1 P1 A1 =
2, and a
k ... a12 k ... a22 .. .
... ...
... ... ... ...
k akk . . . k ank
k a1 n k a2 n . . . k akn . . . k ann
k = Pk Ak and After an eventual pivoting with a pivoting matrix Pk , we dene A Ak +1 = Ek Ak with
Gauss elimination IV
Ek =
..
a k 1 ,k k+ a k kk a k nk a k kk
1 . . .
..
. .. .
Last step
After n 1 steps, by we obtain an upper triangular matrix An = En1 Pn1 . . . E1 P1 A The invertible matrix M = En1 Pn1 . . . E1 P1 is then an invertible matrix such that MA is upper triangular.
Gauss elimination V
Remark. Choice of pivoting In order to avoid dramatic roundo errors with nite precision computers, we adopt one of the following pivoting strategies. Partial pivoting. At step k , we select Pk = P (k , i ) such that k| |aik k | = max |aik Total pivoting. At step k , we select i and j such that k | and we permute lines and columns by dening |aik j | = max |aij
k = P (k , i )Ak P (j , k ). A
k i n
i k ,j n
Gauss elimination VI
Remark. Computing the determinant of a matrix The Gauss elimination is an ecient technique for computing the determinant of a matrix. Indeed, det (A) = det (An )det (M )1 =
n i =1 n aii
where the sign depends on the number of pivoting operations that have been performed. Remark. In practice, for solving a system Ax = b, we don't compute the matrix M . We rather operate simultaneously on b by computing Mb = bn = En1 Pn1 . . . E1 P1 b Then, we solve the triangular system MAx = MB , or equivalently An x = bn .
Gauss elimination VII
Computational work of Gauss Elimination 2 O ( n3 ) 3 For an arbitrary matrix, it seems that this computational work in O (n3 ) is near the optimal that we can expect. That is the reason why Gauss elimination can be used when no additional information is given on the matrix. Theorem 43 For A Kn (inversible or not), there exists at least one invertible matrix M such that MA is an upper triangular matrix. Proof. For A invertible, the Gauss elimination procedure is a constructive proof for this theorem. Otherwise, the matrix A is singular if and only there exists a matrix k = 0 for k i n. In this case, we can set E = I and Ak with elements aik k Pk = I at step k of the Gauss elimination and go to the next step.

6 7
LU factorization I
The LU factorization of a matrix consists in constructing lower and upper triangular matrices L and U such that A = LU . In fact, this factorization is obtained by the Gauss elimination procedure. k = Ak . Let us consider the Gauss elimination without pivoting, i.e. by letting A k = 0. We then let It is possible if at step k , akk M = En1 . . . E1 and obtain MA = U where U is the desired upper triangular matrix
1 1 1 a11 a12 . . . a1 n 2 2 a22 . . . a2n . .. . . . n ann
M being a product of lower triangular matrices, it is a lower triangular matrix and so is its inverse M 1 . We then have the desired decomposition with
1 1 L = M 1 = E1 . . . En 1
LU factorization II
Matrix L = (lij ) is directly obtained from matrices Ek

Ek =
..
1
lk +1,k
..
. . . lnk
..
..
1 Ek
1 lk +1,k . . . lnk
..
..
LU factorization III
Theorem 44 nn be such that the diagonal submatrices Let AK a11 . . . a1k . . k k are invertible. Then, there exists a lower triangular . . . . K ak 1 . . . akk matrix L and an upper triangular matrix U such that A = LU If we further impose that the diagonal elements of L are equal to 1, this decomposition is unique. Proof. The condition on the invertibility of submatrices ensures that at step k , the k is nonzero and therefore that pivoting can be omitted. diagonal term akk

6 7
Cholesky factorization I
Theorem 45 If A Rnn is a symmetric denite positive matrix, there exists at least one lower triangular matrix B = (bij ) Rnn such that A = BB T If we further impose that the diagonal elements bii > 0, the decomposition is unique.
Cholesky factorization II
Proof. We simply show that the diagonal submatrices k = (aij ), 1 i , j k , are positive denite. Therefore, they are invertible and there exists a unique LU factorization A = LU such that L has unit diagonal terms. Since the k are positive denite, we have k (kk ) > 0, for all k 1. We then i =1 uii = det dene the diagonal matrix D = diag ( uii ) and we write A = (L)(1 U ) = BC where B = L and C = 1 U have both diagonal terms bii = cii = uii . The symmetry of matrix A imposes that BC = C T B T and therefore 1 ... 1 1 1 . . . = B 1 C T CB T = =. . . . .. .. . . . . 1 ... 1

and this last equality is only possible if CB T = I C = B T . (Prove the uniqueness of the decomposition).

6 7
Householder matrices
Denition 46 For v a nonzero vector in Cn , we introduce the following matrix, called Householder matrix associated with v : vv H H (v ) = I 2 H v v We will consider, although incorrect, that the identity I is a Householder matrix. Theorem 47 n For x = (xi )n i =1 C , there exists two householder matrices H such that (Hx )i = 0 for i 2. Proof. Denoting by e1 the rst basis vector of Cn , one veries that the two householder matrices H (v ) are associated with the vectors v = x x 2 e i e1 , where R is the argument of x1 C, i.e. x1 = |x1 |e i , and we have H (v )x = x 2 e 1
Householder method I
The Householder method for solving Ax = b consists in nding n 1 1 householder matrices {Hi }n i =1 such that Hn1 . . . H1 A is upper triangular. Then, we solve the following triangular system by backward substitution: Hn1 . . . H1 Ax = Hn1 . . . H1 b Suppose that Ak = Hk 1 . . . H1 A is under the form
k a11 Ak =
k ... a12 2 a22 ... .. .
... ...
... ... ... ...
k akk . . . k ank
k a1 n 2 a2 n . . . k akn . . . k ann
k +1 Let c = (ci )n Cnk +1 be the vector with components ci = aik+k 1 . There i =1 exists a Householder matrix H ( vk ), with v k Cnk +1 , such that H ( vk )c has
Householder method II
0 Cn and v k we let Hk = H (vk ) the householder matrix associated with vk . Let us note that zero components except the rst one. Then, we denote vk = Ik 1 0 0 H ( vk ) Hk = H (vk ) =
Performing this operation for k = 1 . . . n 1, we obtain the desired upper triangular matrix An = Hn1 . . . H1 A.
QR factorization I
The QR factorization is a matrix interpretation of the Householder method. Theorem 48 For A Knn , there exist a unitary matrix Q Knn and an upper triangular matrix R Knn such that A = QR Moreover, one can choose the diagonal elements of R 0. Then, if A is invertible, the corresponding QR factorization is unique.
QR factorization II
Proof. The previous householder construction proves the existence of an upper triangular matrix R = Hn1 . . . H1 A where the Hi are householder matrices. The matrix
1 1 Q = (Hn1 . . . H1 )1 = H1 . . . Hn 1 = H1 . . . Hn1 1 H = H ). is unitary (recall that the Hk are unitary and hermitian, i.e. Hk = Hk k This proves this existence of a QR decomposition. Let now denote by i R the arguments of the diagonal elements rkk = |rkk |e i k and let D = diag (e i k ). = QD is still unitary and the matrix R = D 1 R is still upper The matrix Q triangular with all its diagonal elements greater than 0. We then have the R with rkk 0. We can then show the existence of a QR factorization A = Q uniqueness of this decomposition (let as an exercice).
Remark. If A Rnn , Q , R Rnn , with Q an orthogonal matrix.

6 7
Computational complexity
With classical algorithms... Algorithm LU Cholesky QR Operations 3 O( 2 3n ) 1 3 O( 3 n ) 3 O( 2 3n )
Generalities Jacobi, Gauss-Seidel, Relaxation Projection method

6 7

6 7
Basic iterative methods I
For the solution of a linear system of equations Ax = b, basic iterative methods consist in constructing a sequence {xk }k 0 dened by xk +1 = Bxk + c from an initial vector x0 . Matrix B and vector c are to be dened such that the iterative method converges towards the solution x , i.e. lim x = x k k B and c are chosen such that I B is invertible and such that x is the unique solution of x = Bx + c . Theorem 49 Let B Knn . The following assertions are equivalent (1) limk B k = 0 (2) limk B k v = 0 v (3) (B ) < 1 (4) B < 1 for at least one subordinate matrix norm
Basic iterative methods II
Proof. (1) (2). B k v B k v 0 k (2) (3). If (B ) 1, there exists a vector v = 0 such that Bv = v with || 1 and then B k v = k v does not converge towards 0, a contradiction. (3) (4). Consequence of theorem 36 (4) (1). B k B k 0.
k
Theorem 50 The following assertions are equivalent (i) The iterative method is convergent (ii) (B ) < 1 (iii) B < 1 for at least one subordinate matrix norm Proof. The iterative method is convergent if and only if limk ek = 0, with ek = xk x = B k e0 . The proof then results from theorem 49.

6 7
Jacobi, Gauss-Seidel, Relaxation (SOR) I

We decompose A under the form A=M N where M is an invertible matrix and then Ax = b and we compute the sequence
Mx = Nx + b
xk +1 = M 1 Nxk + M 1 b Bxk + c In practice, at each iteration, we solve the system Mxk +1 = Nxk + b. The method is then ecient if M have a simple form (diagonal or triangular). Denition 51 We decompose A = D E F where D is the diagonal part of A, E and F its strict lower and upper parts.
Jacobi, Gauss-Seidel, Relaxation (SOR) II
Denition 52 (Jacobi) M = D, N = E + F Denition 53 (Gauss-Seidel) M = D E, N = F Denition 54 (Successive Over Relaxation (SOR)) M = 1 D E , N = 1 (1 )D + F
Convergence results I
Theorem 55 Let A a positive denite hermitian matrix, decomposed under the form A = M N with M invertible. If the matrix (M H + N ) is positive denite, then (M 1 N ) < 1.
Convergence results II
Proof. From theorem 36, we know that it suces to nd a matrix norm for which M 1 N < 1. We will show this property for the matrix norm subordinate to the vector norm v = v H Av . Let rst note that (M H + N ) is hermitian since
(M H + N )H = M + N H = A + N + N H = A H + N H + N = M H + N .
We have
M 1 N = I M 1 A = sup v M 1 Av
v
=1 1
Denoting w = M Av , we have, for v such that v = 1, v w

2
= 1 v H Aw w H Av + w H Aw = 1 w H M H w w H Mw + w H Aw = 1 w H (M H + N )w
>0
Therefore v = 1 v M 1 Av < 1. The function v Cn v M 1 Av R is continuous on the unit sphere, which is a compact set, and therefore the supremum is reached.
Convergence results III
Theorem 56 (Sucient condition for convergence of relaxation) If A is hermitian positive denite, relaxation method converges if 0 < < 2. Proof. D . Since A is denite positive, we have for the We show that M H + N = 2 H canonical basis vectors vi , vi Avi = viH Dvi > 0. Matrix M H + N is then hermitian positive denite if and only if 0 < < 2, and the proof ends with theorem 55. Theorem 57 (Necessary condition for convergence of relaxation) The spectral radius of the matrix B = M 1 N of the relaxation method veries
(B ) | 1|
and therefore, relaxation method converges only if 0 < < 2.
Convergence results IV
Proof. We have and then Then
B = (1 D E )1 (1 (1 )D + F ) det (B ) = (1 )n =
n i =1 n i =1
i (B )
(B )
i (B )
1/n
= |1 |

6 7
Projection methods I
We consider a real system of equations Ax = b. Projection techniques consists in searching an approximate solution x in a subspace V of Rn . The approximate solution is then dened by x V , b Ax W where W is a subspace of Rn with the same dimension of V . The approximate solution is then dened by orthogonality constraints on the residual. x is called a projection of x onto the subspace V and parallel to subspace W . The case V = W corresponds to an orthogonal projection and the orthogonality constraint is called Galerkin orthogonality. The case V = W corresponds to an oblique projection and the orthogonality constraint is called Petrov-Galerkin orthogonality. Let V = (v1 , . . . , vm ) and W = (w1 , . . . , wm ) dene bases of V and W , the approximation is then dened by x = Vy , with y Rm such that W T AVy = W T b
y = (W T AV )1 W T b
Projection methods II
Projection method Until convergence 1 Select V = (v1 , . . . , vm ) and W = (w1 , . . . , wm ) 2 r = b Ax 3 y = (W T AV )1 W T r 4 x = x + Vy Subspaces must be chosen such that W T AV is nonsingular. Two important particular choices satises this property. Theorem 58 W T AV is nonsingular for either one the following conditions A is positive denite and V = W A is nonsingular and W = AV .
Projection methods III
Theorem 59 Assume that A is symmetric denite positive and V = W . Then, x V is such that Ax b V if and only if x x 2 x x 2 A = min A, x V
T x 2 A = x Ax
Theorem 60 Let A a nonsingular matrix and W = AV . Then, x V is such that Ax b W if and only if it minimizes the 2-norm of the residual b Ax
2
= min b Ax x V
Basic one-dimensional projection algorithms I
Basic one-dimensional projection schemes consist in selecting V and W with dimension 1. Let us denote V = span{v } and W = span{w }. Denoting r = b Axk the residual at iteration k , the next iterate is dened by xk + 1 = xk + v ,
= (w , r ) wT r = T (w , Av ) w Av
Denition 61 (Steepest descent) We let v = r and w = r . We then have xk + 1 = xk + r ,

= (r , r ) (Ar , r )
If A is symmetric positive denite matrix, xk +1 is the solution of x) = x x 2 x x , A( x x )) min f (xk + r ), f ( A = ( We note that f (xk ) = A(x xk ) = b Axk = r , and therefore xk +1 = xk f (xk ). It then corresponds to a steepest descent algorithm for minimizing the convex function f (x ), with an optimal choice of step .
Basic one-dimensional projection algorithms II
Theorem 62 (Convergence of steepest descent) If A is symmetric positive denite matrix, the steepest descent algorithm converges. Denition 63 (Minimal residual) We let v = r and w = Ar . We then have xk +1 = xk + r , which is the solution of
= (Ar , r ) (Ar , Ar )
2
min b A(xk + r )
Theorem 64 If A is positive denite, minimal residual algorithm converges.
Basic one-dimensional projection algorithms III

Denition 65 (Residual norm steepest descent) We let v = AT r and w = Av = AAT r . We then have xk + 1 = xk + A T r , which is the solution of min f (xk + v ), f (x ) = b Ax
(Av , r ) v 2 = (Av , Av ) Av 2
2
= (Ax b , Ax b )
Note that f (xk ) = AT (b Axk ) = AT r = v . It then corresponds to a steepest descent algorithm on convex function f (x ), with an optimal choice of step . Theorem 66 If A is nonsingular, residual norm steepest descent algorithm converges.

6 7
Krylov subspace methods
Krylov subspace methods are projection methods which consists in dening subspace V as the m-dimensional Krylov subspace of matrix A, associated with r0 = b Ax0 , where x0 is an initial guess. This Krylov subspace is dened by
V = Km (A, r0 ) = span{r0 , Ar0 , . . . , Am1 r0 }
The dierent Krylov subspace methods dier from the choice of space W and from the choice of a preconditioner. First class of methods consisting in taking W = Km (A, r0 ) or W = AKm (A, r0 ). Second class of methods consisting in taking W = Km (AT , r0 ). A complete reference about iterative methods Yousef Saad. Iterative Methods for Sparse Linear Systems. SIAM, 2003.
Jacobi Givens-Householder QR Power iterations Krylov
Part IV Eigenvalue problems

9
Jacobi method Givens-Householder method QR method Power iterations Methods based on Krylov subspaces
10
11
12
13
Eigenvalue problems
The aim is to present dierent techniques for nding the eigenvalues and eigenvectors (i , vi ) of a matrix A: Avi = i vi

9
10
11
12
13
Jacobi method I
Jacobi method allows to nd all the eigenvalues of a symmetric matrix A. It is well adapted to full matrices. There exists an orthogonal matrix O such that O T AO = diag (1 , . . . , n ), where the i are the eigenvalues of A, distinct or not. The Jacobi method consists in constructing a sequence of elementary orthogonal matrices (k )k 1 such that the sequence (Ak )k 1 , dened by
T T A k +1 = T k Ak k = (1 . . . k ) Ak (1 . . . k ) = Ok AOk
converges towards the diagonal matrix diag (1 , . . . , n ) (with an eventual permutation). Each transformation Ak Ak +1 consists in eliminating two symmetric extra-diagonal terms by a rotation. Let A = Ak and B = Ak +1 . The matrix k is selected as follows
T + e e T ) + sin()e e T sin()e e T k = I + (cos () 1)(ep ep q q p q q p
where (/4, /4)\{0} is the unique angle such that bpq = bqp = 0. is solution of a app cotan(2) = qq 2apq
Jacobi method II
Theorem 67 (Convergence of eigenvalues) The sequence (Ak )k 1 obtained with the Jacobi method converges and lim A = diag ((i ) ) k k where is a permutation of {1, ..., n}. Theorem 68 (Convergence of eigenvectors) We suppose that all eigenvalues of A are distinct. Then, the sequence (Ok )k 1 in the Jacobi method converges to an orthogonal matrix whose columns form an orthonormal set of eigenvectors of A.

9
10
11
12
13
Givens-Householder method I
Givens-Householder method is adapted to the research of selected eigenvalues of a symmetric matrix A, such as the eigenvalues lying in a given interval. Two steps 1 Determine an orthogonal matrix P such that P T AP is tridiagonal, with the Householder method. 2 Compute the eigenvalues of a tridiagonal symmetric matrix with the Givens method. Theorem 69 For a symmetric matrix A, there exists an orthogonal matrix P, product of n 2 Householder matrices Hk such that P T AP is tridiagonal: P = H1 H2 . . . Hn2
0 T H1 AH1 = 0 .
. .
. . .
. . .
... . . . . . . , . . .
0 T T H2 H1 AH1 H2 = 0 .
. .
0 . . .
. . .
0 0
... . . . . . . . . .
...

9
10
11
12
13
QR method I
The most commonly used method to compute the whole set of eigenvalues of an arbitrary matrix A, even nonsymmetric. QR algorithm Let A1 = A. For k 1, perform until convergence Ak = Qk Rk (QR factorization) Ak +1 = Rk Qk All matrices Ak are similar to matrix A. Under certain conditions, the matrix Ak converges towards a triangular matrix which is the Schur form of A, whose diagonal terms are the eigenvalues of A.

9
10
11
12
13
Power iterations method I
Power iteration method allows the capture of the dominant (largest magnitude) eigenvalue and associated eigenvector of a real matrix A. Power iteration algorithm Start with an arbitrary normalized vector x (0) and compute the sequence x (k +1) = and Theorem 70 Ax (k ) Ax (k )
(k +1) = (Ax (k ) , x (k ) )
If the dominant eigenvalue is real and of multiplicity 1, the sequences (x (k ) )k 0 and ( (k ) )k 0 respectively converge towards the dominant eigenvector and eigenvalue.
Power iterations method II
Proof. Let us prove the convergence of the method when A is symmetric. Then, there exists an orthonormal basis of eigenvectors (v1 , . . . , vn ), associated with eigenvalues (1 , . . . , n ). Let us consider that |1 | > |i | for all i > 1. The initial vector x (0) can be decomposed on this basis: x (0) = n i =1 ai vi and then, since Avi = i vi , Ax (k 1) Ak x (0) x (k ) = = ( k ) Ax Ak x (0) Ak x (0) =
n i =1 k (k ) (k ) ai k i vi = a 1 1 w , w = v1 + n i =2
ai a1
i 1
vi
and since w (k ) v1 , we obtain x (k ) =

(k ) a1 k 1w sign(a1 k 1 ) v1 , k ( a1 1 w k ) k
(k ) (Av1 , v1 ) = 1 k
Let us note that for general matrices, a proof using the Jordan form can be used.
Power iterations method III
Exercice. Power method with deation Under certain conditions, Power method with deation allows to compute the whole set of eigenvalues of a matrix. See exercices. Denition 71 (Inverse power method) For an invertible matrix A, applying the power method to matrix A1 allows to obtain the eigenvalue of A with smallest magnitude and the associated eigenvector (if the smallest magnitude eigenvalue is of multiplicity 1). Denition 72 (Shifted inverse power method) The shifted inverse power method consists in applying the inverse power method to the shifted matrix A = (A I ). It allows the capture of the eigenvalue (and associated eigenvector) which is the closest from the value . Indeed, if we denote by (vi , i ) the eigenpairs of matrix A, A has for eigenpairs (vi , i ). Therefore the inverse power method on A will converge towards the eigenvalue (i ) such that |i | = minj |j |.

9
10
11
12
13
Methods based on Krylov subspaces
A complete reference for the solution of eigenvalue problems Yousef Saad. Numerical Methods For Large Eigenvalue Problems. SIAM, 2011.
Fixed point Monotone operators Dierential calculus Newton method
Part V Nonlinear equations

14
Fixed point theorem Nonlinear equations with monotone operators Dierential calculus for nonlinear operators Newton method
15
16
17
Solving nonlinear equations
The aim is to introduce dierent techniques for nding the solution u of a nonlinear equation A (u ) = b , u K V where K is a subset of a vector space V and A : K V is a nonlinear mapping. We will equivalently consider the nonlinear equation F (u ) = 0, u K V where F : K V .
Innite dimensional framework
Denition 73 A Banach space V is a complete normed vector space. That means that this is a vector space (on complex or real elds) equipped with a norm and such that every Cauchy sequence with respect to this norm has a limit in V . Denition 74 A Hilbert space is a Banach space V whose norm is associated with an scalar (or hermitian) product (, ), with v 2 = (v , v ). Example 75 V = Rn equipped the natural euclidian scalar product is a nite-dimensional Hilbert space. V = Cn equipped the natural hermitian product is a nite-dimensional Hilbert space on complex eld.

14
15
16
17
Fixed point theorem I
We here consider nonlinear problems under the form T (u ) = u , u K V where T : K V is a nonlinear operator. Denition 76 A solution u of the equation T (u ) = u is called a xed point of mapping T . We are interested in the existence of a solution to equation (1) and in the possibility of approaching this solution by the following sequence (uk )k 0 dened by uk +1 = T (uk ) Remark. Let us note that nonlinear equations F (u ) = 0 can be recasted (in dierent ways) in the form (1), by letting T (u ) = F (u ) + u , T (u ) = F (u ) + u ,
...
(1)
Fixed point theorem II
Denition 77 Let V be a Banach space endowed with a norm . A mapping T : K V V is said contractive if there exists a constant , with 0 < 1, such that T (u ) T (v ) u v
is called the contractivity constant.
non-expansive
u , v K
if
T (u ) T (v ) u v
Lipschitz continuous
u , v K
if there exists a constant 0 such that

u , v K
T (u ) T (v ) u v
is called the Lipschitz-continuity constant.
Fixed point theorem III
Theorem 78 (Banach xed-point theorem) Assume that K is a closed set in a Banach space V and that T : K K is a contractive mapping with contractivity constant . Then, we have the following results: There exists a unique u K such that T (u ) = u For any u0 K, the sequence (uk )k 0 in K, dened by uk +1 = T (uk ), converges to u, i.e. u uk 0
k
Fixed point theorem IV

Proof.
Let us prove that {uk } is a Cauchy sequence. We have

u +
for
1, we then have
k 1 uk = m1
T u
( k ) T (uk 1 )
k k uk 1
1 u0
m 1
u +
m uk
=
1 Since [0, 1), um uk 0 as m, k , and therefore, {uk } is a Cauchy sequence. Since the sequence {uk } is Cauchy in a Banach space V , it converges to some u V and since K is closed, the limit u K . In the relation uk +1 = T (uk ), we take the limit k and obtain u = T (u ), by continuity of T . Then, u is a xed point of T . For the uniqueness, suppose that u1 and u2 are two xed points. Then we have
u
i =k k (1 mk )
i 1 ui
u
i =k
1 u0 =
k 1
k 1 u0
m1k i =0
1 u0
1 u0
2 u1 =
T u
( 2 ) T ( u1 )
2 u1
which is possible only if u2 = u1 .
Fixed point theorem V
Example 79 Let V = R and T (x ) = ax + b. If a = 1, the sequence xk +1 = T (xk ) is characterized by 1 ak xk = axk 1 + b = ak x0 + b 1a b , which is the unique xed point of T . If |a| > 1, If |a| < 1, xk converges to 1 a the sequence diverges. Let us note that
|T (x ) T ( x )| = |a||x x |
and therefore, we have that T is a contractive mapping if |a| < 1.

14
15
16
17
Nonlinear equations with monotone operators I

We consider the application of the xed point theorem to the analysis of solvability of a class of nonlinear equations A (u ) = b u V where V is a Hilbert space and A : V V is a Lipschitz continuous and strictly monotone operator. Denition 80 (Monotone operator) A mapping A : V V on a Hilbert space V is said monotone if (A(u ) A(v ), u v ) 0 u , v V strictly monotone if
(A(u ) A(v ), u v ) > 0
strongly monotone
u , v V , u = v
if there exists a constant > 0 such that

2
(A (u ) A (v ), u v ) u v is called the strong monotonicity constant.
u , v V
Nonlinear equations with monotone operators II
Theorem 81 Let V be a Hilbert space and A : V V a strongly monotone and Lipschitz continuous operator, with monotonicity constant and Lipschitz-continuity constant . Then, for any b V , there exists a unique u V such that A (u ) = b Moreover, if A(u1 ) = b1 and A(u2 ) = b2 , then u1 u2 1
b1 b2
which means that the solution depends continuously on the right-hand side b.
Nonlinear equations with monotone operators III

Proof. The equation A(u ) = b is equivalent to T (u ) = u , with T (u ) = u (A(u ) b) for any = 0. The idea is to prove that there exists a such that T : V V is contractive. The application of Banach xed point theorem will then give the existence and uniqueness of a xed point of u , and therefore, the existence and uniqueness of a solution to A(u ) = b. We have T (w ) T (v )
2
= (w v ) (A(w ) A(v )) = w v
2
2 2
2 (A(w ) A(v ), w v ) + 2 A(w ) A(v )

2
(1 2 + 2 2 ) w v
For 2 < 2/ 2 , we have (1 2 + 2 2 ) < 1, and T is a contraction. Now if A(u1 ) = b1 and A(u2 ) = b2 , we have A(u1 ) A(u2 ) = b1 b2 and
u1 u2
2
(A(u1 ) A(u2 ), u1 u2 ) = (b1 b2 , u1 u2 ) b1 b2
u1 u2
where the second inequality is the Cauchy-Schwartz inequality satised by the inner product of a Hilbert space. This proves the continuity of the solution u with respect to b.

14
15
16
17
Frchet and Gteaux derivatives I
Let F : K V W be a nonlinear mapping, where K is a subset of a normed space V and W a normed space. We denote by L(V , W ) the set of linear applications from V to W . Denition 82 (Frchet derivative) F is Frchet-dierentiable at u if and only if there exists A L(V , W ) such that F (u + v ) = F (u ) + Av + o ( v ) as v 0 A is denoted F (u ) and is called the Frchet derivative of F at u . If F is Frchet-dierentiable at all points in K , we denote by F : K V L(V , W ) the Frchet derivative of F on K . Property 83 If F admits a Frchet derivative F (u ) at u, then F is continuous at u.
Frchet and Gteaux derivatives II
Denition 84 (Gteaux derivative) F is Gteaux-dierentiable at u if and only if there exists A L(V , W ) such that F (u + tv ) F (u ) lim = Av v V (2) t 0 t A is denoted F (u ) and is called the Gteaux derivative of F at u . If F is Gteaux-dierentiable at all points in K , we denote by F : K V L(V , W ) the Gteaux derivative of F on K . Property 85 If a mapping F is Frchet-dierentiable, it is also Gteaux dierentiable and the derivatives F coincide. Conversely, if a mapping F is Gteaux-dierentiable at u and if F is continuous at u or if the limit in (2) is uniform with v such that v = 1, then F is also Frchet-dierentiable and the two derivatives coincide.
Convex functions I
Denition 86 A subset K of a vector space V is said convex if

u , v K , t [0, 1],
tu + (1 t )v K
Denition 87 A function J : K R, dened on a convex set K of V , is said convex if for all u , v K J (tu + (1 t )v ) tJ (u ) + (1 t )J (v ) t [0, 1]
strictly convex
if for all u , v K with u = v ,
J (tu + (1 t )v ) < tJ (u ) + (1 t )J (v ) t (0, 1)
Convex functions II
Theorem 88 Let J : K V R be Gateaux-dierentiable. The following statements are equivalent: (1) J is convex (2) J (v ) J (u ) + (J (u ), v u ), for all u , v K (3) J is monotone, i.e. (J (v ) J (u ), v u ) 0 , for all u , v K Theorem 89 Let J : K V R be Gateaux-dierentiable. The following statements are equivalent: (1) J is strictly convex (2) J (v ) > J (u ) + (J (u ), v u ), for all u , v K with u = v (3) J is strictly monotone, i.e. (J (v ) J (u ), v u ) > 0 , for all u , v K with u = v
Convex functions III
Denition 90 A function J : K V R is said strongly convex if it is Gateaux-dierentiable and if its Gteaux derivative is strongly monotone, i.e. if there exists a constant > 0 such that (J (v ) J (u ), v u ) u v 2
Convex optimization I
Theorem 91 Let K be a closed convex subset of an Hilbert space V . Assume that J : K R be a convex and Gteaux dierentiable mapping. Then, there exists u K such that J (u ) = inf J (v )
v K
(3)
if and only if there exists u K such that

(J (u ), v u ) 0 v K
(4) (5)
When K is a linear subspace, the last inequality reduces to

(J (u ), v ) = 0 v K
Convex optimization II
Proof.
Assume (3). Then v K and t [0, 1], J (u ) J (tv + (1 t )u ) tJ (v ) + (1 t )J (u ) and then J (u + t (v u )) J (u ) J (v ) J (u ) t (0, 1]
t
Taking the limit t 0+ , we obtain

(J (u ), v u ) J (v ) J (u ) 0
Now, assume (4). Since Finally, if

K
is convex, we have v K
J (v ) J (u ) + (J (u ), v u ) 0 is a subspace, then for all v K , u v K and therefore (J (u ), v ) 0 (J (u ), v ) = 0 v K

14
15
16
17
Newton method I
Let U and V be two Banach spaces and F : U V a Frchet-dierentiable function. We want to solve F (u ) = 0 The Newton method consists in constructing a sequence {un }nN by solving successive linearized problems. At iteration n, we introduce the linearization F of F at un , dened by
(v ) = F (un ) + F (un )(v un ) F (un+1 ) = 0. The Newton iterations are then and we dene un+1 such that F dened as follows.
Newton iterations Start from an initial guess u0 and compute the sequence {un }nN dened by un+1 = un F (un )1 F (un )
Newton method II
Theorem 92 (local convergence of Newton method) Assume u is solution of F (u ) = 0 and assume that F (u )1 exists and is a continuous linear map from V to U. Assume that F is locally Lipschitz continuous at u , i.e. F (u ) F (v ) L u v
u , v N (u )
where N (u ) is a neighborhood of u . Then, there exists > 0 such that if u0 u , the sequence {un }n1 of the Newton method is well-dened and converges to u . Moreover, there exists a constant M < 1/ such that un+1 u un u and un u (M )2 /M
n
2
Proof. See [Atkinson & Han (2009, section 5.4)]
Newton method for nonlinear systems of equations I

Let F : Rm Rm and consider the nonlinear system of equations F (u ) = 0 The iterations of the Newton method are dened by un+1 = un F (un )1 F (un ) where F (un ) Rmm is called the tangent matrix at un . In algebraic notations, F (u ) and F (u ) can be expressed as follows:
F1 (u ) . . . a1 F1 (a1 , . . . , am ) a 1. . . . . u= . , F ( u ) = , F ( u ) = . . . Fm am Fn (a1 , . . . , am ) (u ) . . . a1
F1 am
Fm am
. . .
(u ) (u )
Modied Newton method
One iteration of the (full) Newton method can be written as a linear system of equations An n = F (un ), n = un+1 un where An = F (un ). In order to avoid the computation of the tangent matrix F (un ) at each iteration, we can use modied Newton iterations where An is only an approximation of F (un ). For example, we could update An when the convergence is too slow or after every k iterations: An = F (um ) for n = mk + j , j {0, . . . , k 1} Remark. The convergence of the modied Newton method is usually slower that (full) Newton method but more iterations can be performed for the same computation time.
Interpolation Best approximation Orthogonal polynomials
Part VI Interpolation / Approximation

18
Interpolation Lagrange interpolation Hermite interpolation Trigonometric interpolation Best approximation Elements on topological vector spaces General existence results Existence and uniqueness of best approximation Best approximation in Hilbert spaces Orthogonal polynomials Weighted L2 spaces Classical orthogonal polynomials
19
20
Introduction
Principle of approximation The aim is to replace a function f , known exactly or approximately, by an approximating function p which is more convenient for numerical computation. The most commonly used approximating functions p are polynomials, piecewise polynomials or trigonometric polynomials. There are several ways of dening the approximating function among a given class of functions: interpolation, projection, ...
Lagrange interpolation Hermite interpolation Trigonometric inte

18
19
20

18
19
20
Preliminary denitions
We denote by Pn (I ) the space of polynomials of degre n dened on the closed interval I R: Pn (I ) = {v : I R; v (x ) =

n i =0
vi x i , vi R }
We denote by C (I ) the space of continuous functions f : I R. C (I ) is a Banach space when equipped with the norm f C (I ) = sup |f (x )|
x I
We denote by f the i -th derivative of f . We denote by C m (I ) the space of m times dierentiable functions f such that all its derivatives f (i ) of order i m are continuous. C m (I ) is a Banach space when equipped with the norm f C m (I ) = max f (i ) C (I ) i m
(i )
Lagrange interpolation
Let f C ([a, b]) be a continuous function dened on the interval [a, b]. We introduce a set of n + 1 distinct points {xi }n i =0 on [a, b ], such that a x0 < . . . < xn b The Lagrange interpolation pn Pn of f is the unique polynomial of degree n such that pn (xi ) = f (xi ) for all i {0, . . . , n} We can represent pn as follows: pn (x ) =
n i =0
f (xi ) i (x ),
i (x ) =
j =i n where the { i }i =0 form a basis of Pn , called the Lagrange interpolation basis. It
x xj x j =0 i xj
is the unique basis of functions satisfying the interpolation conditions

i (xj ) = ij
i , j {0, . . . , n}
Theorem 93 Assume f C n+1 ([a, b]). Then, for x [a, b], there exists x [a, b] such that f (x ) pn (x ) =
n (x ) (n+1) f (x ), (n + 1)! n (x ) =
n i =0
( x xi )
Inuence of the interpolation grid: Function wn (x ) on [1, 1] Gauss-Legendre grid (blue), Uniform grid (red), Random grid (black)
0.08 0.06 0.04 0.02 0 0.02 0.04 0.06 0.08 1 0.5 0 0.5 1
1.5
0.5
n=5
0.5 1
0.5
0.5
Theorem 93 Assume f C n+1 ([a, b]). Then, for x [a, b], there exists x [a, b] such that f (x ) pn (x ) =
n (x ) (n+1) f (x ), (n + 1)! n (x ) =
n i =0
( x xi )
Inuence of the interpolation grid: Function wn (x ) on [1, 1] Gauss-Legendre grid (blue), Uniform grid (red), Random grid (black)
2 1 0 1 2 3 4
0 0.5 1.5
x 10
5 6 1 0.5 0 0.5 1
n = 11
0.5 1
0.5
0.5
Lagrange interpolation: a famous example...

Runge function f (x ) = Uniform grid: n = 5, 11, 19
1.2 1.2
1 1+x 2
on [5, 5]
9 8
7
0.8 0.8
6 5
0.6
0.6
4
0.4 0.4
3 2 1
0.2
0.2
0
0.2 5 0.2 5 5
1 5 5
Gauss-Legendre grid: n = 5, 11, 19

1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 5 0 5
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 5 0 5
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 5 0 5

18
19
20
Hermite polynomial interpolation
First order interpolation
First order Hermite polynomial interpolation consists in interpolating a function f (x ) and its derivative f (x ). Assume f C 1 ([a, b]). We introduce a set of n + 1 distinct points {xi }n i =0 on [a, b ], with a x0 < . . . < xn b The hermite interpolant p2n+1 P2n+1 of f is uniquely dened by the following interpolation conditions: p2n+1 (xi ) = f (xi ), p2n+1 (xi ) = f (xi ), 0 i n
General Hermite polynomial interpolation
Higher order interpolation
Hermite interpolation can be generalized for the interpolation of higher order derivatives. At a given point xi , it interpolates the function and its derivatives up to the order mi N. Let N = n i =0 (mi + 1) 1. A generalized Hermite interpolant pN PN is uniquely dened by the following conditions
(j ) pN (xi ) = f (j ) (xi ), 0 j mi , 0 i n
Theorem 94 Assume f C N +1 ([a, b]). Then, for x [a, b], there exists x [a, b] such that f (x ) pN (x ) =
N (x ) (N +1) f (x ), (N + 1)! N (x ) =
n i =0
(x xi )mi

18
19
20
Trigonometric polynomials
A trigonometric polynomial is dened as follows pn (x ) = a0 +

n j =1
(aj cos(jx ) + bj sin(jx )) ,
x [0, 2)
pn is said of degree n if |an | + |bn | = 0. An equivalent notation is as follows: pn (x ) = with

n j =n
cj e ijx ,
a0 = c0 , aj = cj + cj , bj = i (cj cj ) or equivalently (under a polynomial-like form) pn (x ) =

n j =n
cj z j = z n
2n
k =0
ck n z k , z = e ix
Trigonometric interpolation
n We introduce 2n + 1 distinct interpolation points {xj }2 j =0 in [0, 2 ). Classically, we use uniformly distributed points
xj = j
2 , 0 j 2n 2n + 1
The trigonometric interpolant of degree n of function f is dened by the following conditions pn (xj ) = f (xj ), 0 j 2n It can be equivalently reformulated as an interpolation problem in the complex plane: nd {ck }n k =n such that
2n
k =0
ck n zjk = zjn f (xj ), 0 j 2n
where we have introduce complex points zj = e ixj .
Elements on topological vector spaces General existence results

18
19
20
The problem of the best approximation
The aim is to nd the best approximation p of a function f in a set of functions K (e.g. a polynomial space, piecewise polynomial space, ...)
pK
min f p
The obtained best approximation p depends on the norm selected for measuring the error (e.g. L2 -norm, L -norm, ...). We will rst introduce some general results about optimization problems
v K
inf J (v )
by giving some general conditions on the set K and the function J for the existence of a minimizer.
An rst comprehensive case: extrema of real-valued functions I

Consider a real-valued continuous function J C ([a, b]). The problem is to nd a minimizer of J inf J (v )
v [a,b]
The classical result of Weierstrass states that a continuous function on a closed interval K = [a, b] has a minimum in K (and a maximum). We recall the main steps of a typical proof in order to obtain more general requirements on K and J. 1 We denote by = inf J (v )
v K
By denition of the inmum, there exists a sequence {vn } K such that limn J (vn ) = . K is a closed and bounded interval in R, and therefore it is a compact set. Therefore, from the sequence {vn } K , we can extract a subsequence {vnk } which converges to some v K , vnk v
k
An rst comprehensive case: extrema of real-valued functions II

3
Using the continuity of J , we obtain J (v ) = lim J (vnk ) =

k
which proves that v is a minimizer of J in K . Now we come back on the dierent points of the proof in order to generalize the existence result for functionals J dened on a subset K of a Banach space V. 1 The existence of a minimizing sequence {vn } K is the denition of the inmum. 2 In an innite-dimensional Banach space V , a bounded sequence does not necessarily admits a converging subsequence. However, for a reexive Banach space V , there exists a weakly convergent subsequence. We then suppose that V is a reexive Banach space and K V is a bounded set. In order for K to contain the limit of this subsequence, K has to be weakly closed. 3 Finally, we want the weak limit of the subsequence to be a minimizer of J . We could then impose to J to be continuous with respect to a weak limit. However, this condition is too restrictive and it is sucient to impose that J is weakly lower semi continuous (allowing discontinuities).

18
19
20
Elements on topological vector spaces I
In the following, V denotes a normed space, i.e. a vector space equipped with a norm . Denition 95 (Strong convergence on V ) A sequence {vn } V is said to converge strongly to v V if
n
lim vn v = 0 vn v
It is denoted Denition 96 (Cauchy sequence) A sequence {vn } V is Cauchy if
n i ,j n
lim sup vi vj = 0
or equivalently, if > 0, there exists n N such that for all i , j n, vi vj .
Elements on topological vector spaces II
Denition 97 (Closed set) A subset K V is said to be closed if it contains all the limits of its convergent sequences:
{vn } K and vn v
v K
The closure K of a set K is the union of this set and of the limits of all converging sequences in K . Denition 98 (Compact set) A subset K of a normed space V is said to be (sequentially) compact if every sequence {vn }nN contains a subsequence {vnk }k N converging to an element in K . A set K whose closure K is compact is said relatively compact. Denition 99 (Banach space) A Banach space is a complete normed vector space, i.e. a normed vector space such that every Cauchy sequence in V has a limit in V .
Elements on topological vector spaces III
Denition 100 (Dual of a normed space V ) The dual space of a normed space V is set space V = L(V , R) of linear continuous maps from V to R. V is a Banach space for the norm L =
v V : v
sup
|L(v )| = sup v V
|L(v )|
LV
Denition 101 (Reexive normed space) A normed space V is said reexive if V = V , where V = (V ) is the dual of the dual of V , also called bidual of V . Denition 102 (Strong convergence on V ) A sequence {Ln } V is said to converge strongly to L V if
n
lim Ln L = 0
Elements on topological vector spaces IV
The dual space can be used to dene a new topology on V , called the weak topology. The notions of convergence, closure, continuity... can be redened with respect to this new topology. Denition 103 (Weak convergence on V ) A sequence {vn } V is said to converge weakly to v V if
n
lim L(v vn ) = 0 L V vn v
It is denoted
Denition 104 (Weakly closed set in V ) A subset K V is said to be weakly closed if it contains all the limits of its weakly convergent sequences:
{vn } K and vn
v K
Elements on topological vector spaces V
Denition 105 (Weakly compact set) A subset K of a normed space V is said to be weakly compact if every sequence {vn }nN contains a subsequence {vnk }k N weakly converging to an element in K . A set K whose closure in the weak topology is weakly compact is said weakly relatively compact. Theorem 106 (Reexive Banach spaces and converging bounded sequences) A Banach space V is reexive if and only if every bounded sequence in V has a subsequence weakly converging to an element in V . Let us note that the above theorem could be reformulated as follows: a Banach space is reexive if and only if the unit ball is relatively compact in the weak topology. Theorem 107 A set K in V is bounded and weakly closed if and only if it is weakly compact.
Lower semicontinuity I
Denition 108 (Lower semicontinuity) A function J : V R is lower semicontinuous (l.s.c.) if

{ vn } K
and vn v K
J (v ) lim inf J (vn )

n
Denition 109 (Weak lower semicontinuity) A function J : V R is weakly lower semicontinuous (w.l.s.c.) if
{ vn } K
and vn
v K
J (v ) lim inf J (vn )

n
Proposition 110 Continuity implies lower semicontinuity (but the converse statement is not true) Weak lower semicontinuity implies lower semicontinuity (but the converse statement is not true)
Lower semicontinuity II
Example 111 Let us prove that the norm function . : v V v R in a normed space V is w.l.s.c. Let {vn } V be a weakly convergent sequence with vn v . There exists a linear form L V such that L(v ) = v and L = 1 (Corollary of the Generalized Hahn-Banach theorem). We then have L(vn ) L vn = vn and therefore v = L(v ) = lim L(vn ) lim inf vn
n n
If V is an inner product space, we have a simpler proof. Indeed, v

2
= (v , v ) = lim (vn , v ) lim inf v n n
vn

18
19
20
General existence results I

We introduce the problem
v K
inf J (v )
()
Theorem 112 Assume V is a reexive Banach space. Let K V denote a bounded and weakly closed set. Let J : V R denote a weakly l.s.c. function. Then, problem () has a solution in K. Proof. Denote = inf v K J (v ) and {vn } K a minimizing sequence such that limn J (vn ) = . Since K is bounded, {vn } is a bounded sequence in a reexive Banach space and therefore, we can extract a subsequence {vnk } weakly converging to some u V . Since K is weakly closed, u K . Since J is w.l.s.c. J (u ) lim inf J (vnk ) = and therefore, u K is a minimizer of J .
k
General existence results II
We now remove the boundedness of the set K by adding a coercivity condition on J . Denition 113 A functional J : V R is said coercive if J (v ) + as Theorem 114 Assume V is a reexive Banach space. Let K V denote a weakly closed set. Let J : V R denote a weakly l.s.c. and coercive function. Then, the problem () has a solution in K. v
General existence results III
Proof. Pick an element v0 K with J (v0 ) < and let K0 = {v K ; J (v ) J (v0 )}. Since J is coercive, K0 is bounded. Moreover, K0 is weakly closed. Indeed, if {vn } K0 is such that vn v , then v K (since K is weakly closed) and J (v ) lim inf n J (vn ) J (v0 ), and therefore v K0 . The optimization problem is then equivalent to the optimization problem
v K0
inf J (v )
of a w.l.s.c. function on a bounded and weakly closed set. Theorem 112 allows to conclude on the existence of a minimizer. Lemma 115 (Convex closed sets are weakly closed) A convex and closed set K V is weakly closed. Lemma 116 (Convex l.s.c. functions are w.l.s.c.) A convex and l.s.c. function is also w.l.s.c.
General existence results IV
For convex sets and convex functions, theorems 112 and 114 can then be replaced by the following theorem. Theorem 117 Assume V is a reexive Banach space. Let K V denote a convex and closed set. Let J : V R denote a convex l.s.c. function. Then, if either (i) K is bounded, or (ii) J is coercive on K, then the minimization problem () has a solution in K. Moreover, if J is strictly convex, this solution is unique. Proof. The existence simply follows from theorems 112 and 114 and from properties 115 and 116. It remains to prove the uniqueness if J is strictly convex. Assume that u1 , u2 K are two solutions such that u1 = u2 . We have J (u1 ) = J (u2 ) = minv K J (v ). Since K is convex, u1 + (1 )u2 K for (0, 1), and by strict convexity of J , we have J (u1 + (1 )u2 ) < J (u1 ) + (1 )J (u2 ) = min J (v )
v K
which contradicts the fact that u1 and u2 are solutions.
General existence results V
In the case of a non reexive Banach space V (e.g. V = C ([a, b])) the above theorems do not apply. However, the reexivity is used for the extraction of a weakly convergent subsequence from a bounded sequence in K . In fact, we just need the completeness of the set K and not of the space V . In particular, for nite-dimensional subset K , we have. Theorem 118 Assume V is a normed space. Let K V denote a nite-dimensional convex and closed set. Let J : V R denote a convex l.s.c. function. Then, if either (i) K is bounded, or (ii) J is coercive on K, then the minimization problem () has a solution in K. Moreover, if J is strictly convex, this solution is unique.

18
19
20
Existence and uniqueness of best approximation I
We apply the general results about optimization on the following best approximation problem. For a given element u V , where V is a normed space, we want to nd the elements in a subset K V which are the closest to u . The problem writes inf u v
v K
Denoting J (v ) = u v , the problem can then be written under the form inf v K J (v ). Property 119 Function J (v ) = u v is convex, continuous (and hence w.l.s.c), and coercive. We then have the two existence results. Theorem 120 Let V be a reexive Banach space and K V a closed convex subset. Then there exists a best approximation u K verifying uu = min u v
v K
Existence and uniqueness of best approximation II
Theorem 121 Let V be a normed space and K V a nite-dimensional closed convex subset. Then there exists a best approximation u K verifying uu = min u v
v K
For the uniqueness of the best approximation, we have to look at the properties of the norm. Theorem 122 I there exists a p > 1 such that v v p is strictly convex, then a solution u of the best approximation problem is unique. Example 123 If V is a Hilbert space equipped with the inner product (, ) and associated norm , v v 2 is a strictly convex function. If V = Lp () with p (1, +), v v p Lp () is strictly convex.

18
19
20
Best approximation in Hilbert spaces I
Let V be a Hilbert space equipped with inner product (, ) and associated norm . Lemma 124 Let K be a closed convex set in Hilbert space V . u K is a best approximation of u V if and only if
(u u , v u ) 0 v K
Best approximation in Hilbert spaces II
Proof. First suppose that u K is a best approximation of u V . Then, u u

2
w u
w K . By selecting w = u + (v u ), with (0, 1) and v K , we have
0 u u 2 ( u u ) + (v u ) 2 = 2 (v u , v u ) 2( u u, v u ) for all (0, 1). That implies ( u u, v u ) 0 v K . Conversely, if ( u u, v u ) 0, v K , then v u

2
= (v u ) + ( u u) = v u u u
2 2
2 2
+ 2(v u , u u) + u u
for all v K .
Best approximation in Hilbert spaces III
Corollary 125 Let K be a closed convex set in Hilbert space V . For any u V , the best approximation in K is unique. Proof. Let u 1 , u 2 K be two best approximations of u V . Then, (u u 1 , u 2 u 1 ) 0 and (u u 2 , u 1 u 2 ) 0. Additionning these inequalities, we obtain
( u2 u 1 , u 2 u 1 ) = u 2 u 1
2
and therefore u 1 = u 2 . We then conclude with the following theorem.
Best approximation in Hilbert spaces IV
Theorem 126 Let K V be a nonempty closed convex set in Hilbert space V . For any u V , there exists a unique best approximation u K dened by uu = min u v
v K
Best approximation in Hilbert spaces V
Remark. Let us give another classical proof for the existence of a best approximation, which uses the inner product structure of the space V . Let {un }nN K be a minimizing sequence such that limn u un = = inf v K u v . Using the parallelogram law satised by the norm . in an inner product space, we have 2 u un 2 + 2 u um 2 = un um 2 + 2u un um 2 Since K is convex, we have (un + um )/2 K and therefore un um
2
= 2 u un 2 u un
2 2
+ 2 u um + 2 u um
2 2
4 u (un + um )/2 42 0 m,n
which proves that {un } is a Cauchy sequence. Since V is complete, {un } K converges to an element u V and since K is closed, u K.
Best approximation in Hilbert spaces: Projection I
Denition 127 (Projector on a convex set) The best approximation u K of u V in a closed convex set K is called the projection of u onto K and is denoted u = PK (u ) where PK : V K is called the projection operator of V onto K . Proposition 128 The projection operator is monotone
( PK ( v ) PK ( u ) , v u ) 0 u , v V
and non expansive PK ( v ) PK ( u ) v u

u , v V
Best approximation in Hilbert spaces: Projection II
Proof. From the characterizations of PK (u ) K and PK (v ) K , we have respectively

(PK (u ) u , PK (v ) PK (u )) 0, (PK (v ) v , PK (u ) PK (v )) 0
Adding these inequalities, we obtain

(v u , PK (v ) PK (u )) (PK (v ) PK (u ), PK (v ) PK (u )) 0
and PK ( v ) PK ( u )
2
(v u , PK (v ) PK (u )) v u
PK (v ) PK (u )
We now introduce the following particular case when K is a subspace of V .
Best approximation in Hilbert spaces: Projection III
Theorem 129 (Projection on linear subspaces) Let K be a complete subspace of V . Then, for any u V , there exists a unique best approximation u = PK (u ) K characterized by
(u PK (u ), v ) = 0 v K
Proof. We have
(u u , w u ) 0 (u u , v ) 0
w K v K
and since K is a subspace, for all v K , w = u v K , and therefore
In the case where K is a subspace, u PK (u ) is orthogonal to K , and therefore, PK is called an orthogonal projection operator.
Best approximation in Hilbert spaces: Projection IV
Let us consider that we know an orthonormal basis {i }n i =1 of K = Kn . The projection PKn (u ) is characterized by PKn (u ) =
n i =1
(i , u )i
Example 130 (Least square approximation by polynomials) Let V = L2 (1, 1) and Kn = Pn (1, 1) the space of polynomials of degree less than n. An orthonormal basis of Kn is given by the Legendre polynomials {Li }n i =0 dened by Li ( x ) = 1 di (2i + 1)/2 i 2 i ! dx i
(x 2 1)i
Best approximation in Hilbert spaces: Projection V
Example 131 (Least square approximation by trigonometric polynomials) Let V = L2 (0, 2) and Kn the space of trigonometric polynomials of degree less than n. The best approximation u n = PKn (u ) is characterized by u n (x ) = a0 /2 + with aj = 1 1 (u (x ), cos(jx )) = (cos(jx ), cos(jx )) 1 1 bj = (u (x ), sin(jx )) = (sin(jx ), sin(jx ))
2 0 0 2
n j =1
(aj cos(jx ) + bj sin(jx ))
u (x ) cos(jx )dx , j 0 u (x ) sin(jx )dx , j 1
Note that u n tends to the well-known Fourier series expansion of u .
Weighted L2 spaces Classical orthogonal polynomials

18
19
20

18
19
20
Weighted
L2
spaces
Let I R and : I R be a weight function which is integrable on I and almost everywhere positive. We introduce the weighted function space L2 (I ) = {v : I R; v is measurable on I , L2 (I ) is a Hilbert space for the inner product
(u , v ) =
|v (x )|2 (x )dx < +}
u (x )v (x )(x )dx u (x )2 (x )dx

1/2
and associated norm u =

2
Two functions u , v L (I ) are said orthogonal if (u , v ) = 0.

18
19
20
Classical orthogonal polynomials I
A system of orthonormal polynomials {pn }n0 , with pn Pn (I ), can be constructed by applying the Gram-Schmidt procedure to the basis of monomials {1, x , x 2 , . . .}. For a given interval I and weight function , it leads to a uniquely dened system of polynomials. In the following table, we indicate classical families of polynomials for dierent interval domains I and weight functions. Classical orthogonal polynomials I
(1, 1) (1, 1) (1, 1) (1, 1) R (0, +)
(1+x )a1 (1x )b1 2a+b1 B (a,b) (1x 2 )1/2 B (1/2,1/2) (1x 2 )1/2 4B (3/2,3/2)
(x )
1 2
{pn }
exp ( x2 ) x a exp (x ) (a)

1
2 1
Jacobi Legendre Chebyshev of rst kind Chebyshev of second kind Hermite Laguerre
Classical orthogonal polynomials II
denotes the Euler Gamma function dened by (a) =
x a1 exp (x )dx
B (a, b) denotes the Euler Beta function dened by B (a , b ) =

(a)(b ) (a + b )
Remark. The given weight functions are such that

I
(x )dx = 1
It then denes a measure with density (d (x ) = (x )dx ) and with unitary mass. Equivalently, (resp. ) can be interpreted as the probability law (resp. probability density function) of a random variable.
Classical orthogonal polynomials III
Exercice. Construct by the Gram-Schmidt procedure the orthonormal polynomials of degree n = 0, 1, 2 on the interval I = (0, 1) and for the weight function (x ) = log(1/x ).
Basic quadrature formulas Gauss quadrature
Part VII Numerical integration
21
Basic quadrature formulas
22
Gauss quadrature
Numerical integration
Given a function f : R, the aim is to approximate the value of the integral I (f ) =
f (x )dx
n
using evaluations of the function I (f )

k =1
f (xk )k
or eventually of the function and its derivatives I (f )

n k =1
f (xk )k +
n k =1
f ( xk ) k + . . .
These approximations are called quadrature formulas. A quadrature formula is said of interpolation type if it uses only evaluations of the function.
Integration error and precision
We denote by In (f ) the quadrature formula. Denition 132 A quadrature formula have a degree of precision k if it integrates exactly all polynomials of degree less or equal to k In (f ) = I (f ) f Pk () In (f ) = I (f ) for some f Pk +1 ()
21
22
Gauss quadrature
Rectangle formula (precision degree 1)

b a b a b
f (x )dx (b a)f (
a+b ) 2
Trapezoidal formula (precision degree 1) f (x )dx (b a) f (a ) + f (b ) 2 a+b ) + f (b )) 2
Simpson formula (precision degree 3)

a
f (x )dx
(b a )
(f (a) + 4f (
...
Composite quadrature formulas
In order to compute I (f ; ) =
f (x )dx ,
m i =1 m i =1
we divide the domain into m subdomains {}m i =1 such that I (f ; ) = I (f ; i )
and we introduce a basic quadrature formula on each subdomain I (f ; ) I n (f ; i )
21
22
Gauss quadrature
Gauss quadrature I
We want to approximate the weighted integral of a function f I w (f ) =

b a
f (x )w (x )dx
where w (x )dx denes a measure of integration. A Gauss quadrature formula with n points is dened by
w (f ) = I w (f ) In n i =1
i f (xi )
with points and weights such that it integrates exactly all polynomials f P2n1 (a, b). The xi (resp. i ) are called Gauss points (resp. Gauss weights) associated with the present measure. We introduce the function space L2 w (a, b ) and its natural inner product
(f , g )w =
b a
f (x )g (x )w (x )dx
Gauss quadrature II
Theorem 133 In (f ) = I (f ) for all f P2n1 (a, b) if and only if the points xi are such that the polynomial z n (x ) = n i =1 (x xi ) Pn (a, b ) is orthogonal to Pn1 (a, b ), i.e.
(zn (x ), p (x ))w = 0 p Pn1 (a, b )
the weights are dened by i = I (Li ), where Li is the Lagrange interpolant at xi , dened by Li (x ) = n j =1,j =i (x xj )/(xi xj ) Corollary 134 The n Gauss points of a n-points Gauss quadrature are the n roots of the degree n orthogonal polynomial. For (a, b) = (1, 1) and w (x ) = 1, the xi are the n roots of the degree n Legendre polynomial. For (a, b) = (, ) and w (x ) = exp(x 2 ), the xi are the n roots of the degree n Hermite polynomial. ...

M1 Numerical Analysis Course

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

M1 Numerical Analysis Course

Uploaded by

Copyright:

Available Formats

M1 SMA International Ecole Centrale de Nantes

Origin of problems in numerical analysis References

Origin of problems in numerical analysis

Origin of problems in numerical analysis References

Origin of problems in numerical analysis

Origin of problems in numerical analysis I

Origin of problems in numerical analysis References

Example 1 (1D diusion equation, beam in traction, ...)

d du ( ) = b (x ) for x = (0, 1), u (0) = u (1) = 0 dx dx

Origin of problems in numerical analysis II

Origin of problems in numerical analysis References

and replace function space V by approximation space V n = {v (x ) = n i =1 vi hi (x )} V

Origin of problems in numerical analysis III

Origin of problems in numerical analysis References

In order to construct the system of equation (matrix A and right-hand-side b):

Origin of problems in numerical analysis IV

Origin of problems in numerical analysis References

d du ((x , u ) ) = b (x , u ) for x = (0, 1), u (0) = u (1) = 0 dx dx

Eigenproblems Find (u, ) Cn C such that

where A, B Cnn are matrices.

Origin of problems in numerical analysis V

Origin of problems in numerical analysis References

for x = (0, 1), u (0, t ) = u (1, t ) = 0

for which we search solutions of the form u (x , t ) = w (x ) cos(t ): w Vn ,

Ordinary dierential equations in time d u(t ) + A(u(t ); t ) = b(t ) dt

Origin of problems in numerical analysis References

Origin of problems in numerical analysis

References for the course

Origin of problems in numerical analysis References

K. Atkinson and W. Han. Springer, 2009.

Theoretical Numerical Analysis: A Functional Analysis Framework

An Introduction to Numerical Analysis

a clear and simple presentation of all the ingredients of the course

additional material for numerical solution of PDE and optimization

problems a natural continuation of the course

Matrices Reduction of matrices Vector and matrix norms

Part II Linear algebra

Matrices Reduction of matrices Vector and matrix norms

Matrices Reduction of matrices Vector and matrix norms

Part II Linear algebra

Matrices Reduction of matrices Vector and matrix norms

Matrices Reduction of matrices Vector and matrix norms

where a denotes the complex conjugate of a.

Canonical inner product

Matrices Reduction of matrices Vector and matrix norms

We denote by (, ) : V V K the canonical inner product dened for all u , v V by

It is called euclidian inner product if K = R and hermitian inner product if

Matrices Reduction of matrices Vector and matrix norms

Matrices Reduction of matrices Vector and matrix norms

where the coecients aij are such that

Matrices Reduction of matrices Vector and matrix norms

We denote AT the transpose of a real matrix A = (aij ) Rnm , dened by

We have the following characterization of AH and AT :

Matrices Reduction of matrices Vector and matrix norms

Matrices Reduction of matrices Vector and matrix norms

Matrices Reduction of matrices Vector and matrix norms

Matrices Reduction of matrices Vector and matrix norms

Properties of triangular matrices

Matrices Reduction of matrices Vector and matrix norms

Matrices Reduction of matrices Vector and matrix norms

Matrices Reduction of matrices Vector and matrix norms

Matrices Reduction of matrices Vector and matrix norms

Matrices Reduction of matrices Vector and matrix norms

Eigenvalues and eigenvectors I

Example 1 (1D diusion equation, beam in traction, ...)

Ordinary dierential equations in time d u(t ) + A(u(t ); t ) = b(t ) dt

We denote by (, ) : V V K the canonical inner product dened for all u , v V by

where the coecients aij are such that

We denote AT the transpose of a real matrix A = (aij ) Rnm , dened by

Denition 17 Eigenvalues i = i (A), 1 characteristic polynomial

Denition 18 The spectral radius (A) of a matrix A is dened by