Introduction to Linear Algebra

Matrix
representation
of size n X d

Symmetric Matrix

Diagonal Matrix

Matrix multiplication with a vector

gives a transformed vector

Inner Product
Norm of a
vector

Cosine of the angle

between 2 vectors

Cauchy Schwartz
Inequality

Linear Independence of vectors / Basis

vectors

Outer Product of 2 vectors

Rank of a matrix
Number of independent rows / columns of a
matrix
Outer product between 2 vectors gives a
matrix of rank 1.

Gradient of a scalar function

Gradient of a vector-valued
function (Jacobian)

Matrix / Vector Derivatives

Gradient of
Vector function

Gradient of scalar
function

Eigen values and vectors

Diagonalization property
M VDV

Matrix, whose columns correspond to

Eigenvectors of

Diagonal matrix, corresponding to

eigenvalues of

Rank deficient square matrices

A matrix A of rank r will have r independent
rows / columns, and is said to be rankdeficient.
There will be r independent eigen
vectorscorresponding to r non- zero eigen
values
r eigen values are zero.

Matrix Inverse

For square matrix M

Orthogonal Matrix
{q1 , q2 ,.....qM }

M vectors

1 i j
qi q j
0 i j
T
Q QI
T

columns of Q denote

q1 q2 ..... qM

Q is of size M XM and is said to be

an orthogonal matrix

Positive Definite Matrix

Positive definite square
matrix A

xT Ax 0

x denotes a vector of size d x 1 .

A denotes a vector of size d x d .

Eigen values of positive definite matrix A are
non-negative.

Real symmetric matrix

Elements of a matrix are real numbers
Matrix is symmetric
Eigen values of a real symmetric matrix are
real.

Probability Theory

Random variables
A random variable x denotes a quantity that is
uncertain
May be result of experiment (flipping a coin) or a real
world measurements (measuring temperature)
If observe several instances of x we get different
values
Some values occur more than others and this
information is captured by a probability distribution

19

Discrete Random Variables

20

Continuous Random Variable

23

Expectation/ Variance of continuous

random variable

Joint Probability
Consider two random variables x and y
If we observe multiple paired instances, then some
combinations of outcomes are more likely than
others
This is captured in the joint probability distribution
Written as P(x,y)
Can read P(x,y) as probability of x and y

25

For 2 random variables x and y,

Marginalization property

26

Marginalization
We can recover probability distribution of any variable in a joint distribution
by integrating (or summing) over the other variables

27

Marginalization
We can recover probability distribution of any variable in a joint distribution
by integrating (or summing) over the other variables

28

Marginalization
We can recover probability distribution of any variable in a joint distribution
by integrating (or summing) over the other variables

Works in higher dimensions as well leaves joint distribution between

whatever variables are left

29

Conditional Probability
Conditional probability of x given that y=y1 is relative
propensity of variable x to take different outcomes given that
y is fixed to be equal to y1.
Written as Pr(x|y=y1)

30

Conditional Probability
Conditional probability can be extracted from joint probability
Extract appropriate slice and normalize

31

Conditional Probability
More usually written in compact form

Can be re-arranged to give

32

Conditional Probability

This idea can be extended to more than two

variables

33

Bayes Rule
From before:

Combining:
Re-arranging:

34

Bayes Rule Terminology

Likelihood propensity for
observing a certain value of
x given a certain value of y

Posterior what we
know about y after
seeing x

Prior what we know

about y before seeing x

Evidence a constant to
ensure that the left hand
side is a valid distribution
35

Independence
If two variables x and y are independent then variable x tells
us nothing about variable y (and vice-versa)

36

Independence
When variables are independent, the joint factorizes into a
product of the marginals:

37

Discrete Random vectors

Continuous Random vectors

Continuous random vectors

Discrete random vectors

Properties of covariance matrix

1 dimensional Gaussian

2 dimensional Gaussian

Multi dimensional Gaussian