INTRODUCTION TO

PROBABILITY
and
BAYES THEOREM

Introduction to probability

Basics of probability
Picturing probability: Venn diagrams, Tree
diagrams
Probability rules
Complement rule
Addition rules
Multiplication rules
Conditional probability
Independence
Reversing the conditioning: Bayes Theorem
Dealing with Random Phenomena
A random phenomenon is a situation in which we
know what outcomes could happen, but we dont
know which particular outcome did or will
happen.
When dealing with probability, we will be dealing
with many random phenomena
Probability
Probability refers to the chance that a particular
event will occur.
The probability of an event will be a value in the range
0.00 to 1.00.
A value of 0.00 means the event will not occur
(impossibility).
A probability of 1.00 means the event will occur
(certainty).
Anything between 0.00 and 1.00 reflects the
uncertainty of the event occurring.
Events and Sample Space:
Experiment
An experiment is a process that produces
a single outcome whose result cannot be
predicted with certainty.
Events and Sample Space:
Elementary Events
Elementary events are the most
rudimentary outcomes resulting from a
simple experiment.
Events and Sample Space
The sample space (S) is the collection of all
elementary outcomes that can result from
an experiment.
Events and Sample Space
An event is a collection of
elementary events.
Example 1:
Sample space and events
An experiment consists of randomly drawing
three components from a batch production and
checking whether they are defective or not.
Lets define the events:
A: acceptable component
D: defective component
Determine the sample space S for this experiment
(tree diagram)
A
D
Example 1
Sample space and events
1
st
drawing
A
D
A
D
A
D
1
st
drawing 2nd drawing
Example 1
Sample space and events
A
D
A
D
A
D
D
A
A
D
A
D
A
D
1
st
drawing 2
nd
drawing 3
rd
drawing
Example 1
Sample space and events
A
D
A
D
A
D
D
A
A
D
A
D
A
D
Sample space (S)
AAA
AAD
ADA
ADD
DAA
DAD
DDA
DDD
E
i
g
h
t

e
l
e
m
e
n
t
a
r
y

e
v
e
n
t
s

1
st
drawing 2
nd
drawing 3
rd
drawing Possible outcomes
Example 1
Sample space and events
B and C are two events defined as follows:

B: three components are acceptable
B = {(AAA)}

C: at least one component is acceptable
C = {(ADD),(DAD),(DDA),(AAD),(ADA),(DAA),(AAA)}
Example 1
Sample space and events
Picturing Probabilities
Tree diagrams can be used to define the sample space of an
experiment
Also, the most common kind of picture to make is called a
Venn diagram.





We will see Venn diagrams in practice shortly

Mutually Exclusive Events (or
Disjoint Events)
Two events are mutually exclusive if the
occurrence of one event precludes the
occurrence of another event.
A
S
B
B: Three components are acceptable
B = {(AAA)}
Let E: at least one component is defective
E = {(AAD),(ADA),(DDA),(ADD),(DAA),(DAD),(DDD)}

B and E are disjoint or mutually exclusive events


Example 1 (cont.)
Two disjoint events
Trial, Experiment


Sample space (S) = all possible outcomes



Trial, Experiment


Sample space (S) = all possible outcomes


A combination of outcomes
Event (E
i
)
Elementary: one outcome of S


A B
A B
Trial, Experiment


Sample space (S) = all possible outcomes


A combination of outcomes
Event (E
i
)
Elementary: one outcome of S
Disjoint = No outcomes in common Non disjoint = can occur together,
= Mutually exclusive outcomes in common
S A S B
Four Types of Probability
Marginal

The probability
of A occurring
Union

The probability
of A or B
occurring
Joint

The probability
of A and B
occurring
Conditional

The probability
of A occurring
given that B
has occurred
B A B A
B
A
) ( A P
) ( B A P
) ( B A P
) | ( B A P
Independent and Dependent
Events
Two events are independent if the
occurrence of one event in no
way influences the probability of
the occurrence of the other event.
See later Probability Rules 4 and 4a
Being
independent is
a totally
different thing
that being
mutually
exclusive
Independent and Dependent
Events
Two events are dependent if the
occurrence of one event impacts the
probability of the other event occurring.
See later Probability Rules 4 and 4a
Assessing Probabilities
Theoretical: Number of Possibilities
Relative Frequency
Subjective
Theoretical Probability
Assessment
Theoretical Probability Assessment refers to the method
of determining probability based on the ratio of the
number of ways the event of interest can occur to the
total number of ways any event can occur when the
individual elementary events are equally likely.
THEORETICAL PROBABILITY MEASUREMENT




Theoretical Probability
Assessment
events elementary of number Total
occur can E ways of Number
) P(E
i
i
=
What is the probability that three components are
acceptable?
B: three components are acceptable
B = {(AAA)}
P(B) = 1/8
What is the probability that at least one component
is acceptable?
C: at least one component is acceptable
C = {(ADD),(DAD),(DDA),(AAD),(ADA),(DAA),(AAA)}
P(C) = 7/8
Example 1 (cont.)
Theoretical probability assessment
Relative Frequency of
Occurrence
Relative Frequency of Occurrence refers to a method
that defines probability as the number of times an
event occurs, divided by the total number of times an
experiment is performed in a large number of trials.
Relative Frequency of
Occurrence
RELATIVE FREQUENCY OF OCCURRENCE



where:
E
i
= the event of interest
RF(E
i
) = the relative frequency of E
i
occurring
n = number of trials
n
occurs E times of Number
RF(E
i
i
= )
Relative Frequency of
Occurrence
Relative Frequency of Occurrence
Example 2
On a production line products are tested
throughout a day.
Out of 10000 products, 52 are found to be
defective.

Probability of defective product
= 52/10000
= 0.52%

Law of Large Numbers (LLN)
If the events are independent, then as the number of
trials increases, the long-run relative frequency of an
event gets closer and closer to a single value.
Subjective Probability
Assessment
Subjective Probability Assessment refers to the
method that defines probability of an event as
reflecting a decision makers state of mind
regarding the chances that the particular event
will occur.
Partner in consulting company puts in a bid for
a contract, and subjectively assesses the chance
of getting the contract as 70%.
The Rules of Probability
PROBABILITY RULE 1

For any event E
i

0.0 s P(E
i
) s 1.0 for all i
The Complement of an Event
The complement of an event E is the
collection of all possible elementary
events not contained in event E. The
complement of event E is represented by
E or E
C
or E

The complement of E is the same as not E
The Rules of Probability
PROBABILITY RULE 2




) ( 1 ) (
E
P E
C
P
=
E E
C
Sample Space
We pick a component at random from a
production line
The inspection reveals that a component can be
good (G) or defective (D)

Two disjoint outcomes are possible : S = {G,D}
G = D
C
and D = G
C

If P(G) = 0.90 determine P(D) = ?
P(D) = P(S) - P(G) = 1 - P(G) = 1 0.90 = 0.10
Example 3:
The complement rule
The Rules of Probability
PROBABILITY RULE 3

Addition or OR rule for any two events E
1
and E
2
:

P(E
1
or E
2
) = P(E
1
) + P(E
2
) - P(E
1
and E
2
)

E1 E2
2
E
1
E and
The Rules of Probability
SPECIAL CASE OF THE OR RULE

OR rule for mutually exclusive (disjoint) events E
1
and E
2
:
P(E
1
or E
2
) = P(E
1
) + P(E
2
)

Let A and B two disjoint events of S, with P(A) =
0.4 and P(B) = 0.45, we have:
Example 4
Addition rule for two disjoint events
0.4 = 0 - 0.4 = B) and P(A - P(A) = ) B and P(A
0.15 = 0.85 - 1 =
B) or P(A - 1 = B) or P(A = ) B and P(A
0.85 = 0.45 + 0.4 = P(B) + P(A) = B) or P(A
0 = B) and P(A
0.6 = 0.4 - 1 = P(A) - 1 = ) P(A
C
C C C
C
A S B
400 students were recently interviewed concerning
the newspapers they read.
The study revealed that:
A: 165 read The Citizen"
B: 240 read The National Post"
A and B: 90 read both
Example 5
Addition rule for any two events
A B
A B
400 students were recently interviewed concerning
the newspapers they read.
The study revealed that:
A: 165 read The Citizen"
B: 240 read The National Post"
A and B: 90 read both
What is the probability that a student reads The
Citizen; The National Post
Example 5
Addition rule for any two events
A B
A B
400 students were recently interviewed concerning
the newspapers they read.
The study revealed that:
A: 165 read The Citizen"
B: 240 read The National Post"
A and B: 90 read both
What is the probability that a student reads The
Citizen; The National Post

Example 5
Addition rule for any two events
A B
A B
0.60
400
240
P(B) and 0.4125
400
165
P(A) = = = =
Example 5
Addition rule for any two events
How many students read only The Citizen (A); only
the National Post (B); None of them?

A B
B: Nat. Post B
C
:(Nat. Post)
C
A : The Citizen 90 165
A
C
: (The Citiz.)
C

240 400
A B
A B
Example 5
Addition rule for any two events
How many students read only The Citizen (A); only
the National Post (B); None of them?

A B
B: Nat. Post B
C
:(Nat. Post)
C
A : The Citizen 90 75 165
A
C
: (The Citiz.)
C
150 85 235
240 160 400
A B
A B
Illustrating using Venn diagram:
75
S
85
90
150
Example 5
Addition rule for any two events
A B
A B
What is the probability that a student reads either
of these two newspapers? (Think OR)

Example 5
Addition rule for any two events
A B
A B
78.75% =
400
315
=
400
90
-
400
240
+
400
165
=
B) and P(A - P(B) + P(A) = B) or P(A
Conditional Probability
Conditional probability refers to the
probability that an event will occur given
that some other event has already
happened.
The Rules of Probability
PROBABILITY RULE 4

Conditional probability for any two events E
1
, E
2
:




0 ) (
) (
) (
) | (
2
2
2 1
2 1
>
=
E P
E P
E and E P
E E P
Definition of Independence
Independence of two events means that the
outcome of one event does not influence the
probability of the other.
With our new notation for conditional
probabilities, we can now formalize this definition:
Events A and B are independent whenever
P(B|A) = P(B).
(Likewise, events A and B are independent whenever
P(A|B) = P(A).)
A: The consumer bought the product
B: The consumer saw the commercial on the TV
A: bought Not_A: did'nt buy Total
B: saw the comm. 5 25 30
Not_B: did'nt see
the commercial 15 55 70
Total 20 80 100
Example 6
Conditional probability and
independence
A: The consumer bought the product
B: The consumer saw the commercial on the TV
A: bought Not_A: did'nt buy Total
B: saw the comm. 5 25 30
Not_B: did'nt see
the commercial 15 55 70
Total 20 80 100
30% =
100
30
= P(B) and 20%
100
20
P(A) = =
Example 6
Conditional probability
What is the probability that the consumer bought
the product (A) and saw the commercial (B)?
% 5
100
5
) and ( = = B A P
Example 6
Conditional probability
Example 6
Conditional probability
What is the probability that the consumer bought
the product (A) and saw the commercial (B)?
% 5
100
5
) and ( = = B A P
What is the probability that the consumer
bought the product (A) given that he saw the
commercial (B)?
% 67 . 16
100 / 30
100 / 5
) (
) and (
) / ( = = =
B P
B A P
B A P
The Rules of Probability
PROBABILITY RULE 4a

Multiplication or AND rule for any two events
E
1
and E
2
:


) | ( ) ( ) (
) | ( ) ( ) (
2 1 2 1 2
1 2 1 2 1
E E P E P E and E P
and
E E P E P E and E P
=
=
Rule 4 revisited
The Rules of Probability
SPECIAL CASE OF THE AND RULE

Multiplication or and rule for two independent events
E
1
, E
2
:


) ( ) ( ) (
2 1 2 1
E P E P E and E P =
Recall: Definition of independence:
Two events are independent if:
P(A|B)=P(A)
Independent IS NOT THE SAME
AS Mutually Exclusive
Independent Check whether P(B|A) = P(B)
Or
Check whether P(A|B) = P(A)
Or
Check whether P(A and B) = P(A) * P(B)
Disjoint (mutually
exclusive)
Check whether P(A and B) = 0
Or
Check whether the events A and B overlap in
the sample space diagram.
Or
Check whether both events can occur together
Are the events A and B independent?
We have to prove that:


Step 1:
Step 2:
Step 3:

Thus, A and B are not independent
Example 6 (cont.)
Independence
P(A B) = P(A) P(B)
P(A B) = 5%
5% 6% =
% 6 6 0 . 0 ) 3 . 0 ( ) 2 . 0 ( ) ( ) ( = = = B P A P
Tree Diagrams
A tree diagram helps us think through conditional
probabilities by showing sequences of events as
paths that look like branches of a tree.
Example 7

A construction Company has a 60% chance of
obtaining an important contract. If this first
contract is obtained, the company has a 2/3 chance
of getting a second contract.
If it doesn't obtain the first contract, its chances of
obtaining the second one fall to 30%.
What is the probability that the company obtains
the second contract?
Let:
A: The company obtains the first contract
B: The company obtains the second contract
We know the following:
P(A) = 0.6 thus P(A
C
) = 0.4
P(B/A) = 2/3 thus P(B
C
/A) = 1/3
P(B/A
C
) = 0.3 thus P(B
C
/A
C
) = 0.7
We have to find P(B) given:
P(A), and
The conditional probabilities
Example 7
Example 7
S
A A
B
P(B) = ?
P B P A B P A B ( ) ( ) ( ' ) = +
P(A)=0.6
P(A')=0.4
Example 7
P(A)=0.6
P(A')=0.4
P(B/A)=2/3
P(B'/A)=1/3
P(B/A')=0.3
P(B'/A')=0.7
Example 7
P(A)=0,6
P(A')=0,4
P(B/A)=2/3
P(B'/A)=1/3
P(B/A')=0.3
P(B'/A')=0.7
P A B P A P B A ( ) ( ) ( / ) =
P A B' P A P B'/A ( ) ( ) ( ) =
P A B P A P B A ( ' ) ( ' ) ( / ' ) =
P A B' P A P B' /A ( ' ) ( ' ) ( ' ) =
Example 7
P(A)=0,6
P(A')=0,4
P(B/A)=2/3
P(B'/A)=1/3
P(B/A')=0.3
P(B'/A')=0.7
P A B P A P B A ( ) ( ) ( / ) =
P A B' P A P B'/A ( ) ( ) ( ) =
P A B P A P B A ( ' ) ( ' ) ( / ' ) =
P A B' P A P B' /A ( ' ) ( ' ) ( ' ) =
=(.6)(2/3)=.4
=(.6)(1/3)=.2
=(.4)(.3)=.12
=(.4)(.7)=.28
Example 7
Example 7
S
A A
B
% 52 52 . 0 12 . 0 40 . 0
) ' / ( ) ' ( ) / ( ) (
) ' ( ) ( ) (
= = + =
+ =
+ =
A B P A P A B P A P
B A P B A P B P
40 . 0 ) ( = B A P
12 . 0 ) ' ( = B A P
Let A be an event of S and E
1
, E
2
, ..., E
n
, a set of mutually
exclusive and exhaustive events of S


Generalization
E
1
A
E
3
E
5
E
4
E
n
E
2
A A E A E
A E A E
n
=

( ) ( )
( ) ( )
1 2
3

P E
i
i
n
( ) =
=
1
1
Let A be an event of S and E
1
, E
2
, ..., E
n
, a set of mutually
exclusive and collectively exhaustive events of S
:
P A P A E P A E P A E
n
( ) ( ) ( ) ( ) = + + +
1 2

A A E A E
A E A E
n
=

( ) ( )
( ) ( )
1 2
3


Generalization
E
1
A
E
3
E
5
E
4
E
n
E
2
P E
i
i
n
( ) =
=
1
1
Let A be an event of S and E
1
, E
2
, ..., E
n
, a set of mutually
exclusive and collectively exhaustive events of S
P A P A E P A E P A E
n
( ) ( ) ( ) ( ) = + + +
1 2

) E / A ( P ) E ( P
) E / A ( P ) E ( P ) E / A ( P ) E ( P ) A ( P
n n
2 2 1 1
+ + + =
) E / A ( P ) E ( P ) A ( P
i
n
1 i
i

=
=

Generalization: Probability rule 10
Example 8
A manufacturing company produces mechanical
components using 3 machines: M
1
, M
2
and M
3
in
the following proportions: 50% from M
1
, 30%
from M
2
and 20% from M
3
.
The probability of having a defective component
given that it was produced by machine M
1
, M
2
and
M
3
are 3%, 4% and 5% respectively.
We pick a component at random from the
components produced during the day.
What is the probability that the component is
defective?
We want to know the probability of the
event:
A: The component is defective
Let:
E
1
: The picked component comes from M
1

E
2
: The picked component comes from M
2

E
3
: The picked component comes from M
3

E
1
, E
2
and E
3
are mutually exclusive
Example 8
We dont have the probability of A (directly), but
we have the following probabilities:
P(E
1
) = 0.5 P(E
2
) = 0.3 P(E
3
) = 0.2
P(A/E
1
) = 0.03 P(A/E
2
) = 0.04 P(A/ E
3
) = 0.05
Example 8
P E
i
i
n
( ) =
=
1
1
S
A
E
1
E
2
E
3
P(A) = ?
We obtain the probability of A using:
Example 8
P A P E P A E
i
i
n
i
( ) ( ) ( / ) =
=

1
% 7 . 3 037 . 0
) 05 . 0 2 . 0 ( ) 04 . 0 3 . 0 ( ) 03 . 0 5 . 0 ( ) (
= =
+ + = A P
Thus, the probability to have a defective
component produced during the day equals 3.7%
Reversing the Conditioning
Reversing the conditioning of two events is rarely intuitive.
Suppose we want to know P(A|B), but we know only P(A),
P(B), and P(B|A).
We also know P(A and B), since
P(A and B) = P(A) x P(B|A)
From this information, we can find P(A|B):

(AandB)
(A|B)
(B)
P
P
P
=
Reversing the Conditioning (cont.)
When we reverse the probability from the
conditional probability that youre originally
given, you are actually using Bayes Rule (or
Theorem) .
Let A be an event of S and E
1
, E
2
, ..., E
n
, a set of mutually
exclusive and exhaustive events of S, To obtain P(E
j
/ A):
P E A
P E A
P A
j
j
( / )
( )
( )
=

Bayes Theorem
Let A be an event of S and E
1
, E
2
, ..., E
n,
, a set of mutually
exclusive and exhaustive events of S, To obtain P(E
j
/ A):
P E A
P E A
P A
j
j
( / )
( )
( )
=

P E A P E P A E
j j j
( ) ( ) ( / ) =
Bayes Theorem
Let A be an event of S and E
1
, E
2
, ..., E
n
, a set of mutually
exclusive and exhaustive events of S, To obtain P(E
j
/ A):
P E A
P E A
P A
j
j
( / )
( )
( )
=

P E A P E P A E
j j j
( ) ( ) ( / ) =
P A P E P A E
i
i
n
i
( ) ( ) ( / ) =
=

1
Bayes Theorem
Let A be an event of S and E
1
, E
2
, ..., E
n
, a set of mutually
exclusive and exhaustive events of S, To obtain P(E
j
/ A):
P E A
P E A
P A
j
j
( / )
( )
( )
=

P E A P E P A E
j j j
( ) ( ) ( / ) =
P A P E P A E
i
i
n
i
( ) ( ) ( / ) =
=

1
P E A
P E P A E
P E P A E
j
j j
i
i
n
i
( / )
( ) ( / )
( ) ( / )
=
=

1
Bayes Theorem
In the previous example (ex. 8), we knew P(A/E
i
),
i.e., the probability that a component is defective
given that it is produced by machine M
i
(event E
i
)
If we now pick a component at random from the
production of the day, and we notice that it is
defective (A), what is the probability that the
component was produced by machine M
3
(event
E
3
)?
Example 9
Bayes Rule
We are interested in determining the probability
that the picked component comes from M
3
(event
E
3
) given that the component is defective, i.e.,
P(E
3
/A).
We know the following:

P(E
1
) = 0.5 P(E
2
) = 0.3 P(E
3
) = 0.2

P(A/E
1
) = 0.03 P(A/E
2
) = 0.04 P(A/ E
3
) = 0.05
Example 9
Bayes Rule
Using Bayes
theorem, we have:
Example 9
Bayes Rule
P E A
P E P A E
P E P A E
j
j j
i
i
n
i
( / )
( ) ( / )
( ) ( / )
=
=

1
P(A)
% 27 27 . 0
037 . 0
01 . 0
) 05 . 0 2 . 0 ( ) 04 . 0 3 . 0 ( ) 03 . 0 5 . 0 (
05 . 0 2 . 0
) / (
3
= = =
+ +

= A E P
What Can Go Wrong?
In most situations where we want to find a
probability, well use the rules in combination.
Beware of probabilities that dont add up to 1.
Dont add probabilities of events if theyre not
disjoint.
Dont multiply probabilities of events if theyre
not independent.
Dont confuse disjoint and independentdisjoint
events cant be independent.
Using Bayes
theorem, we have:
Example 9
Bayes Rule
P E A
P E P A E
P E P A E
j
j j
i
i
n
i
( / )
( ) ( / )
( ) ( / )
=
=

1
P(A)
Thus, once we know that the component is
defective, the probability that it comes from the
machine M
3
is revised upward from 20% to 27%
% 27 27 . 0
037 . 0
01 . 0
) 05 . 0 2 . 0 ( ) 04 . 0 3 . 0 ( ) 03 . 0 5 . 0 (
05 . 0 2 . 0
) / (
3
= = =
+ +

= A E P
Bayes Theorem
When to Use it
I know P(X|Y)


I want to know P(Y|X)
I have an initial estimate
of P(Y) (Prior prob.)


I do a test and find that X
is true

I update my initial
estimate to get P(Y|X)
(Posterior prob.)

Ethics in Action
Hair Salon: Facials or Massages?
Survey of salons offering facials and massages
50% of customers wanting hair styling also want
facials
90% of customers want hair styling or massages
Therefore we should offer massages.
Ethical Issue
Item A: Probabilities not comparable
(conditional/OR)
Ethical Solution
Report all details of survey.