INFORMATION THEORY AND

CODING
5th SEM E&C

JAYANTHDWIJESH H P M.tech (DECS)

Assistant Professor Dept of E&C

B.G.S INSTITUTE OF TECHNOLOGY (B.G.S.I.T)

B.G Nagara, Nagamangala Tq, Mandya District- 571448
 FORMULAS FOR REFERENCE

MODULE 2 (source coding)

Entropy of source or Average information content of the source.

H(S) = = ( ) bits/symbol or H(S) = = ( ) bits/symbol

Average length

L= = bits/symbol or L = = bits/symbol
Source or code efficiency
() ()
= X % or = X %

Source or code redundancy
() ()
= 1- = (1 -
) X % or = 1- = (1 -
) X %

Compute the number of stages required for the encoding operation, which is
given by

=

or =
The probability of 0s and 1s and 2 s in the code are found using the
formulas

P (0) = =[ "0" s in the code for ] [ ] or

P (0) = =[ "0" s in the code for ] [ ] .

P (1) = =[ "1" s in the code for ] [ ] or

P (1) =
=[ "1" s in the code for ] [ ] .

P (2) =
=[ "2" s in the code for ] [ ] or

P (2) = =[ "2" s in the code for ] [ ] .

The variance of the word length is calculated from

Var ( ) = E [( ) =
= ( )
The Smallest integer value of if found using

or

The average length of the 2nd extension is given by

= = bits/symbol or = = bits/symbol
 The average length of the 3rd extension is given by

= = bits/symbol or = = bits/symbol
The entropy of the 2nd extended source is calculated as
H ( ) = 2 H(S)
The entropy of the 3rd extended source is calculated as
H ( ) = 3H(S)
Source or code efficiency of the 2nd extended source is
( ) ( )
() = X % or () = X %

Source or code redundancy of the 2nd extended source is

( ) ( )
() = 1- () = (1 -
) X % or () = () = (1 -

)
%
Source or code efficiency of the 3rd extended source is
( ) ( )
() = X % or () = X %

Source or code redundancy of the 3rd extended source is

( ) ( )
() = 1- () = (1 -
) X % or () = () = (1 -

)
%

The average length of the Huffman ternary code is given by
() =
= trinits /Msg- symbol or
()
=
= trinits / Msg- symbol

The average length of the Huffman quaternary code is given by
() =
= quaternary digits /Msg- symbol or

() =
= quaternary digits / Msg- symbol
The entropy in ternary units/ message symbol is found by using equation
()
(S) = ternary units/ message symbol or

(S) = = ( ) ternary units/ message symbol or

(S) = = ( ) ternary units/ message symbol

The entropy in quaternary units/ message symbol is found by using equation

()
(S) = quaternary units/ message symbol or


(S) = = ( ) quaternary units/ message symbol or


(S) = = ( ) quaternary units/ message symbol

Source or code efficiency of the ternary is given by

() ()
() = X % or () = X % or
() ()
() ()
= X % or = X %

Source or code efficiency of the quaternary is given by
() ()
() = X % or () = X % or
() ()
() ()
= X % or = X %

Source or code redundancy of the ternary is given by
()
= 1- () = (1 - ) X % or
() ()
()
() = () = (1 - ) % or
()
() ()
= 1- = (1 -
) X % or = 1- = (1 -
) X %

Source or code redundancy of the quaternary is given by

()
= 1- () = (1 - ) X % or
() ()
()
() = () = (1 - ) % or
()
() ()
= 1- = (1 -
) X % or = 1- = (1 -
) X %