6.

897 Algorithmi Introdu tion to Coding Theory September 12, 2001

Le ture 3
Le turer: Madhu Sudan S ribe: Arian Shahdadi

Today we will talk about Hamming odes and the Hamming bound. In order to talk about Ham-
ming's odes, we will dene linear odes. The le ture also ontained material on some basi and
linear algebra, whi h is deferred to a separate handout.

1 Linear Codes
There are many ways to spe iy a ode given our established formalisms. One onvenient way is to
give an en oding fun tion, thereby spe ifying how to reate arbitrary odewords. This generalized
mapping an be used to de ribe lasses of odes in a su in t manner. For a spe ial lass of odes
alled linear odes, other su in t options are available. We will dene these odes here.
Linear odes are obtained when the alphabet  is asso iated with the eld Fq for some nite q. In
su h ases, one an think of the \ambient spa e" (n), whi h ontains the odewords and re eived
words, as a ve tor spa e.
Re all that L  Fnq is a linear subspa e of the ve tor spa e Fnq if for every pair x; y 2 L and  2 Fq , it
is the ase that x+y 2 L and 
x 2 L. (Here + is ve tor addition and
is s alar-ve tor multipli ation.
Spe i ally, if x = hx ; : : : ; xn i and y = hy ; : : : ; yni, then x + y = hx + y ; : : : ; xn + yni and
1 1 1 1

x = h
x ; : : : ; 
xn i.)
1

When a ode C  Fnq is a linear subspa e of Fnq , then the ode is alled a linear ode. Notationally,
we express the fa t that an (n; k; d)q ode is linear by using square bra kets and denoting it an
[n; k; d℄q ode.

1.1 Generator matrix and Parity he k matrix

Linear algebra gives us several su in t ways of representing odes. Re all that a linear subspa e
C  Fnq of dimension k an be spe ied by giving a basis Pk
for the subspa e, i.e., by giving an
independent set of ve tors x ; : : : ; xk  Fq su h that C = f i i
xi j  ; : : : ; k 2 Fq g. Note in
1
n
=1 1
parti ular that the ode has qk ve tors and so we are not abusing notation here by using k to mean
two dierent things | the dimension does equal the message length.
Using matrix notation, we an thus represent the ode C by a generator matrix G 2 Fqkn , where
the rows of G are the basis ve tors x ; : : : ; xk . (Throughout this ourse we will denote ve tors by
1
row ve tors, unless otherwise spe ied.) In this notation, C = fG j  2 Fkq g.
Another way of des ribing a linear subspa e is by enumerating the onstraints that ve tors in the
subspa e obey, or equivalently, by des ribing its \null spa e". For x; y 2 Fnq , let hx; yi = Pni xi

yi where the multipli ations and summation denote eld operations. Re all that for every linear
=1

3-1
subspa e C of dimension k, one an nd a linear subspa e, denoted
n k  n
C ? , of dimension n k, su h
that for every x 2 C and y 2 C , hx; yi = 0. Let H 2 Fq
? T ( )
be the generator matrix for
C . The matrix H 2 Fq
? n(n k
)
is alled the parity he k matrix of C . Note that one an express
C = fy j yH = 0g.
Note that under both representations, we an show trivially that the 0 word is a odeword for all
linear odes. We will return to this fa t later on.
Both the generator matrix and the parity he k matrix representations give an O(n log q) sized
2

representation of the ode C . These representations take poly(n log q) bits to des ribe and are thus
fairly ompa t.
Furthermore these representations are equivalent omputationally. With some rudimentary linear
algebra, we an show that parity he k matri es and generator matri ies are essentially the same.
That is, given G we an ompute H and given H we an ompute G.
Although we have shown that generator and parity he k matri es are equivalent in both a mathe-
mati al and omputational sense, do they work in the same manner? As we will see, parity he k
matri es have some properties that are very useful.

1.2 Appli ations of the parity he k matrix

Error-dete tion: Suppose we are given a re eived ve tor y obtained from the transmission of
a odeword from a ode of minimum distan e d. Further, suppose that we are guaranteed that
that d 1 or fewer errors have o urred. As we noted earlier, it is possible to tell, given unlimited
omputational resour es, if an error has o ured or not. Can we do this omputation eÆ iently? It
turns out that the answer, for linear odes C given by their parity he k matrix H, is \YES"! We
simply ompute the ve tor yH. If the result is the all zeroes ve tor, then y is a odeword and hen e
no errors o urred during transmission. If yH 6= 0, then y is not a odeword, by the denition of
the parity he k matrix. Thus we on lude that error-dete tion is \easy" (in polynomial time) for
linear odes.
Another ni e property of parity he k matri ies is that they give a sense of the minimum distan e of
a ode. One important question is that of how we an determine the minimum distan e of a linear
ode. Unfortunately, there are no known eÆ ient methods of omputing this | and the problem is
known to be NP hard [8, 2℄. Nevertheless parity he k matri ies oer some aid in designing odes
with reasonable minimum distan e as we will see shortly. First, we digress in order to dene a new
on ept, the Hamming weight of a odeword.
Denition 1 The Hamming weight of a ve tor x 2 Fnq , denoted wt(x), is the number of non-zero
oordinates in x.

Now, in order to nd the minimum distan e of a linear ode, we an nd a non-zero odeword x

of smallest weight su h that x
H = 0. The weight of this odeword, wt(x), equals the minimum
distan e of the ode. We an prove this assertion using elementary linear algebra.
Proposition 2 The minimum distan e of a linear ode C with parity he k H equals the smallest
integer d su h that there exist d rows of H that are linearly dependent.

Proof First, note that the proposition is equivalent to the assertion that the minimum distan e

3-2
of C equals the weight of the non-zero ve tor x of smallest weight that satises xH = 0. To see this,
let hi denotePthe ith row of H. Now suppose xH = 0 and let xi1 ; : : : ; xid be the non-zero elements
of x. Then dj xij hij = 0 and thus the rows hi1 ; : : : ; hid are linearly dependent. Similarly any
olle tion of d linearly dependent rows lead to a ve tor x of weight d su h that xH = 0.
=1

Now we turn to showing that minfx6 0jxH 0gfwt(x)g is the minimum distan e of the ode. First
= =
note that if xH = 0, then its weight is an upper bound on the minimum distan e sin e its distan e
from 0 (whi h is also a odeword) is wt(x). On the other hand this quantity also lower bounds the
minimum distan e: If y and z are the nearest pair of odewords, then the ve tor x = y z is a
odeword whose Hamming weight equals the Hamming distan e between y and z.

1.3 Systemati Codes

Finally we introdu e another side-ee t of linearity, whi h is the notion of a systemati ode.
Denition 3 An (n; k; d)q systemati ode is a ode in whi h the en oding of a message onsists of
k message symbols followed by n k he k symbols.

In other words, in a systemati en oding, the message is re overable without de oding if there are no
errors. Systemati odes are very ommon in pra ti al implementations of en oding and de oding
s hemes.
The term \systemati " is from the paper of Hamming [4℄. Hamming observed that odes obtained
from \linear" operations an be turned into systemati ones without loss of generality. (Hamming
did not expli itly introdu e linear odes, though linearity did play a dominant role in his odes.) To
do so, note that if the generator matrix G is of the form [I j A℄ where I is the k  k identity matrix,
and A is a k  (n k) matrix, then the ode generated by G is systemati . So what an we do if
G is not of this form? For simpli ity, we des ribe the pro edure when q = 2 and the ode is binary.
(The method also generalizes to other elds.) First we permute the olumns and rows of G till the
leftmost entry of the rst row is a 1. (Permuting the rows does not alter the ode, while permuting
the olumns only hanges the naming of the oordinate indi es.) Now we an arrange it so that all
the leftmost entry of all remaining rows is 0. We do so by subtra ting the rst row from any given
row if the row has a 1 in the leftmost entry. Again this pro ess does not hange the ode | it only
hanges the basis we are using for the ode. Repeating this pro ess for the remaining indi es, we
arrive at a generator matrix with an identity matrix in the left part - as desired.
We on lude that every linear ode is systemati . The onverse need not be true.

2 Hamming Codes
The ma hinery of linear odes developed above allows us to arrive at the Hamming odes with
minimum distan e d = 3 naturally. In order to des ribe these odes, we onstru t their parity he k
matrix H.
For simpli ity, we will onstru t the matrix over the alphabet  = f0; 1g. This puts us in the eld
F , where the addition and multipli ation operations are just arithmeti modulo 2.
2

3-3
Noti e our goal is to onstru t a matrix H 2 Fqn n k su h that xH 6= 0 for any ve tor x of weight
( )

1 or 2. (Then the ode will have distan e  3. We will do so for as large an n as possible, given
a xed hoi e of l = n k. Denote by hi the ith row of this matrix. We will rst determine what
onditions hi must satisfy.
First, onsider a ve tor x of weight 1. Say this ve tor has a 1 in the ith position and is zero elsewhere.
Then xH = hi . Sin e we don't want xH to be zero, this imposes the onstraint that hi should not
be zero. If H satises this property, then it gives the parity he k matrix of a ode with distan e
 2.
Now, onsider a ve tor x of weight 2. Say, this ve tor has a 1 in the ith and j th oordinates and is
zero elsewhere. Here, we get, xH = hi + hj . If this result is to be non-zero, we need hi 6= hj , for
every i 6= j . We on lude that H is the parity he k matrix of a ode of minimum distan e  3 over
F if and only if all its rows are distin t and non-zero.
2

Thus if l = n k, then the largest n we an allow is the number of distin t non-zero ve tors hi 2 Fl .
This is a simple enumeration problem | there are exa tly 2l 1 su h ve tors and so we get n = 2l 1.
2

Using our previously dened notation, we an now fully spe ify the parameters of the generalized
Hamming odes:
Proposition 4 For every positive integer l, there exists an [2l 1; 2l l 1; 3℄ 2 ode, alled the
Hamming ode.
Exer ises:

 Des ribe the parameters and parity he k matrix of the q-ary Hamming odes.
 Des ribe the generator matrix of the Hamming ode. Put it in systemati form [I j A℄ and
des ribe the entries of the matrix A.
To make some sense of the growth of parameters, noti e that if k = 2l l 1, then l = (log k).
Thus the Hamming odes are obtained by appending (log k) parity he k bits to the k-bit message.
This gives a ode of minimum distan e 3, or equivalently, a 1-error- orre ting ode. The next se tion
addresses the issue of whether one needs to expend so mu h redundan y to orre t one error. But
rst we des ribe a simple error- orre ting algorithm for the Hamming ode.

2.1 Error- orre tion in the Hamming ode

Given a re eived ve tor y, where y = x + e for some odeword x and a ve tor e of weight one, how
an we ompute the lo ation i su h that ei = 1?
The naive method for omputing this would be by brute for e. For j = 1 to n, we try ipping the
j th bit of y and see if this gives us a odeword. We an he k if su h a ve tor y0 is a odeword by
omputing y0 H and he king to see if it is zero. This gives an O(n ) algorithm.
3

Hamming wasn't satised with this approa h and instead relied on the parity he k matrix to simplify
this omputation. Sin e we know that xH = 0, we have that yH = eH = hi , where i is the index of
the error lo ation. Furthermore, if we arrange H so that the ith row is simply the number i written
in binary, then the ve tor yH simply gives the lo ation of the error in binary. This speeds up our
omputation by a fa tor of n, giving an O(n ) algorithm to orre t errors.
2

3-4
3 The Hamming bound and perfe t odes
Is it ne essary to add logarithmi ally many bits to orre t 1 error? Hamming proved that this was
the optimal result by introdu ing what is alled the Hamming bound. We now derive this bound.
Let us re all some notation we have used earlier. Let B(y; r) denote the P ball of radius r around
ve tor y (where for this se tion the ve tors will be from Fn ). Let Vol(r; n) = ri ni
be the volume
of su h a ball. The Hamming bound gives the following:
2 =0

Theorem 5 If an (n; k; d)2 ode exists, then


2k Vol d 1  ; n  2n:
2
Proof Let C = (n; k; d) ode and let r =  d . The Hamming bound is based on the simple
2
1

observation that if x and y are odewords of C , then B(x; r) and BP (y; r) are disjoint. Sin e
2

S
x2C B (x; r )  f0; 1g n and the sets on the LHS are all disjoint, we get:
x2C jB (x; r)j  2 . The
n
bound follows immediately.
The Hamming bound generalizes naturally to q-ary alphabets. Let Volq (r; n) = Pri n
(q 1)i.
i
Then we have:
=0

Theorem 6 For every n; k; d and q, an (n; k; d)q ode exists only if


qk Volq
d 1  ; n  q n :
2
For the ase d = 3 and q = 2, the above bound says that 2k (n + 1)  2n, and this bound is met
exa tly by the Hamming odes (and this is true also for q-ary Hamming odes). The Hamming
bound denes a lass of odes known as perfe t odes.
Denition 7 An (n; k; d)q ode is perfe t if it meets the Hamming bound exa tly: i.e.,
  
d 1
q Volq
2 ;n = q :
k n

An interesting fa t here is that other than the Hamming odes and two spe i odes proposed by
Golay [3℄ no other perfe t odes exist. This was supse ted for long and nally proved by van Lint [7℄
and Tietavainen [6℄. This proof is way beyond the s ope of this lass.

3.0.1 Converting odd minimum distan e to even minimum distan e

The Hamming bound is learly \jerky" - it proves nothing interesting for d = 2, but jumps to proving
that the Hamming odes for d = 3 are best possible. It does not improve for d = 4 but then jumps
again when d = 5. Why does this dis ontinuity happen? Is it that the Hamming bound is too weak,
or is it truly easier to onstru t odes of even minimum distan e? Hamming showed, surprisingly
enough, that the latter was the ase. The argument is quite simple - the same that we used to
onstru t distan e 2 odes from distan e 1 odes and we des ribe it next.

3-5
Proposition 8 If an (n; k; 2t + 1)2 ode exists, then so does an (n + 1; k; 2t + 2)2 ode.

Proof Let C be an (n; k; 2t+1)2 ode. We onstru t C 0 , an (n+1; k; 2t+2)2 ode from C as follows:
Let x = hx1 ; : : : ; xn i 2 C . Corresponding to x, C 0 ontains the odeword x0 = hx1 ; : : : ; xn; Pni=1 xi i.
(All summations in this proof are sums modulo 2.) It is lear that C 0 is an (n + 1; k; d)2 ode for
some d, and what we wish to show is that d  2t + 2.
It suÆ es to show that if x; y 2 C , then
(x0 ; y0 )  2t + 2. Noti e rst that
(x0 ; y0 ) 
(x; y).
So if the latter is at least 2t + 2 then we are done. Thus it suÆ es to onsider the ase where

(x; y) = 2t + 1. In this ase, let us permute the oordinates of C so that x and y agree in the
rst m = n (2t + 1) lo ations and disagree in the last 2t + 1 lo ations. Now onsider the quantity
Pn
i=1 (xi + yi ). This quantity equals the parity of \(2 # 1's in x in the rst m oordinates)
+(2t + 1)". Clearly, this parity is odd (sin e the rst expression as well as 2t are even and the last
term, 1 is odd). We thus get that exa tly one of Pni=1 xi and Pni=1 yi is one (modulo 2) and the
other is zero. Thus x0 and y0 disagree in the last oordinate and so the distan e between them is
2t + 2 in this ase also.
Exer ise: Does the proposition above extend for other values of q?
Applying the proposition above to the distan e 3 odes, Hamming got a family of distan e 4 odes.
Going from d = 4 to d = 5 is non-trivial. In parti ular the Hamming bound implies that one needs
approximately 2 log k he k bits to get su h a ode. Getting su h a ode proved to be non-trivial
and was nally dis overed, independently by Bose and Ray-Chaudhuri [1℄, and Ho quenghem [5℄, in
1959. These odes, whi h are quite famous under the label \BCH odes", give binary odes of blo k
length n and odd minimum distan e d using d log n he k bits. The Hamming bound shows this
1

is tight to within lesser order terms.

2

Referen es
[1℄ R. C. Bose and D. K. Ray-Chaudhuri. On a lass of error orre ting binary group odes. Infor-
mation and Control, 3:68{79, 1960.
[2℄ Ilya Dumer, Daniele Mi ian io, and Madhu Sudan. Hardness of approximating the minimum
distan e of a linear ode. Pro eedings of the 40th IEEE Symposium on Foundations of Computer
S ien e, pages 475{484, 1999.
[3℄ M. J. E. Golay. Notes on digital oding. Pro eedings of the IRE, 37:657, June 1949.
[4℄ Ri hard W. Hamming. Error Dete ting and Error Corre ting Codes. Bell System Te hni al
Journal, 29:147{160, April 1950.
[5℄ A. Ho quenghem. Codes orre teurs d'erreurs. Chires (Paris), 2:147{156, 1959.
[6℄ Aimo Tietavainen. On the nonexisten e of perfe t odes over nite elds. SIAM Journal of
Applied Mathemati s, 24(1):88{96, January 1973.
[7℄ Ja obus H. van Lint. Nonexisten e theorems for perfe t error- orre ting odes. In G. Birkho
and M. Hall Jr., editors, Pro eedings of the Symposium on Computers in Algebra and Number
Theory, New York, 1970, pages 89{95. Ameri an Mathemati al So iety, Providen e, RI, 1971.
[8℄ Alexander Vardy. The intra tability of omputing the minimum distan e of a ode. IEEE
Transa tions on Information Theory, 43:1757{1766, November 1997.

3-6