A dissertation submitted in partial ful llment of the requirements for the degree of Doctor of Philosophy (Electrical Engineering: Systems) in The University of Michigan 1998
Doctoral Committee: Professor David L. Neuho , Chair Associate Professor Je rey A. Fessler Professor Edward D. Rothman Professor Demosthenis Teneketzis Assistant Professor Kimberly M. Wasserman
To Michelle
ii
ACKNOWLEDGEMENTS
I am deeply indebted to my advisor, Dave Neuho , for his guidance and encouragement throughout the course of this research. I have bene tted tremendously from his instruction and expertise over the past ve years. I would like to thank Professors Je Fessler, Ed Rothman, Demosthenis Teneketzis and Kim Wasserman for serving on my doctoral committee. My gratitude also goes to Professors Lorne Campbell and Paul Wittke of Queen's University for encouraging my interest in communications as an undergraduate. I would like to thank my ancee Michelle for her constant love and patience. She has been an boundless source of support. I also thank my parents, Warren and Josephine Moo, and my sister Rachel, for providing a loving family atmosphere to grow up in and for their continued love and support. My grandparents, Charles and Berthebell Moo, and Alex and Hilda Watts, have been a true source of inspiration I thank them for their love and encouragement. I would also like to thank my friends and colleagues here at Michigan, who have kept me sane and taught me a great deal. Finally, I thank the National Science Foundation for its generous nancial support.
iii
TABLE OF CONTENTS
DEDICATION : : : : : : : : ACKNOWLEDGEMENTS LIST OF FIGURES : : : : : LIST OF TABLES : : : : : : CHAPTER ii : : : : : : : : : : : : : : : : : : : : : : : : : : iii : : : : : : : : : : : : : : : : : : : : : : : : : : vi : : : : : : : : : : : : : : : : : : : : : : : : : : vii
::::::::::::::::::::::::::
1
I. Introduction : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
II. Asymptotically Optimal FixedRate Lattice Quantization for a Class of Generalized Gaussian Sources : : : : : : : : : : : : : 16
Introduction : : : : : : : : : : : : : : : : : : : : Distortion Bounds : : : : : : : : : : : : : : : : : Bounds to Overload Distortion : : : : : : Bounds to Granular Distortion : : : : : : Main Results : : : : : : : : : : : : : : : : : : : : Minimizing the Upper and Lower Bounds Minimizing Lattice Quantizer Distortion Summary : : : : : : : : : : : : : : : : : : Proofs and Derivations : : : : : : : : : : : : : : Proof of Lemma 1 : : : : : : : : : : : : : Proof of Lemma 2 : : : : : : : : : : : : : Proof of Lemma 3 : : : : : : : : : : : : : Proof of Proposition 5 : : : : : : : : : : : Proof of Proposition 6 : : : : : : : : : : : Proof of Proposition 10 : : : : : : : : : : Appendix : : : : : : : : : : : : : : : : : : : : : : References : : : : : : : : : : : : : : : : : : : : : iv
: : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : :
16 22 22 23 24 24 26 28 28 28 29 30 31 32 35 38 42
: : : : : : : : :
: : : : : : : : :
: : : : : : : : :
: : : : : : : : :
: : : : : : : : :
: : : : : : : : :
: : : : : : : : : : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : : : : : : : : : :
45 48 51 54 57 57 58 59 63 65 69 70 71 73 76 78 80 83 83 84 84 85 85 90 90
V. Conclusions : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 95
Summary of Contributions : : : : : : : : : : : : : : : : : : : : : : 95 Future Research Issues : : : : : : : : : : : : : : : : : : : : : : : : 96
LIST OF FIGURES
Figure
1.1 1.2 2.1 2.2 3.1 4.1 4.2 4.3 4.4 A lattice quantizer based on the twodimensional square lattice with hexagonal support. : : : : : : : : : : : : : : : : : : : : : : : : : : : 15 A lattice quantizer based on the twodimensional hexagonal lattice with hexagonal support. : : : : : : : : : : : : : : : : : : : : : : : : 15 The optimal scaling factor aN for N xed. : : : : : : : : : : : : : : 44 Illustration of inner and outer supports. : : : : : : : : : : : : : : : : 44 Block diagram of multidimensional companding. : : : : : : : : : : : 64 Examples of polar quantizers: (a) restricted nonuniform, (b) restricted uniform, (c) unrestricted nonuniform. : : : : : : : : : : : : 92 The height H and width W of a polar quantization cell. : : : : : : : 92 Optimal phase rates for restricted uniform polar quantization. The optimal rate from Proposition 6 is R . The twoterm and threeterm approximations are R 2 and R 3, respectively. : : : : : : : : : : : 93 Optimal magnitude quantizer support for restricted uniform polar quantization. The optimal support from Proposition 6 is LM . The twoterm and threeterm approximations are LM 2 and LM 3, respectively. : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 93 Maximum signaltonoise ratio (SNR) for restricted uniform polar quantization. The optimal rate from Proposition 6 is S . The twoterm and threeterm approximations are S2 and S3 , respectively. : : 94 Signaltonoise ratio (SNR) of restricted nonuniform polar quantization, restricted uniform polar quantization and uniform scalar quantization. : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 94 vi
4.5 4.6
LIST OF TABLES
Table
3.1 4.1 Losses of optimal companding, Lcomp , and optimal product VQ, Lprod , for an IID Gaussian source. : : : : : : : : : : : : : : : : : : : 54 Comparison of magnitude allocation M and MSE D = ;=N for a pair of IID Gaussian random variables. : : : : : : : : : : : : : : : : 74
vii
CHAPTER I
Introduction
The core of this dissertation is composed of three selfcontained manuscripts, each serving as a chapter. This chapter presents an introduction to the research area considered in the manuscripts, a summary of some open problems in the area, and an overview of this dissertation and its results. The work presented in this dissertation analyzes three quantization methods, which have at their core a lattice structured codebook. As we will discuss, these methods are of interest because some lattice encoding techniques have low complexity and because lattices have a strong connection to optimal quantizers. We begin our discussion with a brief introduction to vector quantization.
Vector quantization (VQ) is the process of mapping a kdimensional realvalued
encoder maps the random vector to a reproduction vector and transmits a nitelength binary codeword, representing the index of the reproduction vector, across a channel to the decoder. The decoder maps the received binary sequence to a reproduction vector. In general, the channel is noisy and introduces errors to the binary codeword, but for this discussion, we will assume that the channel is noiseless. 1
the index set of the codevectors. The mapping performed by a VQ is given by the quantization rule Q(x) = yi if x 2 Si. There are three key measures of vector quantizer performance: distortion, rate, and complexity. Distortion is a measure of the accuracy of the reproduction vector in comparison to X . Throughout this dissertation, we will only consider a widely used distortion measure, average meansquared error (MSE), which is given by, Z D(Q) = k kx ; Q(x)k2 p(x) dx
<
where p(x) is the probability density of X . The rate of a VQ measures the average length of the binary codewords produced by the VQ encoder. If the codebook is of size N and all codevectors have binary codewords of the same length (modulo one bit), then the quantizer is said to be xedrate, and the rate is given by, 1 RFR(Q) = k log2 N: On the other hand, if the binary codewords are of varying lengths, then the quantizer is variable rate. One can think of a variablerate VQ as consisting of a partition, a codebook, and a lossless binary code. It is known that Hu man's algorithm designs binary codes with the smallest possible average codeword lengths for a given set of probabilities 11]. We will refer to a variable rate VQ that employs such a binary code as an entropycoded vector quantizer, because the rate of a quantizer employing a Hu man code is approximately the entropy of the quantizer output, that is, X REC (Q) = ; Pi log2 Pi where Pi = P (X 2 Si). The complexity of a VQ is a measure of the computational and storage requirements of the encoder and decoder. For example, in fullsearch
i2I
3 encoding, the encoder must compare the source vector to every codevector to nd the closest codevector and must therefore also store the codebook. In this case, the decoder need only store the codebook. Ideally, we would like to design VQs with simultaneously low distortion, rate and complexity, but there are in fact tradeo s that must be made among these three performance measures. Research in vector quantization looks to characterize these tradeo s, both for arbitrary VQs and for various classes of VQs. This dissertation focuses on the distortionrate performance of a particular class of vector quantizers, namely latticebased quantization. Lattice quantization has at its core an in nite lattice , which is a countably in nite set of points in <k that is closed under addition 6, 10]. A lattice quantizer codebook is given by R \ , where R <k , is the quantizer support. Figures 1.1 and 1.2 show examples of lattice quantizers with hexagonal support. Note that R is bounded for xedrate quantization but may be unbounded for variablerate quantization. The complement RC is the overload region. Lattice quantization in one dimension is called uniform scalar quantization. We will scale the lattice quantizer by a scaling factor a 2 <+ . For uniform scalar quantization, the scaling factor is often called the step size. We loosely de ne a latticebased quantization method as a VQ that utilizes as its codebook a subset of either or a transformation of . Before continuing our discussion of latticebased quantization, we will consider optimal vector quantization. It will be seen that optimal VQ provides a distortionrate performance benchmark and motivates our consideration of latticebased VQ. An optimal vector quantizer minimizes distortion for a given rate and dimension. In general, we are interested in designing optimal VQs and in characterizing their performance. The design of optimal VQ is well understood. Necessary and su cient
4 conditions for an optimal xedrate scalar (k = 1) quantizer were developed by Lloyd 13], who developed an iterative descent algorithm for scalar quantizer design. Lloyd's optimality conditions can be extended to multiple dimensions, and his design algorithm was later generalized to xedrate vector quantizers by Linde et al. 12]. In multiple dimensions, an optimal vector quantizer satis es two necessary conditions. First, each codevector is the centroid of its respective cell, that is, yi = E X jX 2
Si]. Second, each quantization cell is the Voronoi region of its codevector, that is, Si = fx : kx ; yik kx ; yj k j 6= ig. The descent algorithms in 13, 12] iteratively
design a codebook using these necessary conditions. In general, they converge to local minima, but various techniques can be used to increase the chances of nding global minima. These algorithms yield optimal codebooks, but require fullsearch encoding that is, the encoder must compare the source vector to every codevector to nd the closest codevector. As we will see, this causes the search complexity to increase exponentially as rate and dimension increase. Another important issue is the problem of indexing, which is the process of assigning binary codewords to each codevector. Again, as rate and dimension increase, e cient indexing is a signi cant problem. To characterize the performance of optimal VQ, we are interested in determining the least possible distortion of vector quantizers with rate R or less, called the operational distortionrate function (ODRF). The ODRF among VQs of any dimension is denoted (R), while the ODRF among VQs with dimension k is denoted k (R). We would like to nd analytical expressions for (R) and k (R). In his celebrated 1948 paper, Shannon 18] showed that (R) equals D(R), which is the distortionrate function and is de ned using informationtheoretic quantities that depend on the source statistics. It was shown in 18] that D(R) can be achieved arbitrarily
5 closely by using VQs of increasing vector dimension k and that when k is large, a good VQ has a uniform distribution of codevectors in the region of typical sequences, where the source distribution is approximately uniform. When rate R is large, k (R) for xedrate VQ can be characterized using high resolution quantization theory, which is also called asymptotic quantization theory or high rate theory. Bennett's integral for vector quantizers 1, 8, 14] gives an expression for the distortion of an arbitrary VQ, when the number of codevectors N is large, 1 Z m(x) p(x) dx D(N ) = N 2=k (x)2=k (1.1) where the point density (x) is the normalized density of codevectors near x, the inertial pro le m(x) is approximately the normalized moment of inertia of the quantization cells near x, and the normalized moment of inertia (NMI) of a cell S is given R by M (S ) = k;1 v(S );(1+2=k) S kxk2 dx. The ODRFs for kdimensional xedrate VQ can then be derived by choosing m(x) and (x) to minimize Bennett's integral. It is not known how to optimize m(x), but according to Gersho's widely accepted conjecture 8], the best inertial pro le m (x) is a constant, that is, m (x) = mk , where mk is the least NMI of any tesselating polytope in k dimensions. Using variational calculus or Holder's inequality then shows that the best inertial pro le is (x) = c p(x)k=(k+2). This gives the ODRF for kdimensional VQ, which expressed as a function of rate, is lim R!1 22R
k (R)
= mk
p(x)k=(k+2) dx
(k+2)=k
(1.2)
This was rst derived by Zador 19], who determined the righthand side of (1.2) to within a constant Gersho then conjectured that the constant is mk . Equivalently, the least distortion of a xedrate VQ with N codevectors is approximately 1 m Z p(x)k=(k+2) dx (k+2)=k : D (N ) = N 2=k k (1.3)
6 Since Bennett's integral gives the distortion of an arbitrary VQ, we can use (1.1) and (1.3) to quantify the loss L of suboptimal VQ. Optimal VQs have optimal point density and inertial pro le, which implies that the total loss L can be factored as R ) R ) R ) 2 2 D(N ) = m()x=k p(x) dx = m()x=k p(x) dx mx)(2x=k p(x) dx (x (x ( L = D (N ) R m (x) R m (x) R m (x) (x)2=k p(x) dx (x)2=k p(x) dx  (x)2=k p(x) dx  {z } {z } Lce Lpt where cell shape loss Lce is the loss due to suboptimal inertial pro le and point density
loss Lpt is the loss due to suboptimal point density. For product quantization, which
is scalar quantization used k times, cell shape loss can be further decomposed as the product of cubic loss Lcu , which is due to product quantization's inability to form cells with lower NMI than cubes, and oblongitis loss, Lob , which is due to the fact that the quantization cells are rectangles instead of cubes. High resolution theory also gives an analytical expression for k (R) for entropycoded VQ. In this case, a high resolution formula for distortion is given by, Z m(x) D = (x)2=k p(x) dx where (x) is the unnormalized point density. A high resolution formula for rate is, 1 Z p(x) log (x) dx R = hk (X ) + k 2 1 R where hk (X ) = ; k p(x) log2 p(x) dx is the kthorder di erential entropy of the source. To optimize entropycoded VQ, we wish to choose m(x) and (x) to minimize D when R is xed. The best m(x) is not known, so we again apply Gersho's conjecture which yields m (x) = mk . It can then be shown using variational calculus that (x) = c, that is, the optimal point density is uniform. It follows that the lim 22R k (R) = mk 22hkn (X ) ODRF of VQ with nth order entropy coding is
R!1
7 which was rst derived by Zador 19]. We have characterized the ODRFs for kdimensional VQ using high resolution quantization theory. In addition, as we have seen, there are algorithms for designing optimal VQs using a training sequence from the source. However, the complexity of optimal VQ is potentially prohibitive, especially as rate and dimension increase. To see this, consider an optimal unstructured codebook and full search encoding. The encoder needs to store and search over N = 2kR codevectors to implement full search encoding, and the decoder needs to store N = 2kR codevectors. Therefore, the complexity of such a VQ is O(2kR ), which increases exponentially in dimension and rate. In order to achieve performance close to that of optimal VQs under reasonable complexity constraints, one must consider structured vector quantizers. There are many types of structured VQs 9], such as product, multistage, treestructured, trellis, lattice, and hierarchical, many of which have been shown to achieve good performance with moderate complexity.
8 use an e cient algorithm for indexing. E cient encoding and indexing algorithms have been developed for certain quantizer support regions. Given an arbitrary point x 2 <k , Conway and Sloane 4] have shown that the closest point to x in can be found with search complexity O(k), which is signi cantly less that the search complexity of fullsearch VQ, O(2kR ). This result shows that xedrate lattice quantization of bounded sources can be implemented with low complexity. However, this algorithm applies only to the in nite lattice and does not directly extend to lattice quantizers that use only a nite number of points from . For xedrate lattice quantization of unbounded sources, some suboptimal lowcomplexity encoding schemes for source vectors in the overload region have been proposed. However, it has not yet been shown that source vectors in the overload region can be optimally encoded with complexity less than O(2kR ), the complexity of full search encoding. Thus a signi cant open problem involves developing a lowcomplexity algorithm that nds the closest lattice point in the support to an arbitrary point in the overload region. Conway and Sloane have also developed indexing algorithms of order O(k) for lattice quantizers, whose support region is the Voronoi region of a sublattice of 5]. However, these algorithms do not extend to other support regions, such as spherical. E cient indexing for nonVoronoi support regions is an important open problem. The optimal scaling of a lattice quantizer has, for the most part, been determined experimentally. If the source has bounded support, it can be seen that, at least for high rate, the lattice should be scaled to cover the support of the source density. For unbounded sources, however, it is not clear how to scale the lattice or how the scaling should vary as rate increases. Optimal scaling may vary with the shape of the support region. For example, the scaling for a spherical support might di er
9 from that for a cubic support, especially in higher dimensions. In Chapter II of this dissertation, we derive analytical expressions for the optimal scaling factor for spherical support regions. There are still more reasons for considering lattice quantization. In our discussion of optimal VQ, we saw a number of intriguing connections between optimal VQ and lattices. Shannon's asymptotic equipartition property shows that for large dimension, the source is approximately uniformly distributed this in turn suggests that lattice quantization may be optimal. It is clear that for xed dimension, lattice quantizers are not optimal however, they may achieve performance close to optimal. From high resolution theory, we know that an optimal xedrate VQ has a point density that matches the source density, which implies that lattice quantizers are optimal only for uniformly distributed sources. However, Gersho's conjecture suggests that optimal VQs are locally a lattice quantizer. This suggests that an optimal VQ might be obtained from a transformation of a lattice quantizer. In fact, this is known to be the case in one dimension. Bennett showed that any scalar quantizer, e.g. optimal, can be implemented by a companding structure, which consists of a nonlinear mapping, called a compressor function, followed by a uniform scalar quantizer and the inverse of the compressor function. A multidimensional compander utilizes a vectorvalued compressor function and a lattice quantizer. In multiple dimensions, it has been shown that optimal VQs cannot be implemented using a companding structure 8, 2, 3], except for a very restricted class of source densities. It may be that the best companders are close to optimal however, the best multidimensional companders are not currently known. In Chapter III, we determine the optimal high resolution compressor function for memoryless sources, under certain technical conditions on the compressor function.
10 Following from the companding idea, it is also interesting to consider lattice quantization with other transformations. For example, the source can be quantized in terms of its polar coordinates 17], 7]. Lattice quantization can also be used in twostage quantization to quantize the rststage errors 16]. In Chapter IV, we optimize uniform polar quantization, a simple latticebased method that uses a polar transformation and a twodimensional integer lattice. Recall that for asymptotically optimal variablerate VQ, the best point density is uniform. This suggests that lattice quantization may be optimal. The encoding technique of Conway and Sloane can be implemented because the lattice VQ codebook consists of the entire lattice therefore, entropycoded VQ has low encoding complexity. However, in order to achieve good performance, variablerate VQs must use higher order entropy coders, which have high complexity. Thus, optimal VQs have high complexity for xedrate VQ, the complexity lies in the quantizer, while for variablerate VQ, the complexity lies in the entropy coder 15]. As we have seen, optimal variablerate VQ is well understood. The best asymptotic quantizer has a uniform point density and may use an in nite number of codevectors. There are few interesting open questions in variablerate lattice VQ, and as a result, this dissertation considers xedrate lattice quantization only.
Main Contributions
In this thesis, three latticebased quantization methods are analyzed and their optimal high resolution distortionrate performance is determined. In Chapter II, we study simple lattice quantization for generalized Gaussian sources. Despite the simplicity of lattice quantization, it is still not known how the quantizer support should increase with rate. A related open question asks how
11 the distortion of optimized lattice quantization decreases with N . We saw that for asymptotically optimal VQ, distortion decreases as N ;2=k , and it has long been suspected that optimized lattice quantization decreases less rapidly with N . In this chapter, we derive asymptotic formulas for the optimal scaling factor, which uniquely determines quantizer support, and the resulting minimum MSE. These expressions are derived by minimizing upper and lower bounds to distortion. It is shown that the optimal scaling factor aN decreases as (ln N )1= N ;1=k and that for scaleoptimized lattice quantization, granular distortion asymptotically dominates overload distortion. Consequently, the minimum distortion is DN =
c (ln N )2= N ;2=k . This result indicates that the performance of optimal lattice quantizers diverges from that of asymptotically optimal vector quantization, as N increases. Because simple lattice quantization has poor distortionrate performance compared to optimal VQ, we are led to consider multidimensional companding in Chapter III. In particular, we would like to determine the compressor function that minimizes compander distortion. For an arbitrary source, this is a di cult open problem. In this chapter, we nd the asymptotically optimal compressor function for a memoryless source, under certain technical conditions on the compressor function. In order to do so, we consider Bucklew's asymptotic expression for compander distortion 2]. We then show, using variational calculus arguments, that the compressor function consisting of the optimal scalar compressor functions for each source component is the minimizing function. As a result, one concludes that optimized companding has distortion that decreases as N ;2=k but nevertheless su ers the same point density and oblongitis losses as optimal scalar quantization. Companding is able to recover the cubic loss by appropriate choice of .
12 Although it is not able to achieve performance near that of optimal VQ, companding o ers a signi cant improvement over simple lattice quantization. As a result, we are interested in knowing whether there are simple latticebased methods that can o er similar improvement. One such method is polar quantization, which we consider in Chapter IV. In particular, we consider uniform polar quantization, where the source vector is quantized in terms of its magnitude and phase, using a uniform scalar quantizer for each. This is also a latticebased method, involving a polar transformation and an integer lattice quantizer in two dimensions. For a Gaussian source, we are able to use a recent result on the support and distortion of an optimal uniform scalar quantizer for a Rayleigh random variable, similar to that in Chapter II, to nd the optimal step size for the magnitude quantizer. The asymptotically optimal rate allocation between magnitude and phase is then derived. The results show that the optimal rate allocation gives increasingly more rate to the magnitude as total rate increases, compared to nonuniform polar, where the di erence between optimal magnitude and phase rates is asymptotically a constant. In Chapter IV, we also present a uni ed analysis of several nonuniform polar quantization methods by focusing on their point densities and inertial pro les and using Bennett's integral to express the meansquared error. The subsequent analysis is straightforward and leads to new insights into the relationship between polar quantization and Cartesian quantization. With this approach, unrestricted polar quantization, which is arguably the best method, may be optimized essentially by inspection. As another example, a new polar quantization method, called power law polar, is analyzed and optimized.
13
References
1] W.R. Bennett, \Spectra of quantized signals," Bell Syst. Tech. J., vol. 27, pp. 446472, July 1948. 2] J.A. Bucklew, \Companding and random quantization in several dimensions," IEEE Trans. Inform. Theory, vol. IT27, pp.207211, Mar. 1981. 3] J.A. Bucklew, \A note on optimal multidimensional companders," IEEE Trans. Inform. Theory, vol. IT29, p. 279, Mar. 1983. 4] J.H. Conway and N.J.A. Sloane, \Fast quantizing and decoding algorithms for lattice quantizers and codes," IEEE Trans. Inform. Thy., vol. IT28, pp. 227232, Mar. 1982. 5] J.H. Conway and N.J.A. Sloane, \A fast encoding method for lattice codes and quantizers," IEEE Trans. Inform. Thy., vol. IT29, pp. 820824, Nov. 1982 6] J.H. Conway and N.J.A. Sloane, Sphere Packings, Lattices, and Groups, 2nd ed. New York: SpringerVerlag, 1993. 7] T.R. Fischer, \A pyramid vector quantizer," IEEE Trans. Inform. Theory, vol. IT32, pp. 568583, July 1986. 8] A. Gersho, \Asymptotically optimal block quantization," IEEE Trans. Inform. Theory, vol. IT25, pp. 373380, July 1979. 9] A. Gersho and R. Gray, Vector Quantization and Signal Compression. New York: Springer, 1991. 10] J.D. Gibson and K. Sayood, \Lattice quantization," Advances in Electronics and Electron Physics, vol. 72, pp. 259330, 1988. 11] D.A. Hu man, \A method for the construction of minimum redundancy codes," Proc. IRE, vol. 40, pp. 10981101, Sept. 1952. 12] Y. Linde, A. Buzo, and R.M. Gray, \An algorithm for vector quantizer design," IEEE Trans. Commun., vol. 28, pp. 8495, Jan. 1980. 13] S.P. Lloyd, \Least squares quantization in PCM," Bell Laboratories Technical Note, 1957. 14] S. Na and D.L. Neuho , \Bennett's integral for vector quantizers," IEEE Trans. Inform. Theory, vol. IT41, pp. 886900, July 1995. 15] D.L. Neuho , \Source coding strategies: simple quantizers vs. simple noiseless coders," Proc. 1986 CISS, pp. 267271, 1986. 16] J. Pan and T.R. Fischer, \Two stage vector quantizationlattice vector quantization," IEEE Trans. Inform. Theory, vol. 41, pp. 155163, Jan. 1995. 17] M.J. Sabin and R.M Gray, \Product code vector quantizers for waveform and voice coding," IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP32, no. 3, June 1984.
14 18] C.E. Shannon, \A mathematical theory of communication," Bell Syst. Tech. J., vol. 27, pp. 379423, 623656, 1948. 19] P.L. Zador, \Topics in the asymptotic quantization of continuous random variables," Bell Laboratories Technical Memorandum, 1966.
15
Figure 1.1: A lattice quantizer based on the twodimensional square lattice with hexagonal support.
Figure 1.2: A lattice quantizer based on the twodimensional hexagonal lattice with hexagonal support.
CHAPTER II
Asymptotically Optimal FixedRate Lattice Quantization for a Class of Generalized Gaussian Sources1
I. Introduction
Lattice quantization is one of the simplest methods of vector quantization (VQ) 1, 2], but for sources with unbounded support, it is still not known how the quantizer support should vary with the number of codevectors. To be speci c, consider a lattice codebook of size N and let the quantizer codevectors be scaled by a real number a.2 We would like to determine the best scaling factor aN and the resulting minimum distortion DN . In this paper, we nd asymptotic expressions for aN and DN for a class of generalized Gaussian sources. Past work on the optimal support of uniform scalar quantization, which is lattice quantization in one dimension, has included numerical optimization 3, 4] and curve tting 5] 8]. Nonlinear equations that can be solved numerically to give good
This work was supported by an NSF Graduate Fellowship and by NSF Grant NCR9415754 and was submitted for publication in the IEEE Transactions on Information Theory, with coauthor David L. Neuho . Part of this work was presented in September 1996 at the International Symposium on Information Theory and Its Applications in Victoria, Canada. 2 For a given N , the scaling factor a uniquely determines the quantizer support.
1
16
17 approximations to aN and DN have also been developed 6, 9, 10]. The most comprehensive work to date is that of Hui and Neuho 11], who found asymptotic formulas for aN and DN for a large class of sources. For multidimensional lattice quantization, determination of aN has been performed largely by experimentation 12] 17]. Jeong and Gibson 18] developed a nonlinear equation whose solution gives an approximation to aN . Their analytical solution was then shown to match simulation results for the 16dimensional integer lattice, when the source is memoryless Gaussian and memoryless Laplacian. In this paper, we consider a class of stationary, memoryless, generalized Gaussian sources and nd asymptotic formulas for aN and DN . This work generalizes some of the results in 11] to arbitrary dimension. For the strictly Gaussian case, similar expressions have been derived independently by Eriksson and Agrell 19]. We consider a zeromean random vector X with independent and identically distributed components, where each component has variance follows that X has kdimensional density " #k " X # k p(x) = 2;( 1 + 1) exp ; jxi j
i=1
where
v u 1 u ;(3= ) = t ;(1= )
;(x) =
Z1
0
tx;1 e;tdt:
(2.1)
18 where c makes the density integrate to one. For = 1 and = 2, the univariate GG density reduces to the Laplacian and Gaussian densities, respectively. When
! 1, the univariate GG density approaches a uniform density. In general the GG
density is de ned for all positive values of , but in this work, we restrict attention to 1, for reasons to be made clear later. The contours of equal probability of (r) = fx 2 <k : kxk where
kxk =
k X i=1
jxi j
!1=
(2.2)
where
Let v(F ) denote the volume of a kdimensional set F and let V = v( (1)) denote the volume of the unit radius ;sphere. It is known that 21, p. 108] h 2 1 ik ; V = : ; k +1 Consider a kdimensional in nite lattice with a fundamental cell S0 that is, is a countably in nite set of points f ig1 that is closed under addition, and S0 i=1 is a set with nite diameter such that f i + S0g1 is a partition of <k . Without i=1 loss of generality, assume that k ik
k i+1 k for all i. Consider the sequence
19 of lattice VQ codebooks fCN g1=1, where CN contains the N codevectors fyigN , N i=1 where y1 = quantizer as
SN =
N i=1
1 y2 = 2
::: yN =
N.
Si :
C The complement SN is the overload region. The lattice quantizer maps x 2 <k to a
codevector in CN according to the rule, 8 > yi < if x 2 Si QN (x) = > : C > arg min kx ; yk2 if x 2 SN : y2CN Note that as conventionally de ned, the ith quantization cell contains the ith support cell and possibly a portion of the overload region. We consider scaling the codebook by a positive and nite scaling factor a. The scaled codebook, denoted aCN , has codevectors ay1 ::: ayN and support cells aS1 ::: aSN . In this work, we use meansquared error (MSE) as the distortion measure, as given by 1Z D(N a) = k k kx ; QN (x)k2 p(x) dx: 2 < minimizes meansquare error, that is, (2.3)
For the codebook CN , we are interested in determining the scaling factor aN that
aN 6= 1. Therefore, the minimum exists but may not be unique. If the minimum is
not unique, we allow aN to be any scaling factor that achieves the minimum. The
DN = D(N aN ):
We will derive asymptotic expressions for aN and DN . We decompose the distortion into
D(N a) = Dg (N a) + Do (N a)
where Dg and Do are, respectively, granular distortion and overload distortion, given
N 1 X Z kx ; ay k2 p(x) dx Dg (N a) = k i 2 i=1 aSi 1Z kx ; QN (x)k2 p(x) dx: Do (N a) = k 2 (aSN )C Figure 2.1 illustrates aN and its relation to granular and overload distortion.
by
In general, it is di cult to work with the exact expressions for granular and overload distortion. As a result, we develop upper and lower bounds to granular distortion. These induce upper and lower bounds to total distortion. We then nd asymptotic expressions for the scaling factors aN and aN that minimize the upper and lower bounds, respectively. As with aN , the minima of the upper and lower bounds exist but are not unique. We allow aN and aN to be any scaling factors that achieve their respective minima. We show that aN =aN ! 1 as N ! 1 and that
d =
x y2cl(S0 )
max kx ; yk
21 where cl(S0) is the closure of S0. Because cl(S0) is compact, the maximum exists, and d is nite. The normalized moment of inertia (NMI) of S0 is given by 1R 2 k S0 kx ; y k2dx M (S0) = v(S )1+2=k 0 where y is the codevector of S0. The NMI is normalized in the sense that it is invariant to scaling and also in a perdimension sense. We de ne the equivalent
sphere of the codebook CN by
Seq N =
( N)
where
N
Nv(S0) V
!1=k
(2.4)
Note that SN and Seq N have the same volume and that Seq N does not depend on the ordering of the i's. Finally, in statements and proofs of some of our results, we use the notation c to denote a constant whose value is not important, and we use oN to indicate a quantity that tends to zero as N tends to in nity. The values of c and oN in one such statement are not necessarily the same as those in another. This rest of this paper is organized as follows. Section II presents the upper and lower bounds. Section III presents the main results of this paper, namely, an asymptotic expression for the scaling factor that minimizes lattice quantizer distortion. Proofs are given in Section IV. Some details of the derivation are relegated to the Appendix.
22
( (
N N
; 3d )
+ 3d ):
For a Gaussian source, inner and outer supports are illustrated in Figure 2.2. Corresponding to a codebook scaled by a, the inner and outer supports are aS N and aS N . The following result relates the support SN to its inner and outer supports S N and
SN .
Using the inner and outer supports, we can then derive bounds to overload distortion.
Lemma 2: For all su ciently large N , upper and lower bounds to overload distortion are given by
Do (N a)
; d ))2 p(x) dx
23
and
Do(N a)
where
N
Do(N a)
N
def =
c2 Z L k (aSN )C kxk ; a
p(x) dx
; 3d ,
+ 3d ,
(2.5)
Note that the maximum (resp. minimum) exists, because p is continuous and cl(Si) is compact, but may not be unique. If the maximum (resp. minimum) is not unique, we allow yi (resp. yi) to be any vector that achieves the maximum (resp. minimum).
Dg (N a)
def =
M (S0 ) v(S0
)1+2=k ak+2
N X i=1
p(ayi)
Dg (N a)
D g (N a )
def =
M (S0) v(S0
)1+2=k ak+2
N X i=1
p(ayi):
24
Thus we have reduced the problem of characterizing the scaling factor aN of a kdimensional lattice quantizer to the solution of a onedimensional integral equation.
25 Note that (2.6) is similar to Equation (52) in Hui and Neuho 11], which characterizes the optimal support of uniform scalar quantization for all N . The integral equation in (2.6) leads to the following asymptotic expression for aN .
V v(S0)
!1=k
DN 2 2= M (S0)V 2=k : = k 2 (ln N )2= N 2=k This optimized upper bound indicates that lattice quantizer distortion decreases to
zero at least as quickly as (ln N )2= N ;2=k . We also see that when the lattice is scaled by aN , the overload distortion of the upper bound is asymptotically negligible, so that total distortion asymptotically equals granular distortion, i.e. DN = a2 M (S0) v(S0), N when N is large. However, derivation of aN requires consideration of both granular and overload distortion. In order to provide more de nitive conclusions about the distortion of lattice quantization, we now consider minimizing the lower bound to distortion. Because the upper and lower bounds have a similar form, minimizing the lower bound is very similar to minimizing the upper bound. Accordingly, we present the following results without proofs.
26
V v(S0)
!1=k
2 2= M (S0)V 2=k : DN = k 2 (ln N )2= 2=k N Again, we see that overload distortion is asymptotically negligible.
27 is negligible when analyzing VQ. For scaleoptimized lattice quantization, as with the upper and lower bounds, the overload distortion is asymptotically negligible. Therefore for large N , DN has the form
2=k 2= 2=k DN = M (S0)V (ln N 2=k) : 2 N
(2.7)
optimal vector quantization 22], 23], where c is a constant that depends on and
while the distortion of asymptotically optimal quantization is (R) = c 2;2R . Equivalently, the signaltonoise ratio (SNR) of scaleoptimized lattice quantization is
S (R) = 6R ; 20 log10 R + c dB
as compared to 6R + c dB for asymptotically optimal vector quantization, where the \c" constants depend on and k. In other words, both lattice quantization and optimal quantization have SNRs that increase as 6 dB per bit, but the di erence between their SNRs increases to in nity as rate increases. As ! 1, the GG distribution tends uniform. For large , assuming Gersho's conjecture 23] and that and S0 are chosen optimally, it can be shown that DN =
28
N.
Thus our results are consistent with the widelyheld belief that lattice quantizers
C. Summary
We have derived asymptotic expressions for the optimal scaling factor and resulting minimum distortion of lattice vector quantization. This scheme assumes xedrate quantization and xed dimension, and our results apply to stationary, memoryless, generalized Gaussian sources. It is shown that scaleoptimized lattice quantizer distortion diverges from that of asymptotically optimal quantization as rate increases.
inequality in this and other proofs. In order for the inner support to make sense, we
; 3d ) to be nonnegative. This occurs when N is su ciently
For all x 2 SN ,
kxk kx ; QN (x)k + kQN (x)k
N.
d +
Therefore,
N
x2
def =
+d )
min kyk : nC
N
kxk
ky k ; ky ; xk >
; 2d :
Therefore
x 62
def =
C N
; 2d )
N
SN .
From the above, we know that v( N ) v(SN ) v( N ). Since v(SN ) = v(Seq N ), we see that v( N ) v(Seq N ) v( N ). Because 2d
N N +d
Seq N
at 0 with increasing volumes, it follows that they have increasing radii, i.e., . As a result,
N N +3d N +d
and S N
SN . Similarly,
B. Proof of Lemma 2
To prove the upper bound, let naS N (x) be the closest point in aS N to x. Then for all x 2 (aS N )C ,
kx ; QN (x)k2 kx ; QN (naS N (x))k2 kx ; naS N (x)k2 + knaS N (x) ; QN (naS N (x))k2
(2.8) (2.9)
; 3d )] + a d )
30 where (2.8) follows from the triangle inequality and (2.9) follows from (2.2). Since (aSN )C (aS N )C by Lemma 1, we then have 1Z Do (N a) = k kx ; QN (x)k2 p(x) dx 2 (aSN )C Z 1 kx ; QN (x)k2 p(x) dx 2 C k (aSN ) c2 Z U (kxk ; a( N ; d ))2 p(x) dx C k (aS ) = Do(N a):
N
+ 3d ))
where the rst inequality follows from (2.2). Therefore, observing that (aS N )C (aSN )C by Lemma 1, 1Z kx ; QN (x)k2 p(x) dx Do (N a) = k 2 (aSN )C Z 1 kx ; QN (x)k2 p(x) dx 2 C k (aS N ) 2 c2 Z L kxk ; a N p(x) dx C k (aSN ) = D o ( N a)
C. Proof of Lemma 3
It can be easily shown that for i = 1 ::: N ,
ayi = arg xmini p(x) 2aS ayi = arg x2aSi p(x): max
(2.10) (2.11)
31 To prove the upper bound, consider granular distortion and note that N 1 X Z kx ; ay k2 p(x) dx Dg (N a) = k i 2 i=1 aSi N X 1 Z kx ; ay k2 dx p(ayi) k i 2 aSi i=1 N X = M (S0) v(S0)1+2=k ak+2 p(ayi) = Dg (N a)
i=1
where the inequality follows from (2.11) and the penultimate equality follows from the de nition of NMI and the fact that v(aSi) = ak v(Si). Using (2.10), it can similarly be shown that Dg (N a) Dg (N a). This completes the proof of the lemma.
D. Proof of Proposition 5
It is easily seen that aN is a solution to
@ D(N a) = @ D (N a) + @ D (N a) = 0: @a @a g @a o
To evaluate the partial derivative of overload distortion, one can use Leibniz' rule to show that 1 @ D (N a) a @a o
= M (S0) v(S0)2=k (k + 2)
;
N X
i=1 N X i=1
Therefore,
E. Proof of Proposition 6
To prove the asymptotic expression for aN , start by de ning, Z1 K (y ) = 1 y y x pR(x)dx R 1 (x ; y) p (x) dx R J (y) = y R 1 x p (x)dx
y R
which were rst considered in 11]. Rewriting (2.6) from Proposition 5 in terms of
; d ) K (aN
N)
d +(
; d ) J (aN
N )]
+ oN
(2.14)
33
0 to obtain
N )]
; d ) K (aN
N)
N N)
; d ) J (aN
N)
( = (
; d ) K (aN
d +(
; d ) J (aN
+ oN
; d ) K (aN
N)
and take the lim sup. To show (b), use the bound J (aN N ) 1 in (2.14) to obtain
N N N)
( =
; d ) K (aN
d +(
; d ) J (aN
N )]
+ oN
and take the lim inf of the resulting inequality. 2 Returning to the proof of Proposition 6, we note that R is a generalized Gamma random variable, so from Hui and Neuho 11],
yields
(2.15)
Taking the logarithm of both sides and using (2.4), we see that (2.15) is equivalent
v(S0) V
=k
aN N 1=k
(1 + oN ):
34 Therefore, we see that asymptotically, the radius of the optimal quantizer support,
aN N 1=k , grows as (ln N 2=k )1= . This in turn implies that ! aN N 1=k = 1 V 1=k (1 + o ) N (ln N 2=k )1= v(S0)
and taking the limit N ! 1 gives the asymptotic expression for aN . We now establish the asymptotic expression for DN . It is immediately seen that
Dg (N aN ) satis es
lim Dg (N2 aN ) = M (S0) v(S0)2=k Nlim N !1 !1 a
N N X i=1
= M (S0) v(S0)2=k
where the last equality follows from Lemma A1. We now show that overload distortion is asymptotically negligible by showing that
N !1
lim
R1
aN
N
(r ; aN (
N ;d a2 N
))2 pR(r)dr
= 0:
(2.17)
We rst observe that R 1 (r ; a ( ; d ))2 p (r)dr N N R aN N 2 a R 1 (rN; a )2 p (r)dr N N R = aN N + 2 aN d a2 N Z1 + a2 d2 pR (r)dr N aN N R1 2 aN N (r ; aN N ) pR (r)dr + 2 aN d a2 N Z 2 d2 1 p (r)dr + aN a R
N N
R 1 (r ; a ) p (r)dr N N R aN N aN vR 1 u u aN N (r ; aN N )2 pR (r)dr t a2 N
where lim N !1
Z1
aN
N
pR(r)dr = 0
35 which follows from Lemma 4. Therefore, in order to show (2.17), it is su cient to show
R 1 (r ; a )2 p (r)dr N N R lim aN N = 0: N !1 a2 N
2 N
(2.18)
To show (2.18), we note that R1 2 aN N (r ; aN N ) pR (r)dr = a2 N Therefore, by Lemma 11, R 1 (r ; a )2 p (r)dr N N R lim sup aN N a2
N !1 N
= = = 0
R 1(r ; y)2p (r)dr lim sup y yR 1(r ; y)pR (r)dr y!1 R y R 1(r ; y)p (r)dr 2 R lim sup y R 1 p (ry dr ; R 1 (r ; y)p (r)dr y!1 R y R ) y 2 y) lim sup 1 ;J2(J (y) y!1
where the second equality follows from L'Hopital's Rule and where the last equality follows because J (y) ! 0 as y ! 1 for generalized Gamma densities 11]. Therefore, (2.18) holds, which establishes the asymptotic negligibility of Do(N aN ). Proposition 6 then follows from (2.16).
F. Proof of Proposition 10
Note that
D(aN )
D(aN )
D(aN )
D(aN ):
(2.19)
Propositions 6 and 8 imply that D(aN )=D (aN ) ! 1 and hence by (2.19), lim D(aN ) = 1: N !1 D (a )
N
(2.20)
36
e We will show that if a sequence of scaling factors aN satis es e ) lim D(aN ) = 1 N !1 D(a
N
(2.21)
then
e a lim aN = 1: N !1
N
(2.22)
e With aN
aN , this fact, together with (2.20), then proves the proposition. We will
prove that (2.21) implies (2.22) by showing the contrapositive, namely, if (2.22) does not hold, then (2.21) does not hold. Accordingly, we assume that there exists > 0 and a sequence of integers increasing to in nity, denoted fNng1=1 , such that either n
en b > 1+ bn
or
for all n
(2.23)
en b < 1; bn
b D(en) > 1 + D(bn)
for all n
(2.24)
where en = aNn and bn = aNn . We will show that this implies b e for all su ciently large n (2.25)
these properties, because if not, it is obvious that D(en) 6! 0, whereas D(bn ) ! 0, so b (2.25) holds. Because bn and en have these properties, an argument similar to that b in (2.16) shows that Dg (en ) = e2 (1 + on ) and Dg (bn ) = b2 (1 + on ), where is a b bn n constant.
37 Now suppose (2.23) holds. It is easily shown that a condition similar to (2.18) holds for D(bn ), which implies that D(bn ) = Dg (bn) (1 + on ). Therefore, we see that Dg (en ) b b b D(en) = Dg (en ) + Do(en) b D(bn ) Dg (bn ) + Do(bn ) Dg (bn)(1 + on ) e2 n = b2 (1 + on) (1 + )2 (1 + on ) bn (1 + on) so that for large enough n, D(en)=D(bn ) > 1 + , which establishes (2.25). b Next suppose (2.24) holds. We will show later that D e lim D o(bn) = 1: n!1 g (bn ) It will follow that en 2 Dg (bn) (1 + on ) + Do(en) b b en) + Do(en) en) b = bn D(b = Dg (b D(bn) Dg (bn ) + Do(bn) Dg (bn)(1 + on ) # " 2 2 D (e ) en Dg (bn ) 1 + bn b o bn + o bn en Dg (bn ) n b = Dg (bn)(1 + on ) 0 !1 Do (en) + o en 2A (1 + o ) b b @ n D (b ) n b
g n n
(2.26)
D e = Do(bn) + on ! 1 g (bn ) b so that for n large enough, D(en)=D(bn ) > 1 + , which again establishes (2.25).
In order to prove (2.26), we will show that R 1 r ; e 2 p (r)dr bn n R e b lim n n = 1 n!1 b2 n where n = Nn . It can be shown that 11] Z1 (r ; y)2 pR(r) dr = c exp ; y ] (1 + on ): y (2.27)
The above fact, together with en < (1 ; )bn , shows that b Z1 Z1 2 2 r ; en n pR (r)dr b r ; (1 ; )bn n pR (r)dr en n b (1; )bn n h i = c exp ; ((1 ; )bn n ) (1 + on ): (2.28)
V v(S0)
!1=k
(2.29)
Then, for large enough n, the righthand side of (2.28) is given by,
(2.30)
In order to prove (2.27), note that (2.29) and (2.30) imply that R 1 r ; e 2 p (r)dr bn n R =k e b c nlim (lnNn )2= (1 + on ) = 1: lim n n !1 n!1 Nn b2 n This completes the proof of (2.26) and the proof of the proposition.
< 1, then
N !1 i=1
lim
N X
39
Remark: This lemma explicitly includes the case = 0, which is needed in the
proof of Proposition 5.
Proof: Notice that the sum is over all support cells. We will group the support cells
and separately evaluate the portion of the above sum due to each of these groups. In order to establish these groups, we x s < 1, to be chosen later, and consider an
;sphere with radius s. The various groups are determined by the relation of the
(s)g
R(1)bN = N
i2IN bN
(1)
bN Si:
(1) Note that, for each N , every index in IN bN corresponds to a support cell that is
(s)C g
R(3)bN = N
i2IN bN
(3)
bN Si
R(2)bN = N
Next we show that for each N ,
i2IN bN
(2)
bN Si:
R(2)bN N
(s + bN d ) n (s ; 2 bN d ):
(2.31)
(2) By the construction of R(2)bN , it is easily seen that for each i 2 IN bN , there exists a N
= kx ; zk + kzk
bN d + s:
40
(2) This holds for all x 2 bN Si and all i 2 IN bN , so R(2)bN N (2) any i 2 IN bN and any x 2 bN Si, we have
(s + bN d ). Similarly, for
kxk
kz k ; kz ; xk > s ; 2 bN d
Using (2.31) and the fact that p(x) is bounded, there exists an M such that
X
i2IN bN
(2)
X
i2IN bN
(2)
v(bN Si)
X
i2IN bN
(2)
(2.32)
since bN ! 0. Since (s) is bounded and p(x) is continuous on (s), we see that Z X eik p(bN yi) v(bN Si) = e kbN y kxk p(x) dx + oN
i2IN bN
(1)
(s)
(2.33)
for all s, because as N ! 1, bN ! 0 and P SN bN Si ! 1, which follows from i=1 the fact that bN N 1=k ! 1. In other words, the lefthand side of (2.33) converges to its Riemann integral. Next, we will show that
X
i2IN bN
(3)
(2.34)
41 where E (s) does not depend on N and E (s) ! 0 as s ! 1. In order to show (2.34), we note that
e kbN yi k
kbN yi k + bN d
2 kbN yik
e kbN y i ; bN yi k
where yi and yi were de ned in (2.5), and where the last inequality in (2.35) and the last inequality in (2.36) assume su ciently large N . Therefore, lim sup
N !1 i2I (3) N bN
2
def =
+1
(3) i2IN bN
bN Si
E (s)
(s)C
kxk p(x) dx
which establishes (2.34). Combining (2.32)(2.34), we then see that Z N X e e kbN yi k p(bN yi ) v (bN Si ) = E kX k ;
i=1
(s)C
which implies that N X e e lim sup kbN yik p(bN yi) v(bN Si ) ; E kX k
N !1 i=1
Z
(s)C
Since the above holds for all s, N X e e lim sup kbN yik p(bN yi) v(bN Si) ; E kX k
N !1 i=1
= 0
42
References
1] J.H. Conway and N.J.A. Sloane, Sphere Packings, Lattices, and Groups, 2nd ed. New York: SpringerVerlag, 1993. 2] J.D. Gibson and K. Sayood, \Lattice quantization," Advances in Electronics and Electron Physics, vol. 72, pp. 259330, 1988. 3] J. Max, \Quantizing for minimum distortion," IEEE Trans. Inform. Theory, vol. IT6, pp. 712, Mar. 1960. 4] M.D. Paez and T.H. Glisson, \Minimum meansquarederror quantization in speech PCM and DPCM systems," IEEE Trans. Commun., vol. COM20, pp. 225230, Apr. 1972. 5] W.A. Pearlman and G.H. Senge, \Optimum quantization of the Rayleigh probability distribution," IEEE Trans. Commun., vol. COM27, pp. 101112, Jan. 1979. 6] J.A. Bucklew and N.C. Gallagher, Jr., \Some properties of uniform step size quantizers," IEEE Trans. Inform. Theory, vol. IT26, pp. 610613, Sept. 1980. 7] F.S. Lu and G.L. Wise, \A simple approximation for minimum meansquared error symmetric uniform quantization," IEEE Trans. Comm., vol. COM32, pp. 470474, Apr. 1984. 8] G. Vasquez, \Comments on `A simple approximation for minimum meansquared error symmetric uniform quantization'," IEEE Trans. Comm., vol. COM34, pp. 298300, Mar. 1986. 9] G.M. Roe, \Quantizing for minimum distortion," IEEE Trans. Inform. Thy., vol. IT10, pp. 384385, Oct. 1964. 10] V.R. Algazi, \Useful approximations to optimum quantization," IEEE Trans. Commun., vol. COM14, pp. 297301, Jun. 1966. 11] D. Hui and D.L. Neuho , \Asymptotic analysis of optimal xedrate uniform scalar quantization," submitted to IEEE Trans. Inform. Theory, August 1997. 12] K. Sayood, J.D. Gibson and M.C. Rost, \An algorithm for uniform vector quantizer design," IEEE Trans. Inform. Thy., vol. IT30, pp. 805814, Nov. 1984. 13] M.C. Rost and K. Sayood, \The root lattices as low bit rate vector quantizers," IEEE Trans. Inform. Thy., vol. 34, pp. 10531058, Sept. 1988. 14] D.G. Jeong and J.D. Gibson, \Lattice vector quantization for image coding," in Proc. 1989 IEEE Int. Conf. Acoust., Speech, Signal Processing, Glasgow, Scotland, May 2326, pp. 17431746. 15] M.V. Eyuboglu and G.D. Forney, Jr., \Lattice and trellis quantization with lattice and trellisbounded codebooks  highrate theory for memoryless sources," IEEE Trans. Inform. Thy., vol. 39, pp. 4659, Jan. 1993. 16] F. Chen, Z. Gao and J. Villasenor, \Lattice vector quantization of generalized Gaussian sources," IEEE Trans. Inform. Thy., vol. 43, pp. 92103, Jan. 1997.
43 17] C. Pepin, J.C. Bel ore and J. Boutros, \Quantization of both stationary and nonstationary Gaussian sources with Voronoi constellations," in Proc. 1997 IEEE Int. Symp. on Information Theory (Ulm, Germany, June 1997), p. 59. 18] D.G. Jeong and J.D. Gibson, \Uniform and piecewise uniform lattice vector quantization for memoryless Gaussian and Laplacian sources," IEEE Trans. Inform. Thy., vol. 39, pp. 786804, May 1993. 19] T. Eriksson and E. Agrell, \Latticebased quantization, Part II," Report No. 18, Department of Information Theory, Chalmers University of Technology, Goteborg, Sweeden, Oct. 1996. 20] A. Wilansky, Functional Analysis. New York: Blaisdell, 1964. 21] R.M. Gray, Source Coding Theory. Boston: Kluwer, 1990. 22] P.L. Zador, \Topics in the asymptotic quantization of continuous random variables," Bell Laboratories Technical Memorandum, 1966. 23] A. Gersho, \Asymptotically optimal block quantization," IEEE Trans. Inform. Theory, vol. IT25, pp. 373380, July 1979.
44
Do (N,a)
D(N,a) D g (N,a)
aN
CHAPTER III
I. Introduction
Multidimensional companding is a type of structured vector quantization (VQ) with low complexity. As shown in Figure 3.1, a compander maps a kdimensional source vector X to a kdimensional vector Y in some rectangular support region, using a continuous compressor function f . The transformed source vector Y is quan^ ^ tized using a lattice quantizer to Y , which is then mapped by f ;1 to X , which is the reproduction of X . In this work, only xedrate quantization (i.e. no entropy coding) is considered. The lattice quantizer has as its codevectors, all points from some in nite lattice that are contained in the rectangle. When given a compressor function and a desired number of codevectors N , we assume that the lattice is maximally scaled so that the number of lattice points in the corresponding support rectangle is at most N . We also assume that the partition of the lattice quantizer is
This work was supported by an NSF Graduate Fellowship and by NSF Grant NCR9415754 and was submitted for publication in the IEEE Transactions on Information Theory, with coauthor David L. Neuho . Part of this work was presented at the 1997 IEEE International Symposium on Information Theory in Ulm.
1
45
46 generated by tesselating some fundamental cell T0 at points of the lattice. (We will not be concerned with the manner in which points outside the support rectangle are assigned to cells.) The resulting compander is a VQ whose codevectors and partition are those of the lattice quantizer transformed by f ;1. In pioneering work, Bennett 1] argued that any scalar quantizer (e.g. optimal) can be implemented by companding and heuristically derived an asymptotic expression for the mean squared error (MSE) of scalar companding. Speci cally, when N is large, companding MSE is given by
where c makes f (x) integrate to one. Using the optimal compressor in Bennett's formula shows that asymptotically optimal scalar quantizers have MSE given by
D (N ) = 121 2 kpk1=3 N
For kdimensional companding, Bucklew 2] extended Bennett's scalar result to show that when N is large and the fundamental cell T0 is white in the sense that that the components of a random vector uniformly distributed over T0 are white, the compander MSE D(f N ) is approximately given by D(f N ) = DB (f N ), where
(3.1)
47 and2
G(f ) =
Z 1 ;1 2 k kF (x)k p(x) dx
(3.2)
v denotes kdimensional volume, f (<k ) is the image of the compressor function f , T0 is the fundamental cell of the lattice
4], F (x) is the derivative matrix of the compressor function f (x), F ;1(x) is the inverse of the matrix F (x), and k k is the
l2 matrix norm.3 M (T0) is the normalized moment of inertia (NMI) of T0, given by R M (T0) = k;1v(T0);(1+2=k) T0 kxk2 dx. Bucklew also showed that an optimal twodimensional vector quantizer for a source with a circularly symmetric density cannot be implemented using a companding structure and a di erentiable compressor function. See also Gersho 5]. Subsequently, Bucklew 3] showed that asymptotically optimal vector quantizers, for vector dimension three and greater, cannot be implemented using a companding structure, except for a very restricted class of source densities. An important open question remains: What is the best multidimensional compressor function for an arbitrary source? This is a di cult unsolved problem. As a rst step, in this paper we nd the asymptotically optimal compressor function for a memoryless source, under certain technical conditions on the compressor function. The result agrees with intuition and shows that the best kdimensional compressor function consists of the best scalar compressor functions for each component of the source vector. Though we suspect that this result holds over a broader class of compressors, we were not able to show this. We also compare the MSE of optimal multidimensional companding to that of optimal VQ and optimal scalar companding. For example, we show that companding su ers the same point density and oblongitis
2 3
In 2], ( ) did not contain ( (<k ))2=k , because it was assumed that (<k ) = (0 1)k . That is, the sum of squares of the matrix elements.
G f v f f
48 losses as scalar quantization, but recovers the cubic loss. The results in this paper can be easily generalized to pth power distortion measures. An outline of the paper is the following. The main result is stated and discussed in Section II. Optimal companding is compared to optimal VQ and scalar quantization in Section III. A derivation of the main result is presented in Section IV, with some details given in the Appendix.
p(x) =
k Y i=1
(3.3)
f : <k ! <k , a lattice quantizer constructed from a lattice , and the inverse function f ;1 . The lattice
is assumed to have a fundamental cell T0 that is white.4 The support region of the lattice quantizer is a rectangle of the form R(a b) := (a1 b1) (ak bk ) for some a b 2 <k . Without loss of generality, we assume ai < bi, for all i.
De nition 1 For a b 2 <k , a function f : <k ! <k is boundary limited to R(a b) if for each i = 1 ::: k , fi(x) ! ai when xi ! ;1 for any xed x1 ::: xi;1 xi+1 ::: xk, and fi(x) ! bi when xi ! +1 for any xed x1 ::: xi;1 xi+1 ::: xk.
In this paper, we only consider compressor functions f in a class C , as de ned below.
4
In any dimension, the integer lattice and the optimal lattice are white 14].
49
De nition 2 The class C is the set of all mappings f : <k ! <k that are onto
and boundary limited to some rectangle R(a b), and whose derivative matrix F (x) is continuous in x and positive de nite5 for all x.
We are interested in nding the compressor function that minimizes G(f ) over the class C and the resulting minimum MSE that is,
(3.4)
Proposition 1 Let the source be memoryless, as given by (3.3). Then over the class C of compressor functions, there exists a minimum of G(f ), and a compressor
function achieves the minimum if and only if it is a memoryless compressor of the form f (x) = (f1 (x) f2 (x)
for some c 2 < and a 2 <k . The image of the resulting compressor function is
R(a b ) where bi = ai + c kpi k1=2 i = 1 ::: k: The resulting MSE is, asymptotically, 1=3 0k 11=k 1 M (T ) @ Y kp k A DB (N ) = N 2=k (3.5) 0 j 1=3
j =1
Remarks:
5
50 1. We do not assume that the compressor functions in C are onetoone. However, Proposition 1 demonstrates that the optimal compressor functions are onetoone. 2. The optimal compressor function operates independently on each component of the source vector with the optimal compressor function for scalar companding. 3. We have restricted attention to compressor functions with a rectangular image. It is possible that a compressor function with a nonrectangular image could yield lower MSE than f . However, the resulting optimal image might have a very complex shape, making the implementation of the lattice quantizer very complex. This would defeat the motivating idea behind companding, namely that it have low complexity. 4. If the functions kpi k1=3 are equal, then the optimal image is a cube. If not, then the optimal image is a rectangle, which induces a kind of rate allocation among the components of X , with Xi receiving bits in proportion to log2 bi . This is easiest to see when = Z k , the kdimensional integer lattice for in this case, companding is simply product quantization, and optimal compander implements an optimal product quantizer, which as is well known, assigns k1 Qkkpikp=j3k N levels to Xi . But it also holds for any other white lattice, as does
j =1
1=3
the fact that the resulting distortions are the same for each component, which is well known for optimal product quantizers. 5. If f (<k ) were restricted to be a cube, then one may see from the proof of Proposition 1 that the optimal compressor function fe would have component
51 functions
fei(x) :=
where c 2 < and a 2 <k . In this case, the loss in MSE of companding to a cube instead of to the optimal rectangle is given by DB (fe N ) = 1 Pk=1 kpj k1=3 k i Qk kp k 1=k DB (f N ) j =1 j 1=3 1
kpi k
1=3 1=3
Z xi
;1
pi (y)1=3dy + ai
m = N 2k kpkk=(k+2) =k
under the assumption that optimal VQ has cells that are congruent to the tesselating polytope with least NMI, mk , in k dimensions. In order to facilitate this comparison, we provide a point density, cellshape analysis in the style of 10]. Recall from 10] that Bennett's integral gives the MSE of a VQ with many points in terms of two key characteristics, the point density (x) and inertial pro le m(x). Speci cally, Bennett's integral is given by 1 Z m(x) p(x) dx: B ( m) = N 2=k (x)2=k (x) = c p(x)k=(k+2) and
optimal inertial pro le m (x) = mk . Let the point density and inertial pro le of
Constants in this section make their respective functions integrate to one.
52 companding be denoted by
C (x)
M (To)
(3.6)
where the approximate equality in (3.6) assumes, as usual, that the fundamental cell of the lattice is white, and where the inequality, which is proved in Appendix I, holds with equality if and only if F (x) is a scaling of an orthogonal matrix. Equation (3.6) re ects the fact that the the cells of the compander are stretched versions of T0, except where and only where F (x) is a scaling of an orthogonal transformation. We can isolate the e ect of this stretching, or oblongation, by decomposing the cell shape loss into Lce = Lob Lce T0 , where Lob is the oblongitis loss 10], and Lce T0 =
M (T0) mk
is
the cell shape loss of the lattice, which is independent of the compressor function
53 for i = 1 ::: k. In this case Lpt = 1, but it can be shown that this causes Lob = 1. In other words, when the compander operates with a compressor function speci ed by (3.7), MSE does not decrease as N ;2=k . On the other hand, a compander achieves
Lob = 1 if for all x, k;1 kF ;1(x)k2 (det F (x))2=k = 1, which is approximately true if and only if the compressor function is a scaling of an orthogonal transformation
on some large probability set.7 In this case Lpt = 1. As a result, we see that companding cannot simultaneously achieve Lpt = 1 and Lob = 1 and therefore cannot achieve the performance of optimal VQ, which agrees with Bucklew's analysis 2]. In fact the best compressor f , presented in Proposition 1, is a compromise which yields a compander with Lpt > 1 and Lob > 1. It is immediately seen that companding makes the same tradeo as optimal product VQ 10] (it generates the same point density), and therefore has the same Lpt and Lob. Companding's sole advantage over product VQ is its ability to choose the lattice that achieves Lce T0 = 1, whereas product VQ must incur cell shape loss due to cubic cells, that is Lce cube > 1. Therefore, optimal companding su ers the same point density and oblongitis losses as optimal product VQ while recovering the cubic loss. Table 3.1 summarizes the losses of optimal companding and optimal product VQ, relative to optimal VQ, for an IID Gaussian source. Note that Lcomp = Lpt +
Lob, which is the shape loss8 of optimal scalar quantization relative to optimal VQ,
de ned by Lookabaugh and Gray 9]. On the other hand, optimal product VQ has loss Lprod = Lpt + Lob + Lce cube, which shows that companding achieves the
space lling advantage of vector quantization over scalar 9], or equivalently, the
inverse of the cubic loss of optimal product quantization relative to optimal vector
See Appendix I. Actually, 9] de nes the inverse of this to be the shape advantage of vector quantization over scalar quantization.
7 8
54
Lpt
Lcomp Lprod
1.14 1.61 1.87 2.81 1.30 1.87 2.27 4.35
(dB) (dB) (dB) (dB) 0.64 0.50 0.75 0.86 0.81 1.06 1.71 1.10
Table 3.1: Losses of optimal companding, Lcomp , and optimal product VQ, Lprod, for an IID Gaussian source. quantization 10].
f is the optimal compressor function. It is a well known fact that a stationary point
of a convex functional on a convex set is a minimizing function. Upon initial inspection of the companding problem, it is evident that G(f ) is not a convex functional and that C is not a convex set (due to the onto assumption), so that we cannot apply the above fact directly. However, we will consider certain subclasses of C and decompose G(f ) into the product of two functionals, one of which is strictly convex on each subclass and the other of which is constant on each subclass. Though these subclasses are not convex, it will nevertheless be possible to minimize G(f ) over them using the above fact. We then perform an auxiliary minimization that leads to the main result.
55 We have de ned C so that we can use the above approach for optimization. Continuity of the derivatives and the boundary limited assumptions are needed to show that a function is a stationary point. The assumption that f be onto R(a b) is needed to establish the aforementioned product decomposition of G(f ). Finally, we require that the derivative matrix of f be positive de nite to ensure the convexity of
De nition 3 For a b 2 <k , the class Cab is the subset of C such that f is onto and
boundary limited to R(a b).
We wish to minimize G(f ) over C and let G denote the resulting minimum value. Then
Gab =
ff :f 2Cabg
min G(f ):
It will be shown that the minima G and Gab exist. We decompose G(f ) into
G(f ) = V (f ) H (f )
56 where
V (f ) = H (f ) =
Z
f (<k )
dy
!2=k
Z
<
det F (x) dx k
2=k
1 h(f F x) = k kF ;1(x)k2 p(x): To nd G , we rst nd Gab for arbitrary a b and then perform the auxiliary minimization of G over the choice of a b. Since V (f ) = Qk (b ; a )2=k , for all f 2 C ,
ab i=1 i i ab
h(f F x) dx
we have
Gab =
k Y
i=1
and the minimum will be shown to exist. Therefore, it remains to minimize H (f ) on Cab. Though Cab is not convex (due to the onto assumption), relaxing the onto e assumption leads to a larger class Cab that is convex, and fortunately, such that the compressor that minimizes H (f ) over this larger class is in Cab.
De nition 4 For a b 2 <k , the class Ceab is the set of all mappings f : <k ! <k
that are boundarylimited to R(a b) and whose derivative matrix F (x) is continuous in x and positive de nite for all x.
e e Cab. And it is easy to check that Cab is convex. e Lemma A1 shows that H (f ) is strictly convex on Cab and Lemma A2 shows that
We see immediately that Cab
fab := fab 1
fab k ]T , where
bi ; ai Z xi p (y)1=3dy + a fab i(x) := i kpi k1=3 ;1 i 1=3 e is a stationary point of H (f ) on Cab. (The de nitions of convexity and stationary
point are given in Appendix II.) Since H (f ) is strictly convex, we see that fab
57
e Cab, we see
that fab uniquely minimizes H (f ) on Cab, as well. It follows immediately that fab
Gab =
To minimize Gab over the choice of a b, we apply the arithmeticgeometric mean inequality. Speci cally, for any a b,
b G(a b) =
k Y
k Y i=1
(bi ; ai)2=k
exists a constant c such that bi ; ai = c kpik1=2 i = 1 ::: k: It follows then that 1=3 0k 11=k Y G = @ kpj k1=3A
j =1
(bi ;a1 )2
kpik1=3
and that the compressor functions that achieve G are f (x) = f1 (x) where
fk (x)]T
fi (x) = c kpi k
Proposition follow directly.
1=6 1=3
Z xi
;1
pi (y)1=3dy + ai i = 1 ::: k
where c 2 < and a 2 <k are arbitrary. The image and MSE formulas stated in the
58
1
m(x) = M (To) k;1 kF ;1(x)k2 (det F (x))2=k ! Y ! k k 2=k ;1 X 1 = M (To) k i (x) 2 i=1 i=1 i (x) ! k k Y 1 ! Y 2=k M (To) i (x) 2=k i=1 i (x) i=1 = M (To)
where it is easily seen that kF ;1(x)k2 = tr F ;1(x)T F ;1(x)] = Pk=1 i
;2 i (x),
and
where the inequality follows from the arithmeticgeometric mean inequality. We have equality if and only if all i (x)2 are equal for all x, which occurs when and only when F (x) is a scaling of an orthogonal transformation.
Appendix II De nitions
This appendix introduces certain de nitions and facts that will be needed in the sequel. Let G denote a linear space of continuous functions, and let F denote a convex subset of G that is, f1 + (1 ; )f2 2 F whenever f1 f2 2 F and 0
at f1 in the direction towards f2 is given by
1.
59 If J has a minimum point on F , then it is either a stationary point or a point in the boundary of F . The functional J is said to be convex on the convex set F if J ( f1 + (1 ; )f2)
convex if the equality is strict whenever f1 6= f2. Equivalently 12, Def. 3.1], J is
J (f2) ; J (f1)
and strictly convex if the inequality is strict whenever f2 6= f1. If f is a stationary point of a convex functional J , then f minimizes J . Furthermore, f is the unique minimizer of J on F if J is strictly convex.
Proof: By the condition for convexity given in the previous appendix it su ces to e show that for all f1 f2 2 Cab, the function H (f ) satis es
H (f2) ; H (f1) H (f1 f2 ; f1)
(3.8)
60 with equality if and only if f1(x) = f2(x) for all x. Let F1(x) and F2(x) be the derivative matrices of f1 and f2. By the de nitions of H and H , the above is equivalent to Z
=0
Lemma A3, given later, shows that the function kA;1k2 is convex on the space of positive de nite matrices A. By the de nition of C , it follows from such convexity that
kF2;1(x)k2 ; kF1;1(x)k2
=0
for all x,
with equality for some x if and only if F1(x) = F2(x). It now follows directly from this that (3.8) holds, with equality if and only if F1(x) = F2(x) for all x, which in turn e is equivalent to f1(x) = f2(x) for all x, because compressors in Cab are continuous and boundary limited to R(a b). (Since the f 's (and F 's) are continuous, if they are equal almost everywhere, they are equal everywhere.) This completes the proof of the lemma. 2.
e Lemma A2 fab is a stationary point of H (f ) on Cab. e Proof: It is easily veri ed that fab 2 Cab. To show that fab is a stationary point of e H (f ) on Cab, we will show that e H (fab f ; fab) = 0 for all f 2 Cab:
Taylor series expansion, (3.9)
Fix f 2 Cab and de ne g = f ; fab and let G be the derivative matrix of g. Using a
61
Z @ = @ (h(fab Fab x) 1 k k XX @ A + @Fij h(fab Fab x) Gij (x) + o( ) i=1 j =1 # k X "Z @ k X = @Fij h(fab Fab x) Gij (x) dx : i=1 j =1
dx
=0
In order to show (3.9), we will show that for i = 1 ::: k and j = 1 ::: k, Z @ (3.10) @Fij h(fab Fab x) Gij (x)dx = 0 e for all f such that f 2 Cab , where g = f ; fab. First consider (3.10) for i 6= j . From the de nition of h and the fact that F is diagonal, the derivative of h reduces to
Therefore, Z @ @Fii h(fab Fab x) Gii (x) dx Z1 Z1 2 Y Z1 k k Y = ; k c3 pm (xm) Gii(x) dxi dxm = 0
;1 ;1
i m=1 m6=i
;1
m=1 m6=i
because
Z1
;1
for all x1 ::: xi;1 xi+1 ::: xk, by the fundamental theorem of calculus and the fact that since g is the di erence between two functions that are boundary limited to
Rab, gi(x) ! 0 when xi ! ;1 for any xed x1 ::: xi;1 xi+1 ::: xk, and gi(x) ! 0
when xi ! +1 for any xed x1 ::: xi;1 xi+1 ::: xk. Note that this argument relies
62 on the continuity of Gii therefore, it is necessary that the diagonal entries of Fab be continuous. By assumption, the densities pi are continuous, which implies that
Fab ii = c pi(xi)1=3 i = 1 :: k are continuous. It follows that (3.10) holds for all i j ,
which implies that fab satis es (3.9), and completes the proof of the lemma. 2
Lemma A3 The function kA;1k2 is strictly convex on the set of positive de nite
matrices.
Proof: Note that kA;1k2 = tr (A;1)T A;1 . We will show that for A B positive
de nite,
=0
(3.11)
Because they are both positive de nite, A and B can be written as A = CIC T and B = CDC T , where D is a diagonal matrix with positive diagonal elements
d11 d22 ::: dkk 8, Cor. 7.6.5]. Then the lefthandside of (3.11) can be written as h i h i tr (B ;1)T B ;1 ; tr (A;1)T A;1 n oT = tr (C ;1)T C ;1 (C ;1)T C ;1 (D;1 )2 ; I ] (3.12)
since DM = MD for any matrix M and any diagonal matrix D. We then see that (3.12) is given by tr
n oT (C ;1)T C ;1 (C ;1)T C ;1
(D;1 )2 ; I ]
= si d12 ; 1 ii i=1
k X
(3.13)
where si is the sum of column i of the matrix (C ;1)T C ;1. In order to simplify the righthandside of (3.11), we note that h i tr f(A + (B ; A));1gT (A + (B ; A));1
63
where (3.14) is derived using an argument similar to that in (3.13). Now the righthandside of (3.11) is given by
i=1
;2 si (dii ; 1)
i=1
k X
(3.15)
where si > 0. Since D and I are diagonal matrices with positive diagonal elements, it is su cient to show that f (x) = Pk s x;2 is strictly convex on fx : x > 0g,
i=1 i i i
which can be easily shown. Therefore (3.15) holds, with equality if and only if dii = 1 for all i, or equivalently when D = I and therefore A = B . 2
References
1] W.R. Bennett, \Spectra of quantized signals," Bell Syst. Tech. J., vol. 27, pp. 446472, July 1948. 2] J.A. Bucklew, \Companding and random quantization in several dimensions," IEEE Trans. Inform. Theory, vol. IT27, pp.207211, Mar. 1981. 3] J.A. Bucklew, \A note on optimal multidimensional companders," IEEE Trans. Inform. Theory, vol. IT29, p. 279, Mar. 1983. 4] J.H. Conway and N.J.A. Sloane, Sphere Packings, Lattices, and Groups, 2nd ed. New York: SpringerVerlag, 1993. 5] A. Gersho, \Asymptotically optimal block quantization," IEEE Trans. Inform. Theory, vol. IT25, pp. 373380, July 1979. 6] H. Gish and J.N. Pierce, \Asymptotically e cient quantizing," IEEE Trans. Inform. Theory, vol. IT14, pp. 676683, Sept. 1966.
64 7] R.M. Gray and A.H. Gray, Jr., \Asymptotically optimal quantizers," IEEE Trans. Inform. Theory, vol. IT23, pp. 143144, Jan. 1977. 8] R.A. Horn and C.R. Johnson, Matrix Analysis. Melbourne: Cambridge Press, 1985. 9] T.D. Lookabaugh and R.M. Gray, \Highresolution quantization theory and the vector quantizer advantage," IEEE Trans. Inform. Theory, vol. 35, pp. 10201033, Sept. 1989. 10] S. Na and D.L. Neuho , \Bennett's integral for vector quantizers," IEEE Trans. Inform. Theory, vol. IT41, pp. 886900, July 1995. 11] P. Panter and W. Dite, \Quantization in pulsecount modulation with nonuniform spacing of levels," Proc. IRE, vol. 39, pp. 4448, Jan. 1951. 12] J.L. Troutman, Variational Calculus and Optimal Control: Optimization with Elementary Convexity, 2nd Ed. New York: SpringerVerlag, 1996. 13] P.L. Zador, \Topics in the asymptotic quantization of continuous random variables," Bell Laboratories Technical Memorandum, 1966. 14] R. Zamir and M. Feder, \On lattice quantization noise," IEEE Trans. Inform. Theory, vol. 42, pp. 11521159, July 1996.
Y
compressor function f: k k
lattice quantizer
expander function 1 f
^ X
CHAPTER IV
I. Introduction
In polar quantization (PQ), a twodimensional random vector is quantized in terms of its magnitude and phase. This is advantageous for many sources, especially those with spherically symmetric densities. Several di erent polar quantization methods have been proposed 1] 11] optimality conditions and numerical methods of determining and optimizing performance have been developed 1, 2, 3, 12, 13, 4, 6, 7, 8, 9, 11, 14] and high resolution analyses have been developed that determine and optimize performance when distortion is small and rate is large 12, 13, 8, 15, 16, 17, 10, 14]. Such analyses are, by and large, individually tailored to the speci c polar method being analyzed. In this paper, we show that for several polar schemes, high resolution analyses can be performed in a uni ed manner, by using Bennett's integral for vector quantizers (VQ). We also analyze and optimize polar quantization with a uniform magnitude quantizer, using recent high resolution results on the optimal step size and support of uniform scalar quantization for sources
This work was supported by NSF Grant NCR9415754 and is to be submitted for publication in the IEEE Transactions on Information Theory, with coauthor David L. Neuho . Part of this work was presented at the 1997 IEEE International Symposium on Information Theory in Ulm.
1
65
66 with unbounded support. Previously, such uniform polar quantization had been analyzed only with numerical methods and with high resolution methods that assumed the support of the magnitude quantizer was independent of the encoding rate. Let the source vector X = (X1 X2) be zero mean with joint density pX (x) and q 2 variance 2 = 1 E kX k2 < 1. The magnitude of X is given by M = X1 + X22 2 and the phase is given by = tan;1(X2=X1 ). The random vector is spherically are independent,
symmetric if pX (x) depends only on kxk. Equivalently, M and
the resulting twodimensional quantization cells are not Voronoi regions, as in shapegain VQ 9]. Let NM and N denote the numbers of levels in the magnitude and phase quantizers, respectively. The step size of the uniform phase quantizer is = 2 =N and the total number of quantization cells is N = NM N . The magnitude and phase rates are given by RM = log2 NM and R = log2 N , respectively. The overall rate is
1 R = 2 log2 N = 1 (RM + R ). Optimizing restricted PQ to minimize distortion for a 2
given overall rate R involves nding the best rate allocation fRM R g (equivalently, c c c the best level allocation fNM N g) and the best magnitude levels m1 m2 ::: mNM . In unrestricted polar quantization, the phase is quantized after the magnitude, and the phase quantizer varies with the quantized magnitude. If there are NM
Uniform phase quantization has been shown to be optimal for spherically symmetric densities 2, 4, 13].
2
67
c c c magnitude levels, denoted m1 m2 ::: mNM , then there are integers N 1 ::: N NM c such that if the magnitude level is mi, then the phase is quantized with a uniform
scalar quantizer with step size quantization cells is N = N 1 +
i
In nonuniform polar quantization, the magnitude is quantized with an arbitrary scalar quantizer (SQ), whereas in uniform polar quantization a uniform scalar quantizer is used for the magnitude. Figure 4.1 shows examples of various polar quantizers. Optimizing uniform PQ involves nding the best rate allocation between the magnitude and phase quantizers and the best support (equivalently, step size) of the magnitude quantizer. Like most prior analysis, we use meansquared error (MSE) as the distortion measure. For restricted nonuniform polar quantization, Gallagher 2], Pearlman 4], and Bucklew and Gallagher 12, 13] found LloydMaxtype necessary conditions for optimality and numerically optimized nonuniform and uniform PQ, by solving for the optimal rate allocation between the magnitude and phase quantizers. Pearlman 1] and Bucklew and Gallagher 12, 13] (for asymptotically large rate) found analytically that the di erence between the optimal phase and magnitude rates is a constant. Bucklew and Gallagher also found asymptotic expressions for the minimum MSE of restricted nonuniform PQ, which is less than that of Cartesian quantization.3 Unrestricted nonuniform polar quantization was rst proposed by Wilson 6], who derived LloydMaxtype necessary conditions for optimality. Subsequently, Swaszek and Ku 17] performed an asymptotic analysis and optimization that showed that optimized unrestricted nonuniform PQ comes close to the distortion of asymptoti3
X1
and
X2
as Cartesian quantization.
68 cally optimal twodimensional vector quantization. In comparison, in this work, we present a uni ed analysis of several nonuniform polar quantization methods, by focusing on their point densities and inertial pro les, and using Bennett's integral for vector quantizers 18] to express the meansquared error. This makes the analysis straightforward, and leads to new insights. With this approach, unrestricted polar quantization, which is arguably the best polar method, may be optimized essentially by inspection. As another example, a new polar quantization method, called power law polar, is analyzed and optimized. Restricted uniform polar quantization was optimized numerically by Gallagher 2] and Pearlman 4] and analytically by Swaszek 15]. However, the analysis in 15] assumed a xed support region for the magnitude quantizer, when in fact for true optimality the support needs to increase with rate. In this paper, we optimize restricted uniform PQ asymptotically by using a recent result from Hui and Neuho 19] on the support of uniform scalar quantization to choose the optimal support of the magnitude quantizer. We then nd the optimal rate allocation between magnitude and phase and the resulting meansquared error. Two styles of argument are used in highresolution quantization theory: informal, as in 20], 21] 17], and rigorous, as in 22], 18], 19]. In this paper, we use primarily the informal style. However, uniform polar quantization requires more care than its nonuniform counterpart because overload distortion is not necessarily negligible. Because of this, we provide some rigorous results in our analysis of uniform PQ. The rest of this paper is organized as follows. In Section II several unrestricted nonuniform PQ methods are analyzed using Bennett's integral. Section III considers restricted uniform polar quantization and optimizes this technique for an IID Gaussian source. Various details are presented in Section IV.
69
(m), which gives the number of phase levels when the magnitude is and (m) are nonnegative and satisfy (4.1) (4.2) (4.3) = 1 (4.4) 1 c = N l(S ) when m = mi M i
p m (m) = N i when m = ci N
Z1
0
M (m) dm
= 1
Z1
0
(m)
M (m) dm
where l(Si) is the length of the ith magnitude cell Si. It can be shown, at least approximately, that (4.1) implies (4.3), and that (4.1), (4.2) and the fact that N =
N 1+
+N
NM
imply (4.4).
70 Power law PQ is of interest because it considers a conceptually simple class of phase level pro les. It may also be of interest when quantizing a complexvalued random variable in situations, such as for synthetic aperture radar, where more phase accuracy is required for larger magnitudes than small. When = 0, power law PQ is equivalent to restricted nonuniform polar quantization. When = 1, all cells of the resulting PQ have approximately the same radial width. We refer to this case as constant width polar quantization. When 0 < < 1, the cells have radial widths increasing as m1; .
N is large:
1 1 Z x c D = 2 E kX ; X k2 = N m((x)) p(x) dx
where (x) is the point density of the polar quantizer and m(x) is the inertial pro le, giving the normalized moment of inertia (NMI) of the cells near x. The NMI of a R cell T0 with a codevector y is given by M (T0) = k;1v(T0);(1+2=k) T0 kx ; yk2 dx. We proceed by expressing the point density and inertial pro le of polar quantization as functions of
M (m), M ,
and
H = N 1 (kxk) = M M
and width
p 1
N
M (kxk)
W =
2 kxk : p (kxk) N
71 Therefore the point density and inertial pro le of polar quantization are given by 1 M (x) = NHW = M (kxk2) kxk (kxk) 2 1 2 m(x) = 12 (Hp +2W )=2 : H W2 Substituting (x) and m(x) into Bennett's integral yields the following asymptotic expression for the MSE of unrestricted nonuniform polar quantization, ! 1 4 2kxk2 p(x) dx: 1 1 Z + (kxk)2 (4.5) D = 24 N 2 2 M M (kxk) This expression was rst derived by Swaszek and Ku 17] using a longer argument. We see that the Bennett integral approach leads to a straightforward derivation of (4.5). Note that because the phase quantizer is uniform, the asymptotic MSE of polar quantization depends only on the magnitude density and not on the phase density that is, not on any assumption of spherical symmetry. However, if the source vector is not spherically symmetric, then it is possible to exploit the symmetry to do better than polar quantization. Because asymptotic distortion does not depend on the phase density, one could assume without loss of asymptotic generality that the phase is uniform and independent of the magnitude. We make this assumption later in the paper.
Optimization of Power Law Polar Quantization Proposition 1 For power law polar quantization, the magnitude point density and
magnitude allocation that minimize (4.5) subject to (4.3) and (4.4) are given by pM (m)1=3 M (m) = c =3
where c makes 1 M = p 2
M (m)
EM 2;2
Z1
0
m2 =3 pM (m)1=3 dm
;1=4
Z1
0
m; =3 pM (m)1=3 dm:
72
The resulting minimum MSE is
; D=N
where
1= 2
; =
2;2 6 EM
Z1
0
m2 =3 pM (m)1=3 dm
3=2
This yields
= p1 2
EM
2;2
;1=4
Z1
0
M (m) dm
;1=2
D( M ) = 6N EM
omitted. Note that when
2;2
1=2
M (m).
form polar quantization 13]. When = 1 (constant width PQ), the resulting twodimensional point density and MSE are identical to those for Cartesian quantization. In 18], it was shown that the loss of Cartesian quantization relative to optimal VQ of the same dimension can be decomposed into the product of point density and cell shape losses. The former isolates the loss caused by the suboptimal point density of Cartesian quantization, while the latter isolates the e ect of suboptimal inertial pro le. Since when optimized constant width PQ and Cartesian quantization have the same point density and the same MSE, it follows that they incur the same point density loss and the same total loss. Since total loss is the product of point density and cell shape losses, we see that constant width PQ and Cartesian quantization also
73 have the same cell shape loss, even though their inertial pro les are quite di erent: constant width PQ has a spherically symmetric inertial pro le, whereas Cartesian quantization does not. For the important special case of IID Gaussian random variables,
M
and ; are
given in Table 4.1 for various values of . We do not have a method for optimizing power law PQ over the choice of , but have found by trialanderror that the best choice of for a Gaussian source is = 0:395. In this case, the resulting signaltonoise ratio (SNR) is 0.62 dB less than that of optimal twodimensional vector quantization 22] and 0.45 dB less than that of optimized unrestricted PQ. We see that the phase is allocated more rate than magnitude for all cases of power law PQ. Note also that
M
also gives the loss of power law PQ relative to optimal twodimensional VQ. This loss, expressed in dB, is then decomposed into the sum of square loss Lsq , oblongitis loss Lob and point density loss Lpt 18]. The square loss is the portion of cell shape loss due to the fact that the cells are at best squares rather than hexagons. The oblongitis loss, which is the remaining portion of the cell shape loss, is due to the fact that the cells are rectangles rather than squares.
74
L2vq
Lsq
Lob
Lpt
0.6133 2.4748 0.89 0.17 0.39 0.33 0.6227 2.4066 0.77 0.17 0.33 0.27 0.6312 2.3595 0.69 0.17 0.29 0.23 0.6388 2.3320 0.63 0.17 0.26 0.21 0.6453 2.3235 0.62 0.17 0.25 0.20 0.6508 2.3339 0.64 0.17 0.26 0.21 0.6551 2.3639 0.69 0.17 0.29 0.24 0.6580 2.4147 0.79 0.17 0.34 0.28 0.6595 2.4886 0.92 0.17 0.41 0.34 0.6593 2.5890 1.09 0.17 0.50 0.42 0.6573 2.7207 1.30 0.17 0.62 0.51 0.707 2.0944 0.17 0.17 0.00 0.00 2.7207 1.30 0.17 0.62 0.51 2.0157 0.00 0.00 0.00 0.00
(optimal PL)
0.395 0.6450 2.3235 0.62 0.17 0.25 0.20 0.4 0.5 0.6 0.7 0.8 0.9
(constant width)
1.0
Table 4.1: Comparison of magnitude allocation M and MSE D = ;=N for a pair of IID Gaussian random variables. L2vq is loss relative to optimal xedrate twodimensional VQ and is decomposed as the sum of square loss Lsq , oblongitis loss Lob and point density loss Lpt.
75 Asymptotically, its cells are rectangles, so the best they could be is squares, in which case m(x) = 1=12 for all x. In order to optimize unrestricted PQ, we would like, if possible, to choose
M (m), M
and
is, H = W ) and (x) = (x). Given the rectangular shape of the quantization cells, this is the best that one could do. In order to achieve H = W , it must be that (x) = 1 = NH 2
M M (kxk)
2 2
c0
pM (m) m
!1=4
yields unrestricted PQ with square cells and the optimal point density. The resulting best magnitude allocation and phase level pro le are given by Z1 ;1=2 Z 1 = 2 m1=2 pM (m)1=2 dm m;1=4 pM (m)1=4 dm M 0 0 Z1 ;1=2 p 3=4 1=4 (m) = 2 m pM (m) m1=2 pM (m)1=2 dm 0 and the resulting minimum MSE is ; D(N ) = N where 1 Z 1 (2 m p (m))1=2 dm 2 : ; = 12 M 0 Compared to optimal twodimensional VQ, optimized unrestricted PQ su ers only a 0.17 dB loss, which is the ratio of the NMI of a cube to that of a hexagon,
76 expressed in dB. In contrast to the optimization in 17], we have optimized unrestricted PQ very simply by inspection of its point density and cell shape. Moreover, our analysis provides the insight that the only shortcoming of unrestricted PQ is its inability to form hexagonal cells, which completely accounts for the 0.17 dB loss.
too slowly with NM , then overload distortion dominates. On the other hand, if the support increases too rapidly, then granular distortion dominates. It is not known a priori how rapidly the support should increase therefore, Bennett's integral is not a reliable asymptotic expression for total distortion, because it accounts only for granular distortion. Thus we use other methods to derive an asymptotic expression for MSE.
77
where
c DM = EM (M ; M )2]
D = 2E 1 ; cos( ; b )] c IM = 1 EM M M ] 2
c and M and b are the quantizer outputs of the magnitude and phase quantizers,
respectively.
lim N 2 D N !1
tortion of the magnitude quantizer, but is not always asymptotically accurate for total distortion. In general, asymptotic expressions for DM depend on the source density and on the dependence of
M
on NM .
M
tizer for a given number of magnitude levels NM and then nding the best magnitude allocation. To optimize the step size, we see from Lemma 4 that IM has an asymptotically negligible e ect on MSE. Therefore, we choose the step size to minimize
DM (
of
M
is source dependent, and we now consider the most important special case.
78
Gaussian Sources
Consider an IID Gaussian source, so that the magnitude is Rayleigh. Hui and Neuho 19] found asymptotic expressions for the optimal step size lemma.
M
and the
lim q
that the overload distortion is negligible. The asymptotic results given above can be used to derive the following proposition, which shows that MSE is approximately
2 2 2 1 f D(NM N ) def 6 ln NM + 3 N 2 : = 2 NM
(4.8)
M
NM N ) of a
uniform polar quantizer with NM magnitude levels and N phase levels satis es
Using the above and replacing NM and N with rates RM and R , respectively, we nd that with the asymptotically optimal magnitude step size, the distortion of uniform PQ with rate allocation (RM R ) is ;f (RM ) + oRM (1) ;g(R ) + oRM (1) + oR (1) D(RM R ) = 2 +2 (4.9)
79 where f (r) = 2r ; log2 r ; log2( 2 ln 2=6) and g(r) = 2r ; log2( to in nity. We want to nd the rates RM and R that minimize D(RM R ) subject to the
1 constraint 2 (RM + R ) = R. The following lemma shows that for large R, RM and 2 2
oRM (1) and oR (1) denote quantities that tend to zero as RM , respectively R , tend
;f (RM )
+2
;g(R )
(4.10)
Lemma 7 Let
DR(r) def 2;f (r)+or + 2;g(2R;r)+or +o2R;r = DR(r) def 2;f (r) + 2;g(2R;r) =
and let rR and rR minimize DR (r) and DR (r), respectively, for a given value of R. Then
= 0 = 1:
(4.11) (4.12)
The rates that minimize D(RM R ) can then be derived using Lagrange multipliers. By the previous lemma these rates asymptotically minimize D(RM R ) as well. The result is summarized in the following.
Proposition 8 Let the source be IID Gaussian. The rates that minimize the MSE
1 D(RM R ) of uniform polar quantization subject to 2 (RM + R ) = R are asymptot
ically given by
(4.13)
80
and R = 2R ; RM , where for x < 0, W (x) is the more negative of the two values
y such that yey = x 23]. The resulting magnitude quantizer support is q LM = 2 ; W (;8 2 e2 2;4R)
and the resulting MSE is
2 2 2 D (R) = 12e eW (;8 2 e22;4R)=2 + 2 3 e 2;4R e;W (;8 2 e2 2;4R)=2:
We now compare the result of Proposition 8 to that of Pearlman's numerical optimization 4] and Swaszek's analysis 15]. The optimization in 4] was limited to values of N for which there was a good factorization N = NM N . Figure 4.3 shows that all three analyses allocate more rate to the phase quantizer than to the magnitude quantizer. Although R overestimates Pearlman's phase rate from rates 3 to 4, it is a good estimate for rates 4 to 5. Figure 4.4 shows that LM provides a good estimate of the increasing optimal support of restricted uniform PQ, especially compared to Swaszek's analysis, which assumed that the support is xed for all rates. As shown in Figure 4.5, our expression for SNR slightly overestimates Pearlman's results. This is because our analytic formula ignores overload distortion. We note that the above analysis can be repeated for other magnitude distributions. We also note that the more accurate expressions in 19] for the support and distortion of optimal uniform scalar quantizers could be used to improve the approximations given here.
Approximations to W
Although the above result gives a solution to the rate allocation problem for uniform PQ, because of the W function, it is di cult to immediately see the behavior of the magnitude and phase rates and the resulting minimum MSE. For this reason,
81 we consider a series expansion for W (x). As rate R increases, the argument of W tends to zero, and we will be able to approximate W with only a few terms from the series. It is shown in 23, Eqn. 4.18] that
L2 L1
(4.14)
when x < 0 and L1 = ln(;x) and L2 = ln(; ln(;x)). We will consider the twoterm
which are good approximations for W (x) when x is small and negative. We denote R
2
From these estimates, one sees that for large values of rate R, the optimal phase rate R is less than R. In other words, magnitude is allocated more rate than phase, which never happened with nonuniform PQ. By numerically equating (4.13) to R, we nd that the magnitude and phase receive equal rate when R = 29. In addition, one sees from the above estimates that RM ; R ! 1 as R ! 1 and, nevertheless,
RM =R ! 1, or equivalently, RM =R ! 1=2.
82 Using W2(x) and W3(x), approximations to the optimal support of the magnitude quantizer are given by q 4 (ln 2) R ; ln(8 2) + ln 4 (ln 2) R ; ] LM 2 = v ! u u 1 t4 (ln 2) R ; ln(8 2) + ln 4 (ln 2) R ; ] 1 + LM 3 = 4 (ln 2) R ; : Figure 4.4 shows that L
3
Approximations to the SNR of optimized restricted uniform PQ are given by p ! 2 S2 (R) = 6:02R ; 10 log10 6 h i ;10 log10 (4(ln 2)R ; );1=2 + (4(ln 2)R ; )1=2 p ! 2 S3 (R) = 6:02R ; 10 log10 6 ! 2 1 1+ 1 ; 6 6 4(ln 2)R ; 6(4(ln 2)R ; ) 2 ;10 log10 6 6 4 !3 1 1 1+ 7 2 4(ln 2)R ; 7 7: 7 + (4(ln 2)R ; ) 7 5 Figure 4.5 shows that S3 (R) is an accurate approximation for S (R) at rates 4 to 5. These approximations show that the SNR of uniform PQ increases asymptotically as 6 dB/bit, but separates from any line with slope 6 dB/bit. Finally, Figure 4.6 compares the SNR of optimized restricted uniform PQ to that of restricted nonuniform PQ and Cartesian product uniform scalar quantization (USQ), as optimized in 19]. We see that the SNRs of all three methods increase asymptotically as 6 dB/bit. However the di erence in SNR between restricted nonuniform and restricted uniform tends to in nity as rate increases. This quanti es the loss in using a uniform scalar quantizer for the magnitude as opposed to using
83 an arbitrary scalar quantizer. Furthermore, the di erence in SNR between restricted uniform PQ and product USQ also tends to in nity as rate increases, which demonstrates the gain of using the polar transformation before quantization. The SNR of unrestricted nonuniform PQ is parallel to and 0.73 dB greater than that of restricted nonuniform PQ.
IV. Details
A. Proof of Lemma 2
When the source is spherically symmetric, the MSE of restricted polar quantization is given by,
D
= = = =
1 Z kx ; xk2p (x) dx b X 2 Z<2 Z 1 2 1 h(m sin ; c sin b)2 + (m cos ; m cos b)2i c m 2 0 0 p ( ) pM (m) dm d (4.15) 1 Z 2 Z 1 hm2 ; 2mm cos( ; b) + m2i p ( )p (m)dmd : c c M 2 Z0 Z0 1 2 1 (m ; c)2 p ( ) p (m) dm d M 2 0 0Z Z m 2 1 b c +1 2 0 0 2mm 1 ; cos( ; ) p ( ) pM (m) dm d (4.16) 1 E (M ; M )2] + 2E 1 ; cos( ; b )] 1 E M M ] c c }  {z } 2 M{z } 2  M {z DM D IM
where the independence of p and pM in (4.15) follows from the spherical symmetry assumption on the source. The integrals in the second term on the righthand side of (4.16) can be separated by the restricted assumption on the quantizer.
84
B. Proof of Lemma 3
We use an in nite series expansion for cosine, cos(x) =
1 X
j =0
(;1)j
2j
where the sum and integral can be exchanged because the integrand is absolutely integrable 24, p. 307]. It then follows that lim sup N 2 D ; 3 N !1
2
(;1)(j+1) 1 (2j + 1)! N 2(j;1) N !1 j =2 1 (j +1) 2j X lim sup 2 (;j1) 1)! 2(j;1) (2 + N !1 j =2 N 1 4 X 1 lim sup 2 2 N !1 N j =0 (2j + 1)! 4 1 = lim sup 2 2 1 e ; 2 N !1 N 2 = 0 = lim sup 2
2j
1 X
C. Proof of Lemma 4
We immediately see that
i 1 hc lim sup IM ; 1 EM 2 = lim sup 2 E (M ; M )M 2 D M !0 DM !0 1 lim sup E h (M ; M )M i c 2 DM !0 1 lim sup E h(M ; M )2i1=2 E hM 2i1=2 c 2
= 0
DM !0
D. Proof of Proposition 6
We use oN to denote a term that tends to zero as N tends to in nity. Lemmas 3, 4 and 5 show that 1 = 3 N 2 + oN IM = 1 EM 2 + oN 22 NM ) = 3 ln NM + oNM : 2 NM
(4.17)
DM (
It is easily seen that the second term goes to zero as NM ! 1 and N ! 1, which proves Proposition 6.
E. Proof of Lemma 7
To show jrR ; rRj as R ! 1, it will su ce to show jf (rR) ; f (rR)j ! 0 as
R ! 1. This follows because rR ! 1 and rR ! 1 and because for r large enough f 0(r) 1, which implies that jf (rR) ; f (rR)j jrR ; rRj.
To show jf (rR) ; f (rR)j ! 0 as R ! 1, we will show
0 jf (rR) ; f (rR)j ! 0
as R ! 1
(4.18)
and
0 jf (rR) ; f (rR)j ! 0
as R ! 1
(4.19)
86
0 where rR is de ned as the rate such that 0 0 f (rR) = g(2R ; rR)
that is, the rate such that the exponents in DR(r) are identical (cf. 26, 27]). We will show (4.18) by contradiction, namely if (4.18) does not hold, then for large
0 enough R, D(rR ) > D(rR ), which contradicts the optimality of rR. Accordingly, we
assume that there exists > 0 and a sequence of rates increasing to in nity, denoted
f (r0 ) ; f (r ) >
for all ,
(4.21)
0 where r = rR and r0 = rR . We will show that there is some > 0 such that
D (r ) D (r 0 ) > 1 +
which establishes the contradiction. It is easily seen that as ! 1,
(4.22)
r ! 1 and 2R ; r ! 1
for otherwise D(r ) would not tend to zero. Also,
(4.23)
r0 ! 1 and 2R ; r0 ! 1:
(4.24)
This follows because either r0 or 2R ; r0 must increase to in nity as R ! 1. The fact that f (r) and g(r) increase to in nity as r ! 1 implies that either f (r0 ) ! 1 or g(2R ; r0 ) ! 1. However, by de nition, f (r0 ) = g(2R ; r0 ). Therefore,
87
f (r0 ) ! 1 and g(2R ; r0 ) ! 1, which implies (4.24). Since there exists r0 such
that f (r) is strictly increasing for all r > r0, we may assume that r , 2R ; r , r0 and 2R ; r0 are all greater than or equal to r0. Before proceeding with the proof, we establish a general expression for the ratio of D(r ) to D(r0 ),
D(r ) 2;f (r )+or + 2;g(2R ;r )+or +o2R ;r D(r0 ) = 2;f (r0 )+or0 + 2;g(2R ;r0 )+or0 +o2R ;r0 2;f (r ) + 2;g(2R ;r ) 2;f (r )+or + 2;g(2R ;r )+or +o2R ;r = ;f (r0 ) ;g(2R ;r0 ) 2 +2 2;f (r ) + {z;g(2R ;r ) 2  } 1+o ;f (r0 ) + 2;g(2R ;r0 ) 2 0 ;f (r0 )+or0 + 2;g(2R ;r )+or0 +o2R ;r0} 2 {z 1+o 0 )+f (r 0 );f (r ) 2;f (r + 2;g(2R ;r0 )+g(2R ;r0 );g(2R ;r ) (1 + o ) = 2;f (r0 ) + 2;g(2R ;r0 ) 1 (4.25) = 2 2f (r0 );f (r ) + 2g(2R ;r0 );g(2R ;r ) (1 + o )
where the 1 + o terms follow from (4.23) and (4.24). Returning to the proof, we begin by assuming that (4.20) holds. The assumption
f (r ) > f (r0 ), together with the monotonicity of f , imply that r > r0 > r0, and it
can be easily seen that
f (r
) ; f (r 0 )
; r0 ) ; log
D(r ) 1 = 2 2f (r0 );f (r ) + 2g(2R ;r0 );g(2R ;r ) (1 + o ) D(r0 ) 1 2f (r0 );f (r ) + 2f (r );f (r0 ) (1 + o ) 2 1 > 2 2; + 2 + o
(4.26)
88 where (4.26) follows from Lemma 9, given later, and the assumption (4.20) that
; = 2 +42 ; 2
which establishes (4.22). Next assume that (4.21) holds. In this case, it is easily shown that
g(2R
; r0 ) ; g(2R
; r ) = f (r
) ; f (r0 ) ; log
! r0 ; r : 1+ r
D(r ) 1 f (r0 );f (r ) + 2g(2R ;r0 );g(2R ;r ) (1 + o ) 0) = 2 2 D(r = 1 2f (r0 );f (r ) + 2f (r );f (r0 ) 2

Note that
{z J( )
(1 + o ):
(4.27)
89 = 1+
r r0 ; 1 0 );2f (r ) 22f (r +1
(4.28) (4.29)
! 1
where (4.29) follows because the denominator in (4.28) is bounded for large enough and r =r0 ! 1, which can be easily shown. Returning to (4.27), we have D(r ) 1 f (r0 );f (r ) + 2f (r );f (r0 ) J ( ) (1 + o ) 0) = 2 2 D(r = 1 2f (r0 );f (r ) + 2f (r );f (r0 ) (1 + o ) 2 > 1 2; + 2 + o 2 Again, let be large enough so that
; jo j < 2 +42 ; 2
(4.30)
where (4.30) follows from Lemma 9 and the assumption (4.21) that f (r0 ) ; f (r ) > .
where Lemma 9 implies that 2; + 2 ; 2 > 0. Then for large enough, D(r ) 1 ; 0) > 2 2 +2 +o D(r 1 2; + 2 ; jo j 2 ; + ; > (2 2 2 ) ; 2 +42 ; 2 = 1+ which establishes (4.22) and completes the proof of (4.18). Since D(r) equals D(r) modulo the o terms, this implies that (4.19) holds as well, which establishes (4.11) in Lemma 7. It then follows immediately that (4.12) in Lemma 7 holds, as well.
Lemma 9 If 0 a b, then
2;a + 2a
with equality if and only if a = b.
2;b + 2b
90
Proof: Note that 2a+b 1 with equality if and only if a = b = 0. We then have 2b ; 2a b ; 2a 2
2a+b with equality if and only if a = b, and simple manipulation gives the result. 2
Acknowledgements
The authors would like to thank Susan Werness for spurring their interest in polar quantization and Dennis Hui for assistance with Table 4.1.
References
1] W.A. Pearlman, \Quantization error bounds for computer generated holograms," Stanford. Univ. Inform. Syst. Lab., Stanford, CA, Tech Rep. 65031, Aug. 1974. 2] N.C. Gallagher, Jr., \Quantizing schemes for the discrete Fourier transform of a random timeseries," IEEE Trans. Inform. Theory, vol. IT24, pp. 156162, Mar. 1978. 3] W.A. Pearlman and R.M. Gray, \Source coding of the discrete Fourier transform," IEEE Trans. Inform. Theory, vol. IT24, pp. 683692, Nov. 1978. 4] W.A. Pearlman, \Polar Quantization of a complex random variable," IEEE Trans. Comm., vol. COM27, pp. 892899, June 1979. 5] A. Buzo, A.H. Gray, Jr., R.M. Gray, and J.D. Markel, \Speech coding based upon vector quantization," IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP28, no. 5, Oct. 1980. 6] S.G. Wilson, \Magnitude/phase quantization of independent Gaussian variates," IEEE Trans. Commun., vol. COM28, pp. 19241929, Nov. 1980. 7] P.F. Swaszek and J.B. Thomas, \Optimal circularly symmetric quantizers," J. Franklin Institute, 313, pp. 373384, June 1982. 8] P.F. Swaszek and J.B. Thomas, \Multidimensional spherical coordinates quantization," IEEE Trans. Inform. Theory, vol. IT29, pp. 570576, July 1983. 9] M.J. Sabin and R.M Gray, \Product code vector quantizers for waveform and voice coding," IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP32, no. 3, June 1984. 10] T.R. Fischer, \A pyramid vector quantizer," IEEE Trans. Inform. Theory, vol. IT32, pp. 568583, July 1986.
91 11] J.P. Adoul and M. Barth, \Nearest neighbor algorithm for spherical codes from the Leech lattice," IEEE Trans. Inform. Theory, vol. 34, pp. 11881202, Sept. 1988. 12] J.A. Bucklew and N.C. Gallagher, Jr., \Quantization schemes for bivariate Gaussian random variables," IEEE Trans. Inform. Theory, vol. IT25, pp. 537543, Sept. 1979. 13] J.A. Bucklew and N.C. Gallagher, Jr., \Twodimensional quantization of bivariate circularly symmetric densities," IEEE Trans. Inform. Theory, vol. IT25, pp. 667671, Nov. 1979. 14] P.F. Swaszek, \A vector quantizer for the Laplace source," IEEE Trans. Inform. Theory, vol. 37, pp. 13551365, Sept. 1991. 15] P.F. Swaszek, \Uniform spherical coordinate quantization of spherically symmetric sources," IEEE Trans. Commun., vol. COM33, pp. 518521, June 1985. 16] P.F. Swaszek, \Asymptotic performance of Dirichlet rotated polar quantizers," IEEE Trans. Inform. Theory, vol. IT31, pp. 537540, July 1985. 17] P.F. Swaszek and T. Ku, \Asymptotic performance of unrestricted polar quantizers," IEEE Trans. Inform. Theory, vol. IT32, pp. 330333, Mar. 1986. 18] S. Na and D.L. Neuho , \Bennett's integral for vector quantizers," IEEE Trans. Inform. Theory, vol. IT41, pp. 886900, July 1995. 19] D. Hui and D.L. Neuho , \Asymptotic analysis of optimal xedrate uniform scalar quantization," submitted to IEEE Trans. Inform. Theory, August 1997. 20] P. Panter and W. Dite, \Quantization in pulsecount modulation with nonuniform spacing of levels," Proc. IRE, vol. 39, pp. 4448, Jan. 1951. 21] A. Gersho, \Asymptotically optimal block quantization," IEEE Trans. Inform. Theory, vol. IT25, pp. 373380, July 1979. 22] P.L. Zador, \Topics in the asymptotic quantization of continuous random variables," Bell Laboratories Technical Memorandum, 1966. 23] R.M. Corless, G.H. Gonnet, D.E.G. Hare, D.J. Je rey, and D.E. Knuth, \On the Lambert W Function", Advances in Computational Mathematics, vol. 5, pp. 329{359, 1996. 24] A.N. Komolgorov and S.V. Fomin, Introductory Real Analysis. Dover, New York, 1970. 25] I.S. Gradshteyn and I.M. Ryzhik, Tables of Integrals, Series, and Products. Academic Press, New York, 1994. 26] C.E. Shannon, \Certain results in coding theory for noisy channels," Inform. Contr., vol. 1, pp. 625, 1957. 27] K. Zeger and V. Manzella, \Asymptotic bounds on optimal noisy channel quantization via random coding," IEEE Trans. Inform. Theory, vol. 40, pp. 19261938, Nov. 1994.
92
(a)
(b)
(c)
Figure 4.1: Examples of polar quantizers: (a) restricted nonuniform, (b) restricted uniform, (c) unrestricted nonuniform.
H W
93
6
5.5
Phase Rate
4.5
3.5
Figure 4.3: Optimal phase rates for restricted uniform polar quantization. The optimal rate from Proposition 6 is R . The twoterm and threeterm approximations are R 2 and R 3, respectively.
4.5
3.5
2.5
1.5
3.2
3.4
3.6
3.8
4 Rate
4.2
4.4
4.6
4.8
Figure 4.4: Optimal magnitude quantizer support for restricted uniform polar quantization. The optimal support from Proposition 6 is LM . The twoterm and threeterm approximations are LM 2 and LM 3, respectively.
94
26
24
22
SNR
20
18
16
14
12
3.2
3.4
3.6
3.8
4 Rate
4.2
4.4
4.6
4.8
Figure 4.5: Maximum signaltonoise ratio (SNR) for restricted uniform polar quantization. The optimal rate from Proposition 6 is S . The twoterm and threeterm approximations are S2 and S3 , respectively.
60
55
50
45
SNR
40
35
30
25
20
5.5
6.5
7.5 Rate
8.5
9.5
10
Figure 4.6: Signaltonoise ratio (SNR) of restricted nonuniform polar quantization, restricted uniform polar quantization and uniform scalar quantization.
CHAPTER V
Conclusions
In this nal chapter, we summarize the contributions of this dissertation and point to future research issues in the area.
Summary of Contributions
In Chapter II, we presented analytical formulas for the optimal scaling factor aN and the resulting minimum distortion DN when the source is generalized Gaussian with decay parameter 1. Upper and lower bounds to distortion were derived, and it was then shown that the ratio of scaling factors that minimize each bound equals one asymptotically and that the ratio of each scaling factor to aN equals one asymptotically. It was also shown that for scaleoptimized lattice quantization, granular distortion asymptotically dominates overload distortion, which leads directly to the asymptotic expression for DN . In Chapter III, multidimensional companding for memoryless sources was optimized asymptotically, under certain technical conditions on the compressor function. The proof of the result was based on variational calculus arguments. It was then seen that optimized multidimensional companding incurs the same point density and oblongitis losses as optimized scalar quantization, while recovering the cubic loss. 95
96 In Chapter IV, we considered uniform polar quantization and found the optimal rate allocation between magnitude and phase. It was shown that the magnitude receives increasingly more rate as total rate increases. The optimal rates were shown to depend on W , an inverse function without a closed form solution. We showed that a simple approximation to W is very accurate for total rates greater than 4. We also analyzed and optimized several nonuniform polar quantization methods by using Bennett's integral.
97 In polar quantization, it would be interesting to extend the uniform polar results in Chapter IV to dimensions greater than two. What is the best latticebased polar quantization method in three dimensions? For the twodimensional case, optimizing unrestricted uniform polar quantization is still an open problem. Such an optimization would give further insight into the tradeo s between restricted and unrestricted polar quantization and between nonuniform and uniform polar quantization.
ABSTRACT
ASYMPTOTIC ANALYSIS OF LATTICEBASED QUANTIZATION by Peter Warren Moo
Chair: David L. Neuho Latticebased quantization is an attractive method of vector quantization, because of its potentially low complexity and its ability to form nearly spherical cells. In this dissertation, three latticebased quantization methods are optimized using high resolution analysis. In the rst part of this dissertation, xedrate lattice quantization for a class of generalized Gaussian sources is considered. Asymptotic expressions for the optimal scaling factor and resulting minimum distortion are presented. These expressions are derived by minimizing upper and lower bounds to distortion. It is also shown that for scaleoptimized lattice quantization, granular distortion asymptotically dominates overload distortion, and the ratio of optimal lattice quantizer distortion to optimal vector quantizer distortion is shown to increase without bound as rate increases. In the second part, multidimensional companding, which utilizes a nonlinear compressor function and a lattice quantizer, is optimized for memoryless sources. Under certain technical assumptions on the compressor function, it is shown that the best
compressor function consists of the best scalar compressor functions for each component of the source vector. The gain of optimal companding compared to scalar quantization is shown to be the gain in normalized moment of inertia of the lattice cell compared to a cube. The third part considers uniform polar quantization, which utilizes a polar transformation and a two dimensional integer lattice quantizer. It is shown that the optimal rate allocation between magnitude and phase gives increasingly more rate to the magnitude as total rate increases, compared to nonuniform polar, where the di erence between optimal magnitude and phase rates is a constant. In addition, a uni ed analysis of several nonuniform polar quantization schemes is developed by focusing on their point densities and inertial pro les and using Bennett's integral to express the meansquared error. The subsequent analysis is straightforward and leads to new insights into the relationship between polar quantization and Cartesian quantization.