If we wish to encode a gaussian source, $X\sim\mathcal{N}(0,\sigma^2)$ at rate $R$, then decode it to create an estimate $\hat{X}$, rate-distortion theory tells us that the lowest mean-squared-error we can achieve through the encoder-decoder is $\sigma^2 2^{-2R}.$ That is, using an optimal encoder and decoder, we can achieve $E(X-\hat{X})^2=\sigma^2 2^{-2R}.$ (possibly asymptotically as we let code block lengths increase)

What might the distribution of $\hat{X}|_{X=x}$ be, for such encoders and decoders? That is, how is our estimate distributed around the right answer?

For illustration, we know obviously that no choice of enc/dec could yield a deterministic $\hat{X}|_{X=x}=x+\sqrt{\sigma^22^{-2R}},$ even though this distribution would give us the correct mean-squared-error. (Otherwise we could design a lossless decoder!)

I believe that it should be possible to design some random coding scheme where, as the code block lengths get large, eventually each $i$th message in our block, $\hat{X_i}|_{X_i=x_i}$, becomes close in distribution to $\mathcal{N}(x_i,\sigma^22^{-2R}).$

Equivalently, I am trying to find a sequence of encoders and decoders where:

$$D(p^n||q^n)\rightarrow 0, \text{ as }n \uparrow \infty$$

where $x^n$ denotes a block of $n$ messages, $\hat{X}^n$ is the decoded estimate of all these $n$ messages, $p^n$ is the distribution of ${\hat{X}^n|_ {X^n=x^n}}$ and $q^n$ is the distribution of $\mathcal{N}(x^n,\sigma^22^{-2R}I_{n\times n})$.

Finding such an encoder/decoder pair would essentially allow us to properly say that quantization of a gaussian source can be modeled as gaussian noise.


1 Answer 1


Yes, you can design a countable-alphabet quantizer for Gaussian RVs where quantization noise approaches Gaussian noise in relative entropy, as codeword length increases.

'On Lattice Quantization Noise' by Zamir and Feder in Transactions on Information Theory Vol. 42 No. 4, July 1996:


Maybe a better question would be whether any sequence of bound-achieving quantizers approach the test channel distribution in relative entropy. (I think if they don't you can get a contradiction but am not engaged enough to think this through).

  • $\begingroup$ This is expected, because the test channel that minimizes rate in the construction of $R(D)$ for a Gaussian source $X$ is: $$X \longrightarrow \otimes \longrightarrow \!\!\!\! \underset{\underset{N\sim \mathcal{N}(0,1)}{\big\uparrow}}{\oplus} \!\!\!\! \longrightarrow \otimes \longrightarrow \alpha \cdot (\alpha X+\beta N),$$ with $N\perp X, \ \alpha = \sqrt{1-D/\sigma_X^2}$ and $\beta=\sqrt{D/\sigma_X^2}.$ $\endgroup$ Feb 16, 2016 at 19:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.