Tagged Questions

Covers theoretical and experimental aspects of information theory and coding.

39 views

Last Inference in proof of conditional limit theorem

I read about the Conditional Limit Theorem from the book "Elements of Information Theory" by Thomas M. Cover and Joy A. Thomas, second edition, page 371. I can't understand the last inference in the ...
81 views

Generalisations of the Kullback-Leibler divergence for more than two distributions

A very fundamental quantity in information theory is the Kullback-Leibler divergence between two probability distributions over the same random variable, $D_{KL}(Q\|P) = \sum_i q_i \log\frac{q_i}{p_i}$...
67 views

Characterizing the optimimum over the space of probability measures

Consider the following optimization problem: $$\max_{\mu \in \mathcal{M}} \int \log\left( \int e^{\alpha U(x,y)} d\mu(y) \right) d\nu(x)$$ where $\mathcal{M}$ is the space ...
144 views

Shannon's proof of the entropy power inequality

In Shannon's paper on information theory, found here, he asserts the entropy power inequality in appendix 6, found on page 52. I was reading his proof and it seems like there is a gap. Through his ...
81 views

information measure for matrix that is analogous to rank

Is there a measure for matrix that is analogous to rank of the matrix, but it is continuous on matrix elements? Say, we could say the information in identity matrix $I_n$ is $n$, and when the off-...
274 views

Digital physics and “Gandy-like” machines

Various physicists, famously John Wheeler, have asserted that physical information is the central object of study in physics, in the sense that an object or concept is "physically meaningful" if it ...
24 views

is any closed form relation that can state the error probability of code versus its variable and check node degree distributions?

In Low Density parity check code design, when bit (or frame) error probability of code is the objective of the design, we need a closed form relation between error probably (or even an approximate or ...
67 views

85 views

Suppose we have a balanced bipartite planar maximum degree $k$ graph. How many such graphs on $2n$ vertices have at most $f(n)$ maximum number of $4$ cycles for a given function $f:\Bbb R^+\... 0answers 71 views Mutual Information - Correlation, Continuous Random Variables For the Gaussian case$I(X,Y)=f( \varrho )$where$\varrho $is the correlation coefficient, and$f$is a known increasing function. Is there any known joint distribution where the$f$is not strictly ... 0answers 89 views Does this set of (structured) equations always have a solution? Let$r_1,\ldots,r_K$be arbitrary positive numbers. Does $$|\mathcal{A}|\log\left(1+\frac{1}{|\mathcal{A}|}\left(\sum_{n\in \mathcal{A}} \sqrt{x_n(\exp(r_n)-1)}\right)^2\right)\leq \sum_{n\in \... 1answer 113 views Sharpened Pinsker inequality for special case Let B(p) denote the Bernoulli distribution over \{0,1\} and B(p)^n the corresponding product distribution over \{0,1\}^n. For n>1 and 0<x<1, define$$P_n(x):=B(\frac12+\frac x2)^n$...
78 views

Can someone link me to some pedagogic example of computing the Renyi divergence between two discrete/continuous distributions? Like examples where someone has been able to obtain a neat closed form or ...
123 views

199 views

57 views

89 views

Maximize mutual information

Assume $P \in \mathbb{R}^{n \times n}$ describe the joint distribution of the random variable $J$ over the finite set $\mathcal{X}\times \mathcal{X}$. I am interested in finding a right stochastic ...
136 views

De Bruijn sequence inside De Bruijn sequence

A binary De Bruijn sequence of index $n$ is a circular sequence $S=a_1a_2\ldots a_{2^n}$, with $a_i∈\{0,1\}$, and such that each of the $2^n$ binary $n$-tuples occurs exactly once in $S$. What is ...
97 views

irregular LDPC code construction algorithm

I want to construct a sparse random binary matrix ${{\bf{H}}_{m \times n}}$ that has the following properties 1- Faction of columns of weight $i$ is ${v_i}$ . 2- Fraction of rows of weight $i$...
36 views

Reference on interaction information

I am looking for the most complete reference on interaction information/co-information/multivariate mutual information. What are the properties of such quantities? Are they convex, like entropy? When ...