Covers theoretical and experimental aspects of information theory and coding.

learn more… | top users | synonyms

0
votes
0answers
34 views

Maximal Correlation with Weak Gaussian Perturbation

Let a pair of random variables $(X,Y)$ be continuous random variables (i.e., they both have density with respect to Lebesgue measure) with joint distribution $P_{XY}$. The maximal correlation ...
7
votes
2answers
166 views

concentration inequality for entropy from sample

Consider a measure $\mu$ on a finite set, and let $x_1, \ldots, x_n$ be i.i.d samples from $\mu$. Then the expression $S_n = -\frac{1}{n} \sum_{i=1}^n \log \mu(x_i)$ converges by a.s. to the entropy ...
5
votes
0answers
81 views

Maximal Correlation versus Correlation Coefficient When one RV is Gaussian

Let a pair of random variables $(X,Y)$ be continuous random variables (i.e., they both have density with respect to Lebesgue measure) with joint distribution $P_{XY}$. The maximal correlation ...
1
vote
0answers
51 views

Is there a geometric meaning behind these specific triples?

Consider the set of triples $(g_1,g_2,g)\in(\Bbb R^+)^3$ such that $$\log g=(\log g_1)(\log g_2)$$ Is there any geometric or information theoretic meaning behind such triples? We have $$2\log g=2 ...
1
vote
0answers
46 views

How effective is using local property to test Shannon capacity?

A key tool in graph theory is the laplacian which is a local property. We can form a semidefinite programming and get an upper bound for Shannon capacity using laplacian. Shannon capacity is ...
10
votes
3answers
259 views

What characterizations of relative information are known?

Given two probability distributions $p,q$ on a finite set $X$, the quantity variously known as relative information, relative entropy, information gain or Kullback–Leibler divergence is defined ...
1
vote
0answers
96 views

Chain Rule for Maximal Correlation

Let a pair of random variables $(X,Y)$ be defined over finite alphabet $\mathcal{X}\times \mathcal{Y}$ with joint distribution $P_{XY}$. The maximal correlation $\rho(X;Y)$ between $X$ and $Y$ is ...
1
vote
1answer
73 views

An inequality for Maximal Correlation over a Markov Chain

Let a pair of random variables $(X,Y)$ be defined over finite alphabet $\mathcal{X}\times \mathcal{Y}$ with joint distribution $P_{XY}$. The maximal correlation $\rho(X;Y)$ between $X$ and $Y$ is ...
1
vote
1answer
129 views

Uniquely describing a graph

According to answers here http://math.stackexchange.com/questions/1524598/a-general-incidence-problem// an unigraph comes from unigraphic degree sequences if it can be uniquely determined by its ...
2
votes
0answers
124 views

capacity of a channel with random phase rotation

Consider a wireless channel $h=e^{j\theta}$, where $\theta$ is a uniform random variable in $[0,2\pi]$ independent of the input messages and the independent of the noise. The channel randomly rotates ...
0
votes
0answers
27 views

Is it possible to estimate the Interaction information of three variables without knowing their joint distributions?

I want to have a measure of the "synergy" between two players in a game. Each player has its own win ratio (won/played), which I'm modeling as two binomial distributed random variables X and Y. A ...
5
votes
1answer
269 views

An Inequality of KL Divergence

Given two probability distributions $P$ and $Q$ defined over a finite set $\mathcal{X}$, one can define the KL divergence between $P$ and $Q$ as $$D(P||Q):=\sum_{x\in ...
1
vote
0answers
78 views

Maximize mutual information

Assume $P \in \mathbb{R}^{n \times n}$ describe the joint distribution of the random variable $J$ over the finite set $\mathcal{X}\times \mathcal{X} $. I am interested in finding a right stochastic ...
7
votes
0answers
125 views

De Bruijn sequence inside De Bruijn sequence

A binary De Bruijn sequence of index $n$ is a circular sequence $S=a_1a_2\ldots a_{2^n}$, with $a_i∈\{0,1\}$, and such that each of the $2^n$ binary $n$-tuples occurs exactly once in $S$. What is ...
6
votes
0answers
80 views

irregular LDPC code construction algorithm

I want to construct a sparse random binary matrix ${{\bf{H}}_{m \times n}}$ that has the following properties 1- Faction of columns of weight $i$ is ${v_i}$ . 2- Fraction of rows of weight ...
0
votes
0answers
31 views

Reference on interaction information

I am looking for the most complete reference on interaction information/co-information/multivariate mutual information. What are the properties of such quantities? Are they convex, like entropy? When ...
13
votes
0answers
316 views

Research situation in the field of Information Geometry

I am now doing an article survey on the field of information geometry started by S.Amari and Barndorff-Nielson. I want to know some research situation in this field. I have read (4) and parts of (3). ...
3
votes
1answer
91 views

Effects of many degree-2 variable nodes in the Tanner graph during the decoding of LDPC codes

Suppose that we have a LDPC code $C$ with a $(n -k )\times n $ parity check matrix $H$, and there exist approximately $ \sqrt n$ numbers of degree-2 columns. It means that there are approximately ...
0
votes
0answers
61 views

Efficient recognition of sequences sortable by transpositions?

While reading the post, Probability of generating a desired permutation by random swaps, by Aaronson, I got interested in this sorting problem: Input: a sequence $A$ of $2N$ positive integers. ...
3
votes
0answers
125 views

Do binary symmetric channels maximize mutual information?

Consider the following setup: $(X, Y)$ is a doubly symmetric binary source with parameter $0 < p < 1/2$, i.e., $X \sim \text{Bernoulli}(1/2)$, $Z \sim \text{Bernoulli}(p)$ and $Y = X \oplus Z$. ...
2
votes
1answer
107 views

Enumerator Polynomials for Linear Anytime Codes

Let $C = \{c \in \mathbb{F}^n_2 : Hc=0\}$ be a binary linear code where $H \in \mathbb{F}^{k \times n}_2$ is a block lower-triangular matrix of full rank called the parity-check matrix of $C$. Clearly ...
3
votes
1answer
236 views

Information theory from negative probability

Szekely provides a convincing argument of negative probability here: http://www.wilmott.com/pdfs/100609_gjs.pdf What does a reformulation of classical information theory built from negative ...
2
votes
0answers
32 views

MLE and CRLB with mismatched likelihoods

Suppose that I can do a Karhunen-Loeve expansion of a log-likelihood function $p(\bf{x};\theta)$ into N terms and that these accounts for a fraction $1-\delta$ of the total energy. Now consider ...
1
vote
0answers
68 views

How do you use the bits you get back from Bits Back Coding?

Bits Back coding is a scheme to transmit an observation x. You can read about it here [1]. To my understanding, it works like this: The encoder samples a message z from a distribution Q(z|x) that it ...
0
votes
1answer
95 views

Information theoretic common sequence agreement (not secret key)

Supposing Alice and Bob share $\rho$-correlated sequences in $\{0,1\}^n$, what coding theory based schemes are available for Alice and Bod to extract sequences $A,B\in\{0,1\}^n$ respectively such that ...
7
votes
1answer
235 views

A Generalized Version of Maximal Correlation and Hypercontractivity of Conditional Expectation Operator

Given a pair of random variables $(X,Y)$ over a product space $\mathcal{X}\times \mathcal{Y}$, the maximal correlation coefficient is defined as ...
5
votes
1answer
83 views

Rate-Distortion theory: What is the distribution of distortion on an optimal Gaussian encoder?

If we wish to encode a gaussian source, $X\sim\mathcal{N}(0,\sigma^2)$ at rate $R$, then decode it to create an estimate $\hat{X}$, rate-distortion theory tells us that the lowest mean-squared-error ...
9
votes
2answers
425 views

Expected centered entropy of the binomial distribution

In short, the function I am interested in is the following: $$I_n(p) = \sum_{k=0}^n \binom{n}{k} p^k (1-p)^{n-k} \left[h(p) - h\left(\tfrac{k}{n}\right)\right],$$ where $h(x) \triangleq -x \log x - ...
1
vote
0answers
42 views

Channels of Practical Interest

Are there channels of practical interest whose capacity achieving probability distributions belong to a parametric family of probability measures? Specifically, suppose $ \theta= (\theta_1 \theta_2 ...
2
votes
0answers
295 views

Good covering of a sphere

Consider a sphere $S_r(0)$ with center at zero and radius $r$ in the Hamming space $\{0,1\}^n$. We will be interested in covering this sphere with balls of radius $\rho < r$. We know that there ...
0
votes
0answers
135 views

Upper bound for chi-square divergence in terms of KL divergence

In my research I need an upper bound for chi-square divergence in terms KL divergence which works for general alphabets. To make this precise, note that for two probability measures $P$ and $Q$ ...
1
vote
1answer
298 views

Coupon Collector Problem for Non-Uniform Coupons: Bound on the number of missed Coupons

Suppose $\mathcal B=\{1,2,..,b\}$ is the set of all possible coupons, with $\mathbf p = ( p_1,p_2,...,p_b)$ assigning the probability of occurrence for all coupons in $\mathcal B$. The "traditional ...
3
votes
0answers
92 views

On Rényi entropy/divergence

The Rényi entropy for a probability density function $f$ with dominating measure $\mu$ of order $\alpha>0$ is defined as $$H_\alpha(f)={1 \over {\alpha-1}}\log\int f^\alpha d\mu.$$ If $f$ is ...
1
vote
1answer
251 views

Correlation between two continuous-time stochastic processes

Consider two continous-time stochastic processes $\{A(t)\}_{t \ge 0}$ and $\{B(t)\}_{t \ge 0}$ with $A(t)=t$ and $B(t)=t$. Each process starts at $t=0$ and emits "ticks" at increasing time slots. For ...
2
votes
2answers
138 views

What is the sum capacity of a scalar gaussian broadcast channel?

"On the Achievable Throughput of a Multiantenna Gaussian Broadcast Channel" by Giuseppe Carie and Shlomo Shamai talks, in part, about the following type of link (paraphrasing): A transmitter with ...
0
votes
0answers
79 views

Continuous self-information

Let $I(X,Y)$ be the mutual information between two continuous random variables $X$ and $Y$. We have $I(X,Y) = H(X)-H(X|Y)$, and setting $X=Y$ leads to $I(X,X) = H(X)-H(X|X)$. If $X$ was discrete, ...
2
votes
0answers
88 views

On subset of Deterministic games

Denote strings $u,v$ from $\{0,1\}^n$. Denote concatenated pair $[uv]$. Denote $$[uv]_{1}=\{[uv]\oplus e_i\}_{i=1}^{2n}$$ collection of pairs with Hamming distance $1$ from $[uv]$ string ...
3
votes
0answers
195 views

Convex Conjugate of Relative Entropy

The convex conjugate of a function, say, $f:X\mapsto \mathbb{R}$ is a function $f^*:X^*\mapsto \mathbb{R}$ defined as $$f^*(x^*):=\sup_{x\in X} ~\langle x, x^*\rangle-f(x),$$ where $X^*$ is the ...
2
votes
0answers
220 views

Interpretation of Shannon Entropy Application

Consider a collection of increasing positive integers $\{a_i\}_{i=1}^m$ and the distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$. Let entropy of $\mathcal{A}=\{a_i\}_{i=1}^m$ be given by ...
1
vote
0answers
34 views

Multidimensional Filters

Say you want to design a LP FIR filter with low pass cutoff $fc$, transition band $fc$ to $fs$ and ripple factor $dp$ at passband and $ds$ at stop band. If one divides the frequencies by $\pi$, then ...
8
votes
1answer
222 views

Higher moments of information and Renyi entropy

For a given discrete probability distribution, Shannon entropy can be though as an expectation value $\langle - \log p \rangle$ (see also: What is entropy, really?, What is the role of the logarithm ...
1
vote
0answers
43 views

Distributing partially known data between n parties

Assume that $n = 2r+1$. There are $n$ elements $a_1,a_2,\ldots,a_n$ from a finite field $\mathcal{F}$, and $n$ parties. Each party knows the values of at least $r+1$ elements out of those $n$ ...
7
votes
0answers
157 views

Cramer's theorem in Hilbert spaces

I am interested under what conditions Cramer's theorem applies in random variables taking values in Hilbert spaces. Following these lecture notes, but using a Hilbert space: Let $X_1,X_2,\cdots$, be ...
4
votes
0answers
102 views

Large Deviations: Exponential decay in normed spaces

Let $(X_1,X_2,\cdots)$ be a sequence of independent and identically distributed random variables taking values in some general normed space $(V,||\cdot||)$. Denote $\mu=E[X_1]$ and ...
0
votes
1answer
170 views

On the inverse problem of Dobrushin

Dobrushin, in this paper, looked into the following problem. Suppose We are given a Markov kernel (conditional distribution) $P_{Y|X}$. Information theorist usually call $W$ a channel. It is known ...
0
votes
2answers
125 views

Form of the Shannon capacity for Heptagon?

Is the $0$-error capacity of $7$-cycle: $(1)$ known to be of form $7^q$ for some $q\in \mathbb Q$?
1
vote
1answer
109 views

Coding for channels with concentrated error

Can we implement a reduced-error transmission over a channel with error frequency having $\liminf<\frac{1}{2}$ and $\limsup>\frac{1}{2}$? We know that if a channel with error flips (in the ...
2
votes
3answers
426 views

Asking for an English version of a paper

I have been looking for the paper "almost independence and secrecy capacity" by Csiszar. But all I could find was a Russian version published in Problems of Information Transmission. I am wondering ...
-1
votes
1answer
113 views

Noise reduction in capacity-0 channels

Suppose we have a binary symmetric channel with $p=\frac{1}{3}$; that is, a communications channel in which each bit is flipped with independent probability $\frac{1}{3}$. I know that there is a code ...
1
vote
1answer
166 views

Total conditional complexity

By $C(|)$ denote conditional complexity. By $CT(|)$ denote total conditional complexity. For every n there exist two strings $x$ and $y$ of length $n$ such that $C(x|y) = O(1)$ but $CT(x|y) \ge n $. ...