# Questions tagged [it.information-theory]

Theoretical and experimental aspects of information theory and coding theory. This tag covers but is not limited to following branches: information theory, information geometry, optimal transportation theory, coding theory.

540 questions
Filter by
Sorted by
Tagged with
30 views

### How can one compute schema quality using information theory?

I initially asked this question on math.SE (linked here). It has been more than a week since the question was posted and I have received no answers even though I offered a 200 reputation bounty (the ...
26 views

### Extension of Data Processing Inequality: If $X \rightarrow Z \rightarrow Y$, $I(X, ZY) \geq I(X, Y)$?

If we have the Markov chain $X \rightarrow Z \rightarrow Y$, we can say $I(X, Z) \geq I(X, Y)$ from the data processing inequality. Then, can we say that $I(X, ZY) \geq I(X, Y)$?
231 views

### Bipartite version of Hamming bound (two families of codewords with large Hamming distance)?

Update: In light of Fedor Petrov's answer, I added an additional requirement that all strings in $A$ and $B$ have Hamming weight exactly $n/2$, which hopefully makes the question more interesting. ...
1 vote
78 views

### A question about mutual information

Let $A$ and $B$ be two, possibly dependent, random variables, and let $X$ be a random variable independent of $(A,B)$. For simplicity, let's concern ourselves with discrete random variables. Is the ...
30 views

14 views

### sample complexity of hypothesis testing with non-uniform prior

Given two hypothesis $$\mathcal{H}_0:\; x_i\underset{iid}{\sim} p_0(x), i=1,\cdots,n\\ \mathcal{H}_1:\; x_i\underset{iid}{\sim} p_1(x), i=1,\cdots,n$$ with priors $p$ and $1-p$ ($p<1/2$) ...
34 views

### Prove the statistical rate lower bound of a given complicated statistics

Given a i.i.d. sequence of random variables $\{Z_i\}_{i=1}^n$ who has mean zero. Two i.i.d. sequence of random vectors $\{X_i\}_{i=1}^n$, $\{Y_i\}_{i=1}^n$ who have the same covariance matrix $\Sigma$....
1 vote
27 views

### Correlating two matrices $A,B$ with stochastic dependency structure imposed by cross-validation

Consider a labelled data set $$D = \{(x_1, y_1),...,(x_n, y_n)\}$$ on which we want to evaluate a machine learning algorithm using $k$-fold cross validation with $m$ different random seeds. This ...
120 views

49 views

81 views

### KL divergence between two sequences

Let us have a random sequence $(X_1, Y_1,\ldots,X_n,Y_n)$, where $X_t$ takes value in some set $\mathcal{X}$ and $Y_i$ are scalars. The sequence is generated by the following process: $X_i$ is chosen ...
135 views

### An inequality in the optimality of Bayes' theorem

$\DeclareMathOperator\Ent{Ent}\newcommand{\prior}{\mathrm{prior}}\newcommand\Data{\mathrm{Data}}$I came across this paper on the optimality of Bayes' theorem https://sinews.siam.org/Portals/Sinews2/...
308 views

### Has the von Neumann entropy ever been used in classical mechanics?

After going through an application of the von Neumann entropy(from quantum information theory) to certain problems in computational neuroscience , it occurred to me that this entropy might have ...
3k views

### John von Neumann's remark on entropy

According to Claude Shannon, von Neumann gave him very useful advice on what to call his measure of information content : My greatest concern was what to call it. I thought of calling it '...
2k views

### Geometric interpretations of the exponential of entropy

Question: Might there be a natural geometric interpretation of the exponential of entropy in Classical and Quantum Information theory? This question occurred to me recently via a geometric inequality ...
3k views

### Who is Mrs. Gerber?

This question on a theorem in information theory called Mrs. Gerber's lemma piqued my curiosity. Who is this individual, and why the "mrs." ? A quick Google search was not informative, ...
1 vote
311 views

77 views

202 views

### Mutual Information after Applying Random Unitary Matrix

Let $\mathbf{U}$ be a random unitary matrix and $\mathbf{z}$ be a random i.i.d complex Gaussian vector (unitary invariant). Assume that the following relation is satisfied: \begin{align} \mathbf{y}=\...
1 vote
In a paper that I am reading the authors defines $\mathbb P(n,q)$ the space of covariance tensors for $\mathbb R^q$-valued Gaussian processes on an abstract finite space $K=\{x_1,\dots,x_n\}$. In his ...