All Questions
Tagged with it.information-theory entropy
24 questions with no upvoted or accepted answers
11
votes
0
answers
307
views
Entropy, magnitude, diversity of finite metric spaces in number theory
I was reading the article by Tom Leinster, (Maximizing
diversity in biology and beyond, arXiv link), and find it very interesting.
Since I was searching for entropies of finite metric spaces I found
...
user6671
6
votes
0
answers
206
views
$q$-deformations of fundamental equation of information and entropies
Classical information theory: fundamental equation of information
In classical information theory, the information $I(A)$ of an event $A$ (any element of the $\sigma$-algebra $\mathcal F$ of a given ...
- 326
6
votes
0
answers
342
views
Maximizing Renyi entropy for a certain channel
The channel under consideration is $T = A + B$, where $A$ and $B$ take on values in $\{0, 1\}$ according to a probability mass function. Let (joint) random vector $(A_1, A_2,\ldots, A_n)$ be denoted ...
5
votes
0
answers
191
views
Divergence for Bhattacharya Information matrix
The Fisher information matrix (in the scalar parameter case) can be obtained from the Kullback-Leibler divergence by
$$g(\theta) = -\frac{\partial}{\partial \theta}\frac{\partial}{\partial \theta'}D(...
- 779
4
votes
0
answers
228
views
Maximazing the joint entropy given the probability of equality
Consider 2 independent random variables $X$ and $Y$ with values in $A=\{0, 1, \ldots, q-1\}$. Suppose that $P(X=Y)$ is equal to some constant $\varepsilon$.
What is the maximal entropy $H(X, Y)$?
At ...
- 41
4
votes
0
answers
573
views
An inequality involving conditional variance and its connection to information theory
Given absolutely continuous random variables $(X, Y)$ with joint distribution $P_{XY}$, we construct $Z:=\sqrt{\gamma} Y+N_\mathsf{G}$ where $N_\mathsf{G}\sim N(0, 1)$ and is independent of $(X,Y)$ ...
- 1,109
3
votes
0
answers
184
views
Is there a known generalization of the Schmidt decomposition based on a maximal set of "locally orthogonal" vectors?
I came across the following unusual generalization of the Schmidt decomposition in my work, which I describe below. I would like to know if this structure has been studied before so I can read more ...
- 846
3
votes
0
answers
180
views
A.G. Vitushkin's "Easily representable families of functions" - can it be generalized?
Background
In his monograph "Estimation of the complexity of the tabulation problem" (translated into English as "Theory of the Transmission and Processing of Information") Vitushkin studies ...
- 959
2
votes
0
answers
120
views
Information inequality for Renyi divergences
Let $X^1 \ldots X^n$ be random variables on $\mathbb{R}^d$ with an arbitrary joint probability distribution $\mu$ on $\mathbb{R}^{n \times d}$. Let $\nu = \nu^1 \times \ldots \times \nu^n$ be a ...
2
votes
0
answers
111
views
Generalization of the min-entropy that looks at the top $n$ probabilities
The min-entropy of a random variable $X$ can often be much easier to compute than the Shannon entropy. This is because the min-entropy is simply a function of the most probable value, and sometimes, ...
- 4,897
2
votes
0
answers
142
views
List decodability of Reed-Solomon codes beyond the Johnson bound
In a paper on a proximity test for Reed-Solomon codes the authors state an "extremely optimistical" conjecture on the list decodability of Reed-Solomon codes (over prime fields $\mathbb F_q$)...
- 81
2
votes
0
answers
264
views
Prove or disprove a mutual information inequality
I have $n$ IID Bernoulli random variables denoted by $X_1,X_2,\ldots X_n$ with parameter $p$.
I am interested in knowing if the following inequality involving mutual information holds :
$\boxed{\max_{...
- 83
2
votes
0
answers
132
views
A result of the covering number
Suppose $\mathcal{F} = \{f_x : x \in \mathbb{R}^d \}$ and each $f_x$ shares the same law $P$. If $\mathcal{F}$ is a class of uniformly bounded functions satisfying $L_r$-continuity, i.e. $\forall f \...
- 331
2
votes
0
answers
92
views
What are the moments of Kolmogorov Complexity for a Random Variable?
Given a random variable $X$ distributed under some computable distribution $P$ we have,
$$0 \le E[K(X)] - H(P) \le K(P)$$
Where $H(P)$ is the entropy of $P$. I tried using Hoeffding concentration ...
2
votes
0
answers
50
views
Do averaged binary symmetric channels maximize mutual information?
This is a refined version of Do binary symmetric channels maximize mutual information?, which was answered negatively.
Let the random variables $(X, Y)$ be a doubly symmetric binary source with ...
2
votes
0
answers
413
views
Interpretation of Shannon Entropy Application
Consider a collection of increasing positive integers $\{a_i\}_{i=1}^m$ and the distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$. Let entropy of $\mathcal{A}=\{a_i\}_{i=1}^m$ be given
by $$H(\mathcal{A}...
- 13.9k
1
vote
0
answers
428
views
When inequality in Mrs. Gerber's lemma is almost equality?
Let $X=x_1\ldots x_n$ be a random variable.
Assume that every $x_i$ takes values in $\{0,1\}$.
Assume also that for every $I \subseteq \{1,\ldots, n\}$ the Shannon entropy of random value $X_I$
[if $I ...
- 2,576
1
vote
0
answers
83
views
Convexity of conditional relative entropy for Markov distributions
Consider two Markov processes $p$ and $q$. The conditional relative entropy between them is
\begin{align}
D(p\parallel q)& =\sum_a p(a)\sum_b p(b\mid a)\log\frac{p(b\mid a)}{q(b\mid a)}\\
& =\...
- 11
1
vote
0
answers
432
views
What is the maximum entropy distribution over the integers
Let $μ=0,σ>0$. What is the maximum entropy distribution over the integers with mean $μ$ and variance $σ^2$?
Is Skellam distribution a maximum entropy distribution? Is there a closed-form ...
1
vote
0
answers
108
views
Median entropy to observe evolution of system?
Hello,
I am studying a dynamical system that takes as an initial condition a list. I want to analyze the evolution of Shannon's entropy in this system. I know the maximum entropy (50) and the minimum ...
- 11
0
votes
0
answers
14
views
What are examples of $\epsilon$-extractable uniform randomness $H_{\rm ext}^\epsilon(\cal P)$?
[Renner and Wolf 2004] introduces the notion of $\epsilon$-smooth Renyi entropies as
$$H_\alpha^\epsilon(P) \equiv \frac{1}{1-\alpha} \inf_{Q\in \mathcal B^\epsilon(P)}\log\left(\sum_z Q(z)^\alpha\...
- 342
0
votes
0
answers
179
views
About the monotonicity of the exponential entropy
This question was previously posted on MSE at About the monotonicity of the exponential entropy.
In the paper The Unifying Frameworks of Information Measures the conditional exponential entropy (see ...
- 127
0
votes
0
answers
85
views
Does there exist an established name for the exponential of surprisal (e.g. the reciprocal of probability?)
There are several different names that I know of for the exponential of the entropy of which "diversity" and "perplexity" are fairly well-established. Tom Leinster has a very ...
- 4,897
0
votes
0
answers
404
views
Is there a difference between using nats and bits to express entropy?
It seems to me like questions involving decimal vs binary representations of some number are not particularly interesting: for instance $\pi$ or $\sqrt{2}$ are conjectured to be normal in every base, ...
- 226