The entropy tag has no wiki summary.

**1**

vote

**0**answers

21 views

### minimal entropy approximation of a discrete random variable

Let $X$ be a $\mathbb{N}$-valued random variable. Define
$$
H^\epsilon_n(X) = \inf_f H(f(X))
$$
where $f$ runs over all functions $\mathbb{N} \to \mathbb{N}$ such that $\Pr(f(X)\neq X)<\epsilon$ ...

**1**

vote

**1**answer

49 views

### Discrete Maximum Entropy Distribution with given mean

For a given mean $\mu$, what is the entropy maximizing probability distribution on the nonnegative integers?
Different sources indicated either the geometric or the Poisson distribution for this. As ...

**2**

votes

**0**answers

42 views

### Sequence transformations that are entropy invariant

Consider a collection of increasing positive integers $\{a_i\}_{i=1}^m$ and the distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$. Define entropy of $\mathcal{A}=\{a_i\}_{i=1}^m$
by ...

**2**

votes

**0**answers

115 views

### Interpretation of Shannon Entropy Application

Consider a collection of increasing positive integers $\{a_i\}_{i=1}^m$ and the distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$. Let entropy of $\mathcal{A}=\{a_i\}_{i=1}^m$ be given
by ...

**1**

vote

**0**answers

42 views

### Limiting Entropy of deterministic sequences - 2

Consider a collection of increasing positive integers $\{a_i\}_{i=1}^m$ and the distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$. Given $\{a_i\}_{i=1}^m$, let $\mathcal{P}_a$ be limiting distribution ...

**1**

vote

**1**answer

101 views

### Limiting Entropy of deterministic sequences - 1

Consider a collection of increasing positive integers $\{a_i\}_{i=1}^m$.
Consider distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$. Given $\{a_i\}_{i=1}^m$, let $\mathcal{P}_{a,m}$ be distribution at ...

**1**

vote

**1**answer

73 views

### Entropy difference dominance of sequences

Consider a collection of positive integers $\{a_i\}_{i=1}^m$ and the distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$.
Similarly for the collection $\{a_i\}_{i=1}^{m+1}$ form the distribution ...

**1**

vote

**1**answer

55 views

### Entropy dominance of certain restricted sequenes

Say you have positive $\{a_i\}_{i=1}^n$ and you have $p_i=\frac{a_i}{\sum_{i=1}^na_i}$, then assume you have a $C$ such that $C<2a_n\ll\sum_{i=1}^na_i$ (that is $C$ is not very large), then define ...

**2**

votes

**1**answer

62 views

### Entropy dominance

Let $0<a<b<c$ be distinct positive reals.
Define four different probability distributions:
$$\mathcal{P}_{ab}:P_{a,ab}=\frac{a}{a+b}=1-P_{b,ab}$$
...

**8**

votes

**1**answer

175 views

### Higher moments of information and Renyi entropy

For a given discrete probability distribution, Shannon entropy can be though as an expectation value $\langle - \log p \rangle$ (see also: What is entropy, really?, What is the role of the logarithm ...

**2**

votes

**0**answers

54 views

### Property of relative entropy [closed]

For $X$ a measurable space and $P,Q$ two probability measure on $X$ s.t. $Q$ is absolutely continuous with respect to $P$, the relative entropy is defined as
$$D(Q\|P)=\int_X ...

**14**

votes

**8**answers

803 views

### When do people actually use the maximum entropy distribution?

One of the standard problems in convex optimization is the calculation of the maximum entropy distribution that satisfies some set of criteria. For example, if $\mathbf{x} \in \mathbb R^n$ is an ...

**2**

votes

**1**answer

55 views

### Average entropy of quantum system in bipartite pure state for finite temperature

[I got halfway through writing this when I found the paper that answers the question in (essentially) the affirmative. I'll post it anyways in case anyone is interested.]
Background: If a random ...

**2**

votes

**1**answer

60 views

### Which is the right way to compute the Approximate Entropy (ApEn)?

My problem is the inconsistency between the definition and the computation of the Approximate entropy (ApEn).
Suppose $u = (u_i:1\leq i \leq N)$ is a sequence of ...

**1**

vote

**1**answer

88 views

### Entropy on a draw from a random distribution.

Suppose I am attempting to calculate the entropy of a continuous, normally distributed random variable $X$, from the distribution $\mathcal{N}(\mu, \sigma)$. This is easy to to do - I just calculate
...

**4**

votes

**0**answers

76 views

### $q$-deformations of fundamental equation of information and entropies

Classical information theory: fundamental equation of information
In classical information theory, the information $I(A)$ of an event $A$ (any element of the $\sigma$-algebra $\mathcal F$ of a ...

**3**

votes

**0**answers

75 views

### How many samples to accurately estimate the entropy

Consider a discrete random variable $N$ with range $\{1,\dots,1000\}$, say. How many samples do you need to take to be sure that you can, at least in principle, compute the entropy Shannon entropy ...

**1**

vote

**0**answers

36 views

### Characterization of the optimal solution in relative entropy minimization

The following optimization problem is related to relative entropy and to the limit of the iterative proportional fitting procedure.
For $1 \leq i,j \leq n$ and fixed $w_{ij} \geq 0$, and fixed $a_i, ...

**1**

vote

**1**answer

78 views

### Invariance of mutual information

Let $I(X,Y):=H(X)+H(Y)-H(X,Y)$ be the mutual information of the joint probability distribution $p_{XY}$ (here $H(\cdot)$ is the Shannon entropy of its argument). I know that the mutual information is ...

**3**

votes

**0**answers

117 views

### On the volume entropy of negatively curved manifolds

Let $X$ be the universal cover of a closed negatively curved Riemannian manifold. Let $x_0\in X$ be a base point, $S$ be the unit sphere in $T_{x_0}X$ and $\exp:T_{x_0}X\rightarrow X$ be the ...

**0**

votes

**1**answer

123 views

### Why private randomization does not help the Shannon's source coding

I am wondering why stochastic encoder and decoder can not help the Shannon source coding? I know the achievability scheme of source coding, which is based on typicality, is deterministic, and hence we ...

**9**

votes

**2**answers

324 views

### Entropy for Haar measure on $O(n)$

Let $G$ be a locally compact group. A measure $\mu$ is the right-Haar measure on $G$ if for every $g\in G$ and $E\subseteq G$ Borel set $\mu(Eg)=\mu(E)$. It is known that every locally compact group ...

**11**

votes

**2**answers

228 views

### Intrinsic significance of differential entropy

Many commentators (e.g. Jaynes, Rota) argue that the notion of "differential entropy" is problematic (as commonly defined by $ h(X) = \int ( \log\frac{1}{p(x)} ) p(x) \, dx $, where $X$ is a random ...

**1**

vote

**3**answers

127 views

### How to estimate the entropy of a distribution on a power set?

Given a probability distribution $(X,p)$, its entropy is defined as $H=-\sum_{x\in X} p(x)\log p(x)$.
Given a sample of observations $x_n,n=1..N$, one can estimate $p(x)=\frac{\#\{i:x_i=x\}}{N}$ and ...

**1**

vote

**1**answer

75 views

### Connection between inf-entropy rate and min-entropy

I am reading the paper "Generating random bits from an arbitrary source: fundamental limits" by Vembu and Verdu. This paper is written in the language of information theory, however, I need to ...

**2**

votes

**0**answers

30 views

### largest size for a randomness extractor

I am not so expert in theoretical computer science, so sorry if the question is trivial, i just could not find it in literature.
Suppose we have a source $X$ with min-entropy $\ell$, the randomness ...

**11**

votes

**0**answers

166 views

### Entropy in elimination theory, or a brief remark by Gelfand-Kapranov-Zelevinsky

In the introduction to their book "Discriminants, resultants and multidimensional determinant", the authors state a very intriguing observation concerning the coefficients of monomials appearing in ...

**5**

votes

**1**answer

158 views

### General additive function of probability

Let $H$ be a function of finite sequences of probabilities (non-negative numbers summing up to 1) into real numbers, such that:
$H$ is continuous,
$H$ is symmetric w.r.t. the order of its arguments,
...

**2**

votes

**0**answers

51 views

### Private Randomness extractor

Suppose we are given two random variables $X$ and $Y$ with fixed marginal and joint distribution. What is the maximum randomness that we can extract from $Y$ that is independent from $X$, that is, if ...

**5**

votes

**0**answers

196 views

### Maximizing Renyi entropy for a certain channel

The channel under consideration is $T = A + B$, where $A$ and $B$ take on values in $\{0, 1\}$ according to a probability mass function. Let (joint) random vector $(A_1, A_2,\ldots, A_n)$ be denoted ...

**2**

votes

**0**answers

137 views

### metric entropy for Lipschitz functions

Suppose $(X,d)$ is a metric space of unit diameer and let $F$ be the collection of all $1$-Lipschitz functions mapping $X$ to $[-1,1]$, equipped with the sup-norm $||\cdot||_\infty$.
I am interested ...

**8**

votes

**2**answers

377 views

### Inequality in information theory

I am reading the paper "chain independence and common information" (http://ttic.uchicago.edu/~yury/papers/independ.pdf). In this paper, an inequality is used several times (without proof) which looks ...

**0**

votes

**1**answer

181 views

### Approximation of the sum involving binary entropy function

Given the following sum:
$S(n) = \sum_{i=1}^{n} \frac{1}{(1-\operatorname{H}(p))^i}$
where $H$ is the binary entropy function defined as:
$\operatorname{H}(p) = -p\log p - (1-p)\log (1-p) $.
Let ...

**3**

votes

**1**answer

378 views

### Simple reason that a mathematician cannot do better than random when guessing contents of a box?

I have a question about the finite analog of the puzzle proposed here involving mathematicians guessing the contents of boxes.
Specifically, suppose there are $k$ unopened boxes each containing a ...

**3**

votes

**1**answer

127 views

### Is there a one-dimensional subshift of positive entropy s, all of whose sub-subshifts also have entropy s?

A subshift is a subset $X$ of $A^\mathbb{N}$ or $A^\mathbb{Z}$ (with $A$ finite), such that $X$ is topologically closed and closed under the shift operation. The shift operation is defined by ...

**16**

votes

**1**answer

748 views

### Gini Coefficient and Renyi Entropy

Gini coefficient (aka Gini Index) is a quantity used in economics to describe income inequality. It is 0 for uniformly distributed income, and approaches 1 when all income is in hands of one ...

**-3**

votes

**1**answer

98 views

### “logical distance” link algorithmic complexity to statistical information [closed]

Someone mentioned what I think was referred to as 'logical distance'.
My hard drive crashed and I dont have the link anymore.
I do recall that I ran across it on this site, in response to linking ...

**2**

votes

**0**answers

130 views

### Reference for and Properties of the $alpha$-entropy

Let $T : X \to X$ be a continuous map on, say, a compact metric space $X$. Let $\mu$ be an invariant borel measure. Under suitable conditions, a result of Brin and Katok states that $\mu$-almost ...

**2**

votes

**0**answers

101 views

### order of convergence of the conditional entropy (3)

I'm sorry for having open two questions which have been solved by elementary counter-examples provided by @AnthonyQuas. Actually I'm not an expert in information theory and I expected that a positive ...

**1**

vote

**1**answer

103 views

### order of convergence of the conditional entropy (2)

Let $X_n$ be a random variable distributed on $A_n:=\{1, \ldots, n\}$ and $g_n\colon A_n \to A_n$ such that $\Pr\big(X_n \neq g_n(X_n)\big) \to 0$. Putting $Y_n=g_n(X_n)$, then by Fano's inequality ...

**3**

votes

**1**answer

132 views

### Example of Girsanov change of density with finite relative entropy, but with infinite integral over squared changed drift

Let $(\Omega, (\mathcal F_t), \mathbb P)$ denote the usual Wiener space where $\Omega = C[0,\infty)$, etc., and where $(W_t)_{t \geq 0}$ denotes the Wiener process.
Let $Z \in L^1(\mathbb P)$ with $Z ...

**3**

votes

**0**answers

85 views

### Is there a known generalization of the Schmidt decomposition based on a maximal set of “locally orthogonal” vectors?

I came across the following unusual generalization of the Schmidt decomposition in my work, which I describe below. I would like to know if this structure has been studied before so I can read more ...

**18**

votes

**2**answers

1k views

### An Entropy Inequality (generalized)

Let $X,Y$ be probability measures on $\{1,2,\dots,n\}$. For $0\le \alpha \le 1$, set $K=\sum_i X(i)^\alpha Y(i)^{1-\alpha}$ so that $Z:=\frac{1}{K}X^\alpha Y^{1-\alpha}$ is also a probability ...

**30**

votes

**1**answer

2k views

### An Entropy Inequality

Let $X,Y$ be probability measures on $\{1,2,\dots,n\}$, and set $K=\sum_i\sqrt{X(i)Y(i)}$ so that $Z:=\frac{1}{K}\sqrt{XY}$ is also a probability measure on $\{1,2,\dots,n\}$. How can we prove the ...

**4**

votes

**3**answers

474 views

### Incremental entropy computation

After a quick internet search I found no method for incremental entropy computation.
Question 1
Let $\{x_i\}_{i=1}^n$ and $\{x_i\}_{i=1+n}^{n+m}$ be two samples and let $S_i^j:=\sum_{k=i}^j x_k$. ...

**3**

votes

**3**answers

963 views

### The relations between the Perelman's entropy functional and notions of entropy from statistical mechanics

I am looking for the relations and analogies between the Perelman's entropy functional,$\mathcal{W}(g,f,\tau)=\int_M [\tau(|\nabla f|^2+R)+f-n] (4\pi\tau)^{-\frac{n}{2}}e^{-f}dV$, and notions of ...

**0**

votes

**1**answer

110 views

### order of convergence of the conditional entropy

Let $X_n$ be a random variable distributed on $A_n:=\{1, \ldots, n\}$ and $g_n\colon A_n \to A_n$ such that $\Pr\big(X_n \neq g_n(X_n)\big) \to 0$. Putting $Y_n=g(X_n)$ then by Fano's inequality ...

**9**

votes

**1**answer

577 views

### Entropy of edit distance

The edit or Levenshtein distance between two strings is the minimum number of single character insertions, deletions and substitutions to transform one string into another. If we take random binary ...

**1**

vote

**1**answer

336 views

### Convexity and semicontinuity of the relative entropy function

There are several different definitions of relative entropy, and some of them are not equivalent. Following is the definition we will use in this question.
Let $M$ be a closed manifold and ...

**1**

vote

**1**answer

349 views

### Calculate channel capacity of general channel under constraint

Hi!
Given a conditional distribution $P_{Y|X}$ I'd like to find the prior distribution $P_X$ that maximizes the mutual information $I(X;Y)$ with $P_Y(y)=\int P_{Y|X}(y|x)P_X(x)\text{dx}$ (this ...