# Tagged Questions

117 views

### Intrinsic significance of differential entropy

Many commentators (e.g. Jaynes, Rota) argue that the notion of "differential entropy" is problematic (as commonly defined by $h(X) = \int ( \log\frac{1}{p(x)} ) p(x) \, dx$, where $X$ is a random ...
113 views

### General additive function of probability

Let $H$ be a function of finite sequences of probabilities (non-negative numbers summing up to 1) into real numbers, such that: $H$ is continuous, $H$ is symmetric w.r.t. the order of its arguments, ...
49 views

### What is entropy of a variable described by Knightian uncertainty? [closed]

I have asked this question at Theoretical Computer Science and received no response. Given a discrete variable whose value is characterized by Knightian uncertainty, that is, belief and plausibility, ...
149 views

### Maximizing Renyi entropy for a certain channel

The channel under consideration is $T = A + B$, where $A$ and $B$ take on values in $\{0, 1\}$ according to a probability mass function. Let (joint) random vector $(A_1, A_2,\ldots, A_n)$ be denoted ...
351 views

### Simple reason that a mathematician cannot do better than random when guessing contents of a box?

I have a question about the finite analog of the puzzle proposed here involving mathematicians guessing the contents of boxes. Specifically, suppose there are $k$ unopened boxes each containing a ...
628 views

### Gini Coefficient and Renyi Entropy

Gini coefficient (aka Gini Index) is a quantity used in economics to describe income inequality. It is 0 for uniformly distributed income, and approaches 1 when all income is in hands of one ...
79 views

### order of convergence of the conditional entropy (3)

I'm sorry for having open two questions which have been solved by elementary counter-examples provided by @AnthonyQuas. Actually I'm not an expert in information theory and I expected that a positive ...
89 views

### order of convergence of the conditional entropy (2)

Let $X_n$ be a random variable distributed on $A_n:=\{1, \ldots, n\}$ and $g_n\colon A_n \to A_n$ such that $\Pr\big(X_n \neq g_n(X_n)\big) \to 0$. Putting $Y_n=g_n(X_n)$, then by Fano's inequality ...
676 views

### An Entropy Inequality (generalized)

Let $X,Y$ be probability measures on $\{1,2,\dots,n\}$. For $0\le \alpha \le 1$, set $K=\sum_i X(i)^\alpha Y(i)^{1-\alpha}$ so that $Z:=\frac{1}{K}X^\alpha Y^{1-\alpha}$ is also a probability ...
2k views

### An Entropy Inequality

Let $X,Y$ be probability measures on $\{1,2,\dots,n\}$, and set $K=\sum_i\sqrt{X(i)Y(i)}$ so that $Z:=\frac{1}{K}\sqrt{XY}$ is also a probability measure on $\{1,2,\dots,n\}$. How can we prove the ...
409 views

### Incremental entropy computation

After a quick internet search I found no method for incremental entropy computation. Question 1 Let $\{x_i\}_{i=1}^n$ and $\{x_i\}_{i=1+n}^{n+m}$ be two samples and let $S_i^j:=\sum_{k=i}^j x_k$. ...
524 views

### Entropy of edit distance

The edit or Levenshtein distance between two strings is the minimum number of single character insertions, deletions and substitutions to transform one string into another. If we take random binary ...
210 views

### Calculate channel capacity of general channel under constraint

Hi! Given a conditional distribution $P_{Y|X}$ I'd like to find the prior distribution $P_X$ that maximizes the mutual information $I(X;Y)$ with $P_Y(y)=\int P_{Y|X}(y|x)P_X(x)\text{dx}$ (this ...
724 views

### What's the maximum entropy probability distribution given bounds [a,b] and mean?

What is the continuous probability distribution that maximizes entropy, given only the bounds of the random variable [a,b] and the mean mu of the probability distribution? For example: if a=0, b=1, ...
251 views

### Maximum entropy priors in infinite dimensional spaces

Is there an extension of maximum entropy probability distributions for function spaces? For $\mathbb{R}^n$ and discrete spaces, there is much literature about this problem under names such as ...
154 views

### Ising entropy of a finite L_1 x L_2 lattice

We know the entropy per site of the 2-d Ising model from Onsager's solution. Has anybody also calculated the entropy for a finite rectangle of size L_1 x L_2 with periodic boundary conditions (i.e. on ...
574 views

### Bounding the entropy of a convolution

Say we have a function $f:\mathbb{Z}_2^n \to \mathbb{R}$, such that $\sum _{x\in \mathbb{Z}_2^n} f(x)^2 = 1$ (so we can think of $\{ f(x)^2\} _{x\in \mathbb{Z}_2^n}$ as a distribution). It is natural ...
52 views

### Information amount of fuzzy data transfer [closed]

Suppose we have binary channel from which we are able to receive zeroes and ones. We also know apriory probability $p$ of receiving "1". Then we can calculate information amount of each digit $q$ we ...
236 views

### Incremental computation of a conditional entropy

Is it possible to compute a conditional entropy (see the two following formulas) in an incremental manner ? That is, the sets C and K are not fix: each time we have a new element c, set K may increase ...
140 views

### asymptotic behaviour of the entropy and degeneracy

For each $n \in \mathbb{N}$ let $X_n$ be a random variable taking its values in a finite set $E_n$ with $P(X_n=x_n)>0$ for all $x_n \in E_n$. Say that $X_n$ is asymptotically degenerate if ...
375 views

### Low degree polynomial approximation for the entropy function

Let $X$ be a discrete random variable with possible values $\{x_1,\ldots,x_n\}$, and let $p$ denote the probability mass function of $X$. In addition, denote $p_i=p(x_i)$. The entropy of $X$ is ...
288 views

### Robust entropy-like measure for analyzing uncertainity

I'm looking for a measure to analysis the uncertainty observed in a set of variables (with multivariate Gaussian distribution). So, I've tried conventional Shanon entropy (differential entropy) which ...
621 views

### Calculate entropy for a set of data

Hi; I am not really a maths person and I have a question regarding shannon entropy. I have different datasets which only consists of three letter(I,N,M) such as : dataset1: {I,I,N,M,I,I} dataset2: ...
3k views

### How to compute KL-divergence when PMF contains 0s?

From the Wikipedia page on Kullback-Leibler divergence, the way to compute this metric is to utilize the following formula: The way I understand this is to compute the PMFs of two given sample sets ...
Consider the standard Ising model on $[0,N]^2$ for $N$ large. By that I mean the square-lattice Ising model without external field, inside an $N$-by-$N$ square. What is its entropy for $N$ large? It ...