Skip to main content
Post Undeleted by kodlu
added 614 characters in body
Source Link
kodlu
  • 10.4k
  • 2
  • 36
  • 55

Let $X_n$ haveSince the pmf supported on $\{0,1,\ldots,n-1\}$ given by $$ (2^{-c},q,\ldots,q) $$ whereOP may be interested in what $q=(1-2^{-c})/(n-1)$.$\epsilon$ are achievable I am providing this alternative to the other answer, where it is correctly stated:

If you fix the maximal atom (say, $p$) of a distribution $\mu$ supported by $n$ points, then its entropy is maximal when all the remaining atoms have the same weight $(1-p)/(n-1)$.

Take $n$ large enough soAlso note that $$ p\geq \frac{1-p}{n-1} \iff np\geq 1\iff p\geq \frac{1}{n} $$ so that $n-1>2^c$ implying $2^{-c}$$p$ is indeed the maximal probability and thus fixingatom of $H_{min}.$$\mu.$

EvaluatingThis means that one actually obtains the equality below for the Shannon entropy gives: $$ H=2^{-c} c + (1-2^{-c}) [\log (n-1)-\log (1-2^{-c})] $$$$ H(\mu) = -p\log p - (1-p)\log(1-p) + (1-p)\log(n-1), $$ and some standard approximations can tell you how large anwhen $n$$p$ is neededfixed which gives $$ H(\mu) = H_2(p) + \left(1-p\right)\log(n-1) $$ or $$ H(\mu) \geq \left(1-p\right)\log(n-1)\sim \left(1-p\right)\log n \quad (1) $$ where $f(n)\sim g(n)$ denotes that $\lim_{n\rightarrow \infty} \frac{f(n)}{g(n)}=1.$ This means that for a given $\epsilon$ in your lower bound$n$ large enough you can pick any $\epsilon \geq p$ for which $$ H(\mu)\geq (1-\epsilon) \log n $$ is indeed satisfied.

Depending on Shannon entropywhat application you have in mind, this may suffice.

Let $X_n$ have the pmf supported on $\{0,1,\ldots,n-1\}$ given by $$ (2^{-c},q,\ldots,q) $$ where $q=(1-2^{-c})/(n-1)$.

Take $n$ large enough so that $n-1>2^c$ implying $2^{-c}$ is the maximal probability and thus fixing $H_{min}.$

Evaluating Shannon entropy gives $$ H=2^{-c} c + (1-2^{-c}) [\log (n-1)-\log (1-2^{-c})] $$ and some standard approximations can tell you how large an $n$ is needed for a given $\epsilon$ in your lower bound on Shannon entropy.

Since the OP may be interested in what $\epsilon$ are achievable I am providing this alternative to the other answer, where it is correctly stated:

If you fix the maximal atom (say, $p$) of a distribution $\mu$ supported by $n$ points, then its entropy is maximal when all the remaining atoms have the same weight $(1-p)/(n-1)$.

Also note that $$ p\geq \frac{1-p}{n-1} \iff np\geq 1\iff p\geq \frac{1}{n} $$ so that $p$ is indeed the maximal atom of $\mu.$

This means that one actually obtains the equality below for the Shannon entropy: $$ H(\mu) = -p\log p - (1-p)\log(1-p) + (1-p)\log(n-1), $$ when $p$ is fixed which gives $$ H(\mu) = H_2(p) + \left(1-p\right)\log(n-1) $$ or $$ H(\mu) \geq \left(1-p\right)\log(n-1)\sim \left(1-p\right)\log n \quad (1) $$ where $f(n)\sim g(n)$ denotes that $\lim_{n\rightarrow \infty} \frac{f(n)}{g(n)}=1.$ This means that for $n$ large enough you can pick any $\epsilon \geq p$ for which $$ H(\mu)\geq (1-\epsilon) \log n $$ is indeed satisfied.

Depending on what application you have in mind, this may suffice.

Post Deleted by kodlu
Source Link
kodlu
  • 10.4k
  • 2
  • 36
  • 55

Let $X_n$ have the pmf supported on $\{0,1,\ldots,n-1\}$ given by $$ (2^{-c},q,\ldots,q) $$ where $q=(1-2^{-c})/(n-1)$.

Take $n$ large enough so that $n-1>2^c$ implying $2^{-c}$ is the maximal probability and thus fixing $H_{min}.$

Evaluating Shannon entropy gives $$ H=2^{-c} c + (1-2^{-c}) [\log (n-1)-\log (1-2^{-c})] $$ and some standard approximations can tell you how large an $n$ is needed for a given $\epsilon$ in your lower bound on Shannon entropy.