Entropy difference dominance of sequences

Question

Consider a collection of positive integers $\{a_i\}_{i=1}^m$ and the distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$.

Similarly for the collection $\{a_i\}_{i=1}^{m+1}$ form the distribution $q_i=\frac{a_i}{\sum_{i=1}^{m+1}a_i}$, for the collection $\{a_i\}_{i=1}^{n}$ form the distribution $r_i=\frac{a_i}{\sum_{i=1}^{n}a_i}$ and for the collection $\{a_i\}_{i=1}^{n+1}$ form the distribution $s_i=\frac{a_i}{\sum_{i=1}^{n+1}a_i}$.

Let $m+1\leq n$. Note all collections are subsets of the collection for the distribution $s$.

For the cases $(1)$ $a_{i+1}=a_i+\theta(\log^ka_i)$, $(2)$ $a_{i+1}=a_i+\theta(a_i)$ where $k$ is a postive constant, is it true that the difference in entropy is strictly decreasing? That is, is $H(q)-H(p)>H(s)-H(r)$ valid? When can one expect difference in Shannon entropy to be dominant in sequences of these types? Clearly in this post Entropy dominance, one criteria for negative result is given. Can $a_i$ be any larger or smaller than in the cases given?

---

Can we guess $\lim_{n\rightarrow\infty}H(s)=\infty$ in all these cases?

What spacing in $a_i$ would guarantee $\lim_{n\rightarrow\infty}H(s)<c\in\Bbb N$?

Answer 1

Here's some examplary evidence that maybe the answers are Yes.

• For Case 2, let $a_{i+1}=2a_i$ for all $i$ with $a_1=1$. Then the binary entropy is $$ -\frac1{2^{n}-1} \sum_{k=0}^{n-1} 2^k \log_2\left(\frac{2^k}{2^{n}-1}\right)$$ $$ \approx-\frac1{2^{n}} \sum_{k=0}^{n-1} 2^k \log_2\left(\frac{2^k}{2^{n}}\right)$$ $$ =\sum_{k=0}^{n-1} \frac{n-k}{2^{n-k}}=\frac{2^{n+1}-n-2}{2^n}\nearrow 2$$ which is the entropy of the geometric distribution with parameter $1/2$. So it seems that in this case yes, the difference in entropy is decreasing. A similar calculation woks if $a_{i+1}=u\cdot a_i$ for a positive integer $u>2$.

• For Case 1, well let's go all the way to the extreme case $a_i=1$ for all $i$. In this case the entropy is just $\log n$ and so by concavity of log, yes, the difference in entropy is decreasing.