Consider a set of states labeled $n=1,2,...$ in order of non-increasing probability $p(n)$. The standard Shannon argument gives meaning to the entropy $S$ of $p$ in terms of the number of states needed to encode the distribution with small error in the limit of many iid copies.
Roughly speaking I want to know how badly this can fail in the one-shot setting. My primary motivation is to understand how non-trivial it is to be able to show that $e^S$ states suffice to give small error for a given probability distribution. The distributions I have in mind have a system size-like parameter (like the number of iid copies) but the "copies" are correlated in a complicated way.
Given the subject matter this may be a very elementary question, but I cannot seem to find much information on it.
Fix a large number $S$. Let $p$ be a probability distribution with $p(n) \geq p(n+1)$ and entropy $S$, and let $p_S$ be the probability truncated to its first $e^S$ states i.e. $p_S(n \leq e^S) = p(n)$ and $p_S(n > e^S) = 0$. Is there a bound on the error $||p - p_S ||_1$ (varying $p$ with fixed $S$) or can I make this as close to one as I want?