Search Results

Suppose we have a probability distribution $P$ on a finite set $S$. We draw $N$ i.i.d. samples according to $P$ and use these samples to define an empirical distribution $R$. We measure the Kullback …

[Edit: for posterity, I'm adding two small comments to the code explaining how to fix it, in light of Iosef Pinelis' answer below. Look for "Should be:" to find the corrections.] Suppose we draw a sam …

These notes are just intended as an extended comment on Iosif Pinelis' answer. In my initial version of this post, I made a mistake, but Iosif pointed out the error in the comments below; it should n …

This isn't an answer; I'm just sharing plots from a few numerical experiments. Each time we repeat the above process for $N$ steps, we generate a (potentially) different multiset (i.e., a different s …

What about the symmetric-ized KL divergence? $$D(p,q) + D(q,p)$$ Recalling that $$D(p,q) = \sum_x p(x) \log \left(\frac{p(x)}{q(x)}\right)$$ then the $\log(p/q)$ is the order of magnitude of the ratio …

Depending on the shape of the domain $\mathcal{X}$ and how $N$ (the number of points) scales with $n$ (the dimension), plain-old rejection sampling might suffice. Specifically, sample uniformly from …

Suppose I have $N$ coins, where coin $i$ has probability $p_i$ of coming up heads. I flip all $N$ coins and let $S_N$ be the number of heads. How can I compute the distribution of $S_N$ efficiently? …

Kostya_I is correct, of course, but taking the closure under convolution can sometimes be a bit ugly. If you're interested in a simple, parametric family, consider the hyperbolic secant distribution. …