There's many good ways to learn a distribution $p_X$ of an r.v. $X$ over $k$ symbols given many i.i.d. samples $X_1,\ldots, X_n$. The simplest is to use the sample relative frequencies $\hat{f}_X$ as your estimate of $p_X$, which is a reasonably alright way to learn the distribution.
But what if you're not sure about the samples, and are instead given a distribution for each sample, i.e. you're told $p_{X_1}, \ldots, p_{X_n}$. This might come up if each of the samples was measured noisily. You can assume the noise involved is independent and possibly differently distributed each time. How would you learn $p_X$ in this case? Bonus points if the procedure is efficient.
Things to note about the answer:
- in the case that the $p_{X_i}$'s are a just a deltas on particular symbols (i.e. perfect samples), the procedure should gracefully reduce to the case of perfect samples.
- as $n\to\infty$, the distribution you learn should go to $p_X$.
For my case, I actually want to find a solution to this problem in the continuous probability case (I'm starting with discrete so I can get somewhere). In particular, I want to find an analog to KDE. Here's a specific example problem: you want to learn a distribution using non-parametric methods. You're given 3 noisy length measurements from 3 different tools that make normally distributed errors: 4cm from a tool with $\sigma^2=2$, 5cm from a tool with $\sigma^2=3$, and 2cm from a tool with $\sigma^2=10$. What's your guess for what the distribution looks like? If these were exact samples, I would just plop a gaussian with appropriate bandwidth at each sample, but how should I take into account the distributions of each of the samples?
