Estimating parameters of a mixture of normal distributions. - MathOverflow most recent 30 from http://mathoverflow.net 2013-05-24T12:48:14Z http://mathoverflow.net/feeds/question/105101 http://www.creativecommons.org/licenses/by-nc/2.5/rdf http://mathoverflow.net/questions/105101/estimating-parameters-of-a-mixture-of-normal-distributions Estimating parameters of a mixture of normal distributions. Peter 2012-08-20T17:24:08Z 2012-08-20T21:49:49Z <p>I want to estimate the parameters $\mu_i$ and $\sigma^2_i$ of a countable mixture of Gaussians with assumed equal weights, variance and identically spaced means. I intially thought that the Fourier transforms of an iid sample from the mixture mentioned would give above another Gaussian in phase space, but when computed in matlab I get a sharp peak spectral distribution. I used a Gaussian windows function, so I don't understand why the power spectral density tends to be unbounded at zero phase.</p> http://mathoverflow.net/questions/105101/estimating-parameters-of-a-mixture-of-normal-distributions/105108#105108 Answer by Douglas Zare for Estimating parameters of a mixture of normal distributions. Douglas Zare 2012-08-20T19:19:56Z 2012-08-20T19:19:56Z <p>If the Gaussians are constrained to be of equal shapes and distances, then you have a scaled theta function or the convolution of a Gaussian with a sha (Ш) function or scaled Dirac comb. Ш is a sum of equally spaced delta functions, and the Fourier transform of Ш is another Ш function, usually with different spacings and amplitudes depending on your conventions. Since the Fourier transform of a Gaussian is a Gaussian, the Fourier transform of your function is a sum of equally spaced delta functions whose amplitudes sample a Gaussian density, something like $$\beta \sum_{n=-\infty}^\infty e^{-\pi^2 z^2/\sigma - 2 \pi i x_0 z}\delta(z-n/\alpha)$$ </p> <p>where the parameters $\alpha, \beta, \sigma,$ and $x_0$ are determined by the parameters of your distribution and your Fourier conventions. </p> <p>In particular, there is a delta function at $0$. You should expect this to happen when you have a periodic function whose average over a period isn't $0$.</p> <p>You can read off the spacing of the Gaussians from the reciprocal of the spacing between the delta functions of the Fourier transform, and the way the amplitude drops off determines the shape of the Gaussian.</p> <p>That said, finding the parameters of a finite Gaussian mixture is an extremely common problem when the components are not contrained to have the same shapes and spacings. This is because Gaussian mixtures are one of the standard models used in machine learning. Gaussians are localized, so there isn't a big difference between the sum of the Guassians with nearby centers and the whole function. So, in practice, if you have (noisy) values from your function then instead of taking the Fourier transform, you may want to use some of these tools, and then project the best fit onto the space of scaled theta functions. The EM (expectation maximization) algorithm is one common method, and you should be able to find many <a href="http://www.mathworks.com/matlabcentral/fileexchange/26184-em-algorithm-for-gaussian-mixture-model" rel="nofollow">Matlab implementations</a>.</p> http://mathoverflow.net/questions/105101/estimating-parameters-of-a-mixture-of-normal-distributions/105113#105113 Answer by Peter for Estimating parameters of a mixture of normal distributions. Peter 2012-08-20T20:39:09Z 2012-08-20T20:39:09Z <p>The description I gave was not precise but I'm not very fluent in mathematics; I've encountered the problem while working on an undergraduate thesis on neuronal coding. To specify, the sample contains a finite, unknown number of equally weighted components, but the full problem is periodic;i.e. the distribution is defined on the entire space $R^2$. </p> http://mathoverflow.net/questions/105101/estimating-parameters-of-a-mixture-of-normal-distributions/105121#105121 Answer by Peter for Estimating parameters of a mixture of normal distributions. Peter 2012-08-20T21:49:49Z 2012-08-20T21:49:49Z <p>Thank you for the answer, I posted the original question and decided to register in the meantime. My research requires that I estimate Cramer-Rao bound of the means, which can be done with the Matlab EM implementation for Gaussian mixtures. However, I do not know a priori how many components are contained in each sample, and am looking for a tool to help me decide on that or solve the problem otherwise. I have read that the Bayes Information Criterion can be used for this, but none of the papers I have read explain it clearly enough.</p>