2
$\begingroup$

Latent Dirichlet allocation - is quite a popular topic in data-mining. Wikepedia mentions thousands citations in few years.

Question 0 Can one give some digest for a math minded person of the key ideas ?

Question 1 Is there non-trivial mathematical result behind ?

Question 2 What practical problem does it help to solve ?

Question 3 How the two questions above are related ?

Ideally I would be happy to see some theorem which would be important for some practical problem.

Improve this question
$\endgroup$
1
  • $\begingroup$ I have a question. I have been working on the anomalies in industrial control systems and I want to model a multi-agent system using LDA model based on particle filter sampling. I want to know how can I use LDA model in such systems. Do you have any information or any reference regarding to this issue? than you $\endgroup$
    – Elham
    Commented Oct 10, 2018 at 16:16

1 Answer 1

Reset to default
5
$\begingroup$

Q1: De Finetti's theorem, that if the set of random variables $\{x_1,x_2,\ldots x_N \}$ is exchangeable, meaning their joint distribution $P(x_1,x_2,\ldots x_N)$ if invariant under permutation, then it can be represented as a mixture for some random variable $\theta$: $$P(x_1,x_2,\ldots x_N)=\int d\theta\,p(\theta)\prod_{i=1}^N p(x_i|\theta).$$ The latent variable $\theta$ is the "L" from LDA. (The "D" refers to the, somewhat arbitrary, choice of a Dirichlet distribution for the $p$'s.)
De Finetti's theorem allows one to treat the joint distribution of words and topics in a document as conditionally independent and identically distributed with respect to an underlying "latent" parameter of a probability distribution. As the authors of LDA state in their 2003 paper, "Conditionally, the joint distribution of the random variables is simple and factored while marginally over the latent parameter, the joint distribution can be quite complex."

The basic assumption that allows this efficient representation is the "bag-of-words" assumption, that the order of words in a document can be neglected when one tries to associate words with topics.

I guess this also answers Q2 and Q3.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ It might be worth mentioning that the heart of this answer (which I think is completely accurate) applies equally well to many probability models that do not go by the name LDA and describes the Bayesian approach to statistics more generally. The answer to a modified Q1 "Is there any new (circa 2003) non-trivial mathematical result behind LDA?" the answer is probably 'no'. $\endgroup$
    – R Hahn
    Commented Jul 5, 2018 at 23:16
  • $\begingroup$ @RHahn Thank you for you comment. Can you give a comment why these papers and LDA considered to be so important ? $\endgroup$
    – Alexander Chervov
    Commented Jul 6, 2018 at 7:18
  • $\begingroup$ @AlexanderChervov Hard to say exactly. Some combination of 1) computational feasibility of the implementation (definitely not true in de Finetti's time), 2) good marketing/branding, 3) reaching a new audience (computer scientists reading JMLR) and 4) the contemporary relevance of the applied problem (text classification in the internet age). Regarding branding, LDA was actually a "taken" term in traditional statistics, referring to a classification method invented by Fisher. en.wikipedia.org/wiki/Linear_discriminant_analysis ("Not to be confused with...") $\endgroup$
    – R Hahn
    Commented Jul 6, 2018 at 19:52
  • 1
    $\begingroup$ and in physics LDA is the en.wikipedia.org/wiki/Local-density_approximation $\endgroup$
    – Carlo Beenakker
    Commented Jul 6, 2018 at 20:29
  • $\begingroup$ @RHahn thank you ! Please take a look at: datascience.stackexchange.com/questions/34083/… $\endgroup$
    – Alexander Chervov
    Commented Jul 10, 2018 at 22:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .