Return to Answer

The physicist Edwin T. Jaynes wrote a lot about entropy and its uses in physics and probability theory:

http://bayes.wustl.edu/etj/node1.html some published works

http://bayes.wustl.edu/etj/node2.html some unpublished works

He insisted that the two are very much related. In one of his works he says he had a conversation with Eugene Wigner in the 1950s in which he told Wigner that physical entropy is a measure of information and Wigner thought that was absurd, because the information one person possesses differs from that of another, whereas entropy can be measured with thermometers and calorimeters. Jaynes says he didn't know at the time how to explain to Wigner why that was wrong, but figured it out much later. As nearly as I understand it at this moment, Jaynes thought entropy measures the amount of information delivered by those thermometers and calorimeters.

Jaynes has a sort of cult following, many members of which are professors in the physical sciences who learned everything they know about probability from Jaynes, and they can sometimes come across a bit like religious fanatics.

OK, a short answer: The entropy in a discrete probability distribution is $\displaystyle\sum_k -p_k\log p_k$, where $\{p_k\}$ are the probabilities assigned to atoms in the probability space. The base of the logarithm can be any number $>1$ (or maybe $<1$ in some cases? I haven't thought about that.). There's also relative entropy of one probablity measure $p$ with respect to another, $q$, given by $\displaystyle\sum_k p_k\log(p_k/q_k)$ and cross entropy given by $\displaystyle\sum_k -p_k\log q_k$.

Via Google, you might be able to find notes taken by scribes in Gian-Carlo Rota's probability course at MIT. Somewhere among those, he says the resemblance between entropy of discrete probability distributions (as defined above) and entropy of continuous distributions (done similarly with integrals.) is only superficial. These observations are explored for finite discrete distributions in Section 1.4 of the book "Combinatorics The Rota Way" by Kung, Rota, and Yan. Deeper relationships are speculated upon in Rota's article "Twelve problems in probability no one likes to bring up" in the book "Algebraic Combinatorics and Computer Science", Crapo and Senato eds. Those speculations are in the section entitled "Problem 4: entropy."

The physicist Edwin T. Jaynes wrote a lot about entropy and its uses in physics and probability theory:

http://bayes.wustl.edu/etj/node1.html some published works

http://bayes.wustl.edu/etj/node2.html some unpublished works

He insisted that the two are very much related. In one of his works he says he had a conversation with Eugene Wigner in the 1950s in which he told Wigner that physical entropy is a measure of information and Wigner thought that was absurd, because the information one person possesses differs from that of another, whereas entropy can be measured with thermometers and calorimeters. Jaynes says he didn't know at the time how to explain to Wigner why that was wrong, but figured it out much later. As nearly as I understand it at this moment, Jaynes thought entropy measures the amount of information delivered by those thermometers and calorimeters.

Jaynes has a sort of cult following, many members of which are professors in the physical sciences who learned everything they know about probability from Jaynes, and they can sometimes come across a bit like religious fanatics.

OK, a short answer: The entropy in a discrete probability distribution is $\displaystyle\sum_k -p_k\log p_k$, where $\{p_k\}$ are the probabilities assigned to atoms in the probability space. The base of the logarithm can be any number $>1$ (or maybe $<1$ in some cases? I haven't thought about that.). There's also relative entropy of one probablity measure $p$ with respect to another, $q$, given by $\displaystyle\sum_k p_k\log(p_k/q_k)$ and cross entropy given by $\displaystyle\sum_k -p_k\log q_k$.

Via Google, you might be able to find notes taken by scribes in Gian-Carlo Rota's probability course at MIT. Somewhere among those, he says the resemblance between entropy of discrete probability distributions (as defined above) and entropy of continuous distributions (done similarly with integrals.) is only superficial. These observations are explored for finite discrete distributions in Section 1.4 of the book "Combinatorics The Rota Way" by Kung, Rota, and Yan. Deeper relationships are speculated upon in Rota's article "Twelve problems in probability no one likes to bring up" in the book Algebraic Combinatorics and Computer Science, Crapo and Senato eds. Those speculations are in the section entitled "Problem 4: entropy."

The physicist Edwin T. Jaynes wrote a lot about entropy and its uses in physics and probability theory:

http://bayes.wustl.edu/etj/node1.html some published works

http://bayes.wustl.edu/etj/node2.html some unpublished works

He insisted that the two are very much related. In one of his works he says he had a conversation with Eugene Wigner in the 1950s in which he told Wigner that physical entropy is a measure of information and Wigner thought that was absurd, because the information one person possesses differs from that of another, whereas entropy can be measured with thermometers and calorimeters. Jaynes says he didn't know at the time how to explain to Wigner why that was wrong, but figured it out much later. As nearly as I understand it at this moment, Jaynes thought entropy measures the amount of information delivered by those thermometers and calorimeters.

Jaynes has a sort of cult following, many members of which are professors in the physical sciences who learned everything they know about probability from Jaynes, and they can sometimes come across a bit like religious fanatics.

OK, a short answer: The entropy in a discrete probability distribution is $\displaystyle\sum_k -p_k\log p_k$, where $\{p_k\}$ are the probabilities assigned to atoms in the probability space. The base of the logarithm can be any number $>1$ (or maybe $<1$ in some cases? I haven't thought about that.). There's also relative entropy of one probablity measure $p$ with respect to another, $q$, given by $\displaystyle\sum_k p_k\log(p_k/q_k)$ and cross entropy given by $\displaystyle\sum_k -p_k\log q_k$.

Via Google, you might be able to find notes taken by scribes in Gian-Carlo Rota's probability course at MIT. Somewhere among those, he says the resemblance between entropy of discrete probability distributions (as defined above) and entropy of continuous distributions (done similarly with integrals.) is only superficial.

The physicist Edwin T. Jaynes wrote a lot about entropy and its uses in physics and probability theory:

http://bayes.wustl.edu/etj/node1.html some published works

http://bayes.wustl.edu/etj/node2.html some unpublished works

He insisted that the two are very much related. In one of his works he says he had a conversation with Eugene Wigner in the 1950s in which he told Wigner that physical entropy is a measure of information and Wigner thought that was absurd, because the information one person possesses differs from that of another, whereas entropy can be measured with thermometers and calorimeters. Jaynes says he didn't know at the time how to explain to Wigner why that was wrong, but figured it out much later. As nearly as I understand it at this moment, Jaynes thought entropy measures the amount of information delivered by those thermometers and calorimeters.

Jaynes has a sort of cult following, many members of which are professors in the physical sciences who learned everything they know about probability from Jaynes, and they can sometimes come across a bit like religious fanatics.

OK, a short answer: The entropy in a discrete probability distribution is $\displaystyle\sum_k -p_k\log p_k$, where $\{p_k\}$ are the probabilities assigned to atoms in the probability space. The base of the logarithm can be any number $>1$ (or maybe $<1$ in some cases? I haven't thought about that.). There's also relative entropy of one probablity measure $p$ with respect to another, $q$, given by $\displaystyle\sum_k p_k\log(p_k/q_k)$ and cross entropy given by $\displaystyle\sum_k -p_k\log q_k$.

Via Google, you might be able to find notes taken by scribes in Gian-Carlo Rota's probability course at MIT. Somewhere among those, he says the resemblance between entropy of discrete probability distributions (as defined above) and entropy of continuous distributions (done similarly with integrals.) is only superficial. These observations are explored for finite discrete distributions in Section 1.4 of the book "Combinatorics The Rota Way" by Kung, Rota, and Yan. Deeper relationships are speculated upon in Rota's article "Twelve problems in probability no one likes to bring up" in the book "Algebraic Combinatorics and Computer Science", Crapo and Senato eds. Those speculations are in the section entitled "Problem 4: entropy."

The physicist Edwin T. Jaynes wrote a lot about entropy and its uses in physics and probability theory:

http://bayes.wustl.edu/etj/node1.html some published works

http://bayes.wustl.edu/etj/node2.html some unpublished works

He insisted that the two are very much related. In one of his works he says he had a conversation with Eugene Wigner in the 1950s in which he told Wigner that physical entropy is a measure of information and Wigner thought that was absurd, because the information one person possesses differs from that of another, whereas entropy can be measured with thermometers and calorimeters. Jaynes says he didn't know at the time how to explain to Wigner why that was wrong, but figured it out much later. As nearly as I understand it at this moment, Jaynes thought entropy measures the amount of information delivered by those thermometers and calorimeters.

Jaynes has a sort of cult following, many members of which are professors in the physical sciences who learned every thing they know about probability from Jaynes, and they can sometimes come across a bit like religious fanatics.

OK, a short answer: The entropy in a discrete probability distribution is $\displaystyle\sum_k -p_k\log p_k$, where $\{p_k\}$ are the probabilities assigned to atoms in the probability space. The base of the logarithm can be any number $>1$ (or maybe $<1$ in some cases? I haven't thought about that.). There's also relative entropy of one probablity measure $p$ with respect to another, $q$, given by $\displaystyle\sum_k p_k\log(p_k/q_k)$ and cross entropy given by $\displaystyle\sum_k -p_k\log q_k$.

Via Google, you might be able to find notes taken by scribes in Gian-Carlo Rota's probability course at MIT. Somewhere among those, he says the resemblance between entropy of discrete probability distributions (as defined above) and entropy of continuous distributions (done similarly with integrals.) is only superficial.

The physicist Edwin T. Jaynes wrote a lot about entropy and its uses in physics and probability theory:

http://bayes.wustl.edu/etj/node1.html some published works

http://bayes.wustl.edu/etj/node2.html some unpublished works

He insisted that the two are very much related. In one of his works he says he had a conversation with Eugene Wigner in the 1950s in which he told Wigner that physical entropy is a measure of information and Wigner thought that was absurd, because the information one person possesses differs from that of another, whereas entropy can be measured with thermometers and calorimeters. Jaynes says he didn't know at the time how to explain to Wigner why that was wrong, but figured it out much later. As nearly as I understand it at this moment, Jaynes thought entropy measures the amount of information delivered by those thermometers and calorimeters.

Jaynes has a sort of cult following, many members of which are professors in the physical sciences who learned everything they know about probability from Jaynes, and they can sometimes come across a bit like religious fanatics.

OK, a short answer: The entropy in a discrete probability distribution is $\displaystyle\sum_k -p_k\log p_k$, where $\{p_k\}$ are the probabilities assigned to atoms in the probability space. The base of the logarithm can be any number $>1$ (or maybe $<1$ in some cases? I haven't thought about that.). There's also relative entropy of one probablity measure $p$ with respect to another, $q$, given by $\displaystyle\sum_k p_k\log(p_k/q_k)$ and cross entropy given by $\displaystyle\sum_k -p_k\log q_k$.

Via Google, you might be able to find notes taken by scribes in Gian-Carlo Rota's probability course at MIT. Somewhere among those, he says the resemblance between entropy of discrete probability distributions (as defined above) and entropy of continuous distributions (done similarly with integrals.) is only superficial.