Timeline for What is entropy, really?
Current License: CC BY-SA 3.0
18 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 1 at 8:59 | history | made wiki | Post Made Community Wiki by Stefan Kohl♦ | ||
| May 22, 2019 at 3:12 | comment | added | Andrés E. Caicedo | Lovely answer. (I just saw this question for the first time.) | |
| Oct 19, 2017 at 7:55 | vote | accept | Mustafa Said | ||
| Jun 17, 2017 at 17:35 | comment | added | Chris Jones | @PaulSiegel the exercise intends that $s$ is a function of $\mathbb{P}(E)$ rather than $E$. As you note, otherwise $s$ is not necessarily a multiple of $-\log \mathbb{P}(E)$. | |
| Feb 19, 2017 at 18:05 | comment | added | Paul Siegel | (My example above wasn't very good since $s(\{H\}) = s(\{T\})$ and $s$ is only well-defined up to a constant. A better example would be to set $P(\{H\}) = 1/4$ and $P(\{T\}) = 3/4$ - we expect $s(\{H\}) = \log 4$ and $s(\{T\}) = \log(4/3)$, but I don't see what goes wrong if we choose $s(\{T\}) = \log(2)$, for instance, instead.) | |
| Feb 19, 2017 at 5:34 | comment | added | Paul Siegel | In general I see no guarantee that a probability space should provide enough pairs of independent events to constrain the possible "surprise" functions on it all that much. Perhaps the right characterization involves looking at all finite probability spaces at once, though I'm not sure what that characterization would look like. | |
| Feb 19, 2017 at 5:30 | comment | added | Paul Siegel | I've encountered this characterization before, but there's something wrong. Consider the fair coin probability space $\{H, T\}$ with $P(\{H\}) = P(\{T\}) = 1/2$. Since $\{H, T\}$ is independent of $\{H\}$ and $\{T\}$ we must have $s(\{H, T\}) = 0$, and since $\emptyset$ is independent of $\{H\}$ and $\{T\}$ we must have $s(\emptyset) = \infty$ (monotonicity is also used here). But there are no other pairs of independent events, so we can't use the independence condition to fix the values of $s(\{H\})$ or $s(\{T\})$, and it seems to be that we can pick any positive value we want. | |
| Jul 24, 2014 at 7:22 | comment | added | Mustafa Said | @Qiaochu Yuan, thank you, I was just wondering because I really liked your explanation and I found a similar explanation in an undergraduate probability book (Ross). | |
| Jul 24, 2014 at 7:19 | comment | added | Qiaochu Yuan | @Mustafa: no, a friend of mine told me once that "surprise is negative log probability" and this seemed like the obvious conclusion to reach from that claim. | |
| Jul 24, 2014 at 7:12 | comment | added | Mustafa Said | @Qiaochu Yaun, I am wondering if your story originates from undergraduate books in probability, such as Ross. He also explains why the second property is intuitive. | |
| S Feb 26, 2014 at 17:52 | history | suggested | CommunityBot | CC BY-SA 3.0 |
Adding 'heads' to example removing 'coins' (implicit)
|
| Feb 26, 2014 at 17:50 | review | Suggested edits | |||
| S Feb 26, 2014 at 17:52 | |||||
| Nov 5, 2013 at 16:39 | comment | added | Suvrit | @Timothy: Actually, "surprise" is more natural if instead of the disjoint additivity above, we actually look at "submodularity" --- the amount of possible surprise diminishes based on what we've already seen---seems to be "natural", and axiomatically related to entropy (see e.g., related discussion also in: arxiv.org/abs/1002.4020 -- and these ideas can be then related to Kolmogorov complexity too... | |
| Nov 2, 2013 at 19:20 | comment | added | Timothy Chow | Using the term "information content" rather than "surprise" may make additivity seem more natural (even though this doesn't quite line up with standard terminology). | |
| Nov 1, 2013 at 14:50 | comment | added | Marek | Nice explanation. So we're reduced to how plausible we find the conditions. The first one seems fine but the second one seems a little arbitrary -- why not weighted sum of powers, for example? I believe there are generalizations of Shannon entropy breaking this condition. So what's missing in this answer is an explanation of why to expect additivity. This is the place where I'd say it's better to put your statistical-mechanics hat on, for then the additivity of entropy as a disorder in the system seems obvious. | |
| Nov 1, 2013 at 11:54 | comment | added | Jon Bannon | This is really nice, Qiaochu! | |
| Nov 1, 2013 at 0:43 | history | edited | Qiaochu Yuan | CC BY-SA 3.0 |
added 1 characters in body
|
| Nov 1, 2013 at 0:38 | history | answered | Qiaochu Yuan | CC BY-SA 3.0 |