Timeline for Profiles of very high dimensional functions
Current License: CC BY-SA 4.0
9 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 3, 2018 at 8:01 | comment | added | Dirk | I would non suspect that the observation of "local minima do not cause trouble" in neural networks is due to a generic feature of the class of high-dimensional functions, but that it also depends on some (probably generic) features of training data that is used and maybe even on the stochastic optimization methods. I would guess that any neural network architecture admits an "adversarial data set" (and objective) such that gradient descent gets caught in a bad minimum. If the same could be true under stochastic gradient descent, I would not try to guess. | |
| Jul 3, 2018 at 4:57 | comment | added | Gerhard Paseman | If we consider a class of bounded functions on a compact domain, we can organize by number of local minima for example. If you then apply a probability measure on the space, you can choose how to select a random function. For neural networks that "perform gradient descent", the performance on those with few local minima will likely be successful, and you can measure that by analyzing existing attempts. For the classes with more local minima, you can record the success/fail rate there, and determine a threshold of success. Gerhard "Not Your Idea Of Random" Paseman, 2018.07.02. | |
| Jul 3, 2018 at 4:43 | comment | added | Igor Rivin | @GerhardPaseman In what sense is this class of functions random? | |
| Jul 3, 2018 at 4:22 | comment | added | Gerhard Paseman | If neural net creators fail on this task, and then fail on minor variations, then it is an example of a subclass of random functions which resist this approach, and challenge a reasonable explanation of why this works in other cases. Gerhard "It Shows What Shouldn't Happen" Paseman, 2018.07.02. | |
| Jul 3, 2018 at 4:17 | comment | added | Igor Rivin | @GerhardPaseman That won't prove or disprove anything (unless it actually finds something new...) | |
| Jul 3, 2018 at 4:15 | comment | added | Gerhard Paseman | Have a neural network try to find Hadamard matrices of order less than 100. If it finds one not isomorphic to a known one, that counts as an initial success and one then can try to go for an unknown order (668 or near that). Mild variations on this should present a substantial challenge. Gerhard "Has Lots Of Local Minima" Paseman, 2018.07.02. | |
| Jul 3, 2018 at 2:30 | comment | added | R Hahn | I'm not sure it is directly relevant to your question, but one thing to bear in mind is that the representation neural networks optimize over is presumably vastly over-parametrized relative to the actual measured inputs, so that several local minima might correspond to the same or very similar functions of the actual input variables. It is hard to say if this helps (in exploring the function space) or hurts (in creating lots of spurious local minima). | |
| Jul 3, 2018 at 1:52 | comment | added | Piyush Grover | "physics of glassy systems" or protein folding has similar problems, so you maybe able to find mathematical physics type approaches in that domain. I also note that the theorists in machine learning have several hypothesis in regard, e.g. see arxiv.org/abs/1412.0233 that uses random matrix theory, or recent work arxiv.org/pdf/1804.06561.pdf for a PDE (mean-field homogenization/propagation of chaos) approach. | |
| Jul 3, 2018 at 1:08 | history | asked | Igor Rivin | CC BY-SA 4.0 |