To begin, there is a family of results which are sometimes referred to as "No free lunch" theorems. Each of these results, in their own way, asserts that any optimization algorithm is just as good as any other if you average over the space of all optimization problems. On the other hand, we know that in specific domains some algorithms vastly outperform all others (that we're aware of) - for detecting objects in images, convolutional neural networks are state of the art, and in computational linguistics the best you can do for most tasks is a neural network with an LSTM or transformer architecture. In both of these cases, the state of the art algorithms perform vastly better than, say, logistic regression.

How can we reconcile the "No free lunch" theorems with our empirical experience? The answer has to be that object detection in images and standard NLP tasks aren't "typical" optimization problems - some combination of the data and the task has some special structure which particular neural architectures are unusually good at detecting. What is this structure? Why are known algorithms so good at learning it? Can we generate new algorithms (neural or otherwise) that are even better?

These are all essentially math problems, sitting somewhere at the intersection of optimization theory and information theory. They are pretty wide open - except in simple cases like logistic regression - there isn't much in the way of theory which characterizes an algorithm as optimal for a particular task among the space of all possible optimization algorithms. An influential paper from 2014 proposes to use the theory of renormalization groups from physics to tackle this question, and there are other attempts using gauge theory or the principle of maximum entropy. Another line of attack involve the so-called "manifold hypothesis", which asserts that real world data sets (presented as a set of points in euclidean space) tend to cluster near a high codimension submanifold.

That's my answer for your main technical question, but I'll also make a remark about the academic politics. There are significantly more job openings in data science than there are people to fill them, so much so that many companies (like AirBnB) have found it cheaper and easier to start internal data science training programs than hire outside people. This problem is not likely to go away any time soon, so it's sensible to incentivize universities to start degree programs in the field, even if it's not yet a fully fleshed-out academic discipline yet. This has plenty of historical precedent - for instance, academic programs in forensic science and financial mathematics sprouted in the same way in the 1990's.

To begin, there is a family of results which are sometimes referred to as "No free lunch" theorems. Each of these results, in their own way, asserts that any optimization algorithm is just as good as any other if you average over the space of all optimization problems. On the other hand, we know that in specific domains some algorithms vastly outperform all others (that we're aware of) - for detecting objects in images, convolutional neural networks are state of the art, and in computational linguistics the best you can do for most tasks is a neural network with an LSTM or transformer architecture. In both of these cases, the state of the art algorithms perform vastly better than, say, logistic regression.

How can we reconcile the "No free lunch" theorems with our empirical experience? The answer has to be that object detection in images and standard NLP tasks aren't "typical" optimization problems - some combination of the data and the task has some special structure which particular neural architectures are unusually good at detecting. What is this structure? Why are known algorithms so good at learning it? Can we generate new algorithms (neural or otherwise) that are even better?

These are all essentially math problems, sitting somewhere at the intersection of optimization theory and information theory. They are pretty wide open - except in simple cases like logistic regression - there isn't much in the way of theory which characterizes an algorithm as optimal for a particular task among the space of all possible optimization algorithms. An influential paper from 2014 proposes to use the theory of renormalization groups from physics to tackle this question, and there are other attempts using gauge theory or the principle of maximum entropy. Another line of attack involves the so-called "manifold hypothesis", which asserts that real world data sets (presented as a set of points in euclidean space) tend to cluster near a high codimension submanifold.

That's my answer for your main technical question, but I'll also make a remark about the academic politics. There are significantly more job openings in data science than there are people to fill them, so much so that many companies (like AirBnB) have found it cheaper and easier to start internal data science training programs than hire outside people. This problem is not likely to go away any time soon, so it's sensible to incentivize universities to start degree programs in the field, even if it's not yet a fully fleshed-out academic discipline. This has plenty of historical precedent - for instance, academic programs in forensic science and financial mathematics sprouted in the same way in the 1990's.