I am working on a project where I want to analyze the limits of information (for concreteness, let's just say Shannon information) that can be efficiently extracted from machine learning techniques built on top of feature selection.
For example, one problem of interest might be trying to learn a grammar for face recognition via machine learning methods built on top of the extraction of histogram of oriented gradient (HoG) features. Since such a grammar is not PAC-learnable, there must be some way to formally show why you get diminishing returns by just continuing to add layers upon layers of machine learning on top of feature extraction.
That is, why couldn't you efficiently learn a face grammar by combining classification schemes that analyze HoG features? The amount of data needed to learn the grammar must be intractably large in some size measure of the grammar, or else the complexity of the layers of machine learning must make it intractable.
Note that I'm not saying that state of the art methods wouldn't try to use this layered machine learning approach. For applications, the best known solution might be to scale up your computing resources so that you can train on exponentially much data. My goal is to show that this process has to hit a complexity ceiling sooner or later.
Are there any suggestions for how to begin such an analysis. Right now the whole thing seems very vague to me. Does anyone know of literature that explore these kinds of information limits to machine learning?
