Advantages of a back-propagation neural network over other function approximation methods

Question

Hello.

Let's say I have a set of input vectors $I = \{\mathbf{x_1}, \dots, \mathbf{x_k}\} \subset \mathcal{R}^m$ and a set of output vectors $O = \{\mathbf{y_1}, \dots, \mathbf{y_k}\} \subset \mathcal{R}^n$, and I want to obtain a mapping $f : \mathcal{R}^m \to \mathcal{R}^n$ such that

$$ f(\mathbf{x_i}) = \mathbf{y_i} + \epsilon_i, \forall i \in \{1, \dots, k\}$$

where $\epsilon_i$ is small, and this mapping should be continuous at least around the input/output pairs.

There are many ways of doing so.

If we suppose that the input/output pairs won't change, what are the advantages of using an Artificial Neural Network over other methods to approximate functions?

EDIT: when I say advantages, I mean the practical advantages on the use of neural networks over other function approximation methods on any domain that use the neural networks.

For example, if we think of (the canonical example of) handwriting recognition that mailing services might use to read zip codes, the neural network is nothing more than a function that maps, let's say, [0, 1]^35 (if we think of a 5x7 grid, in which the values are the "intensity" or "amount" of the ink on each cell) to [0, 1]^10 (corresponding to each digit between 0 and 9, the value being the probability of the digit). So, in this case, if we think that we will write a software to do this recognition, and that the patterns will never change, we could simply have used another technique to map the input to the output. If we use any method that produces a continuous mapping, small "variations" on the handwritting wouldn't affect too much the output of the function.

Thanks.

Answer 1

Artificial neural networks are mainly useful when:

1. There is no information on the form of the function f(x) in advance and the task of specifying the functional form of f(x) from the data is computationally complex.
2. And, on the other hand there is a representative sample of inputs and outputs to be used as a training set.

The main advantage of the neural network method is that it can fairly approximate a large class of functions f(x). On the other hand if additional information on the function f(x) is known, then other estimation techniques are likely to work better. For example if it is known that f(x) is linear, then linear regression would surely give better estimation errors.

The problem presented in the question seems to be well suited to neural networks.

Answer 2

The nice thing about an ANN is that you can "overfit without overfitting". That is, you've got a huge parameter space but a good way of winnowing it down as well.

Also, regarding your particular example of handwriting numbers: you may be interested in Geoff Hinton's Google talk at http://www.youtube.com/watch?v=AyzOUbkUf3M

Answer 3

have you tried Vector Machines approaches ? There is a wonderful book on machine learning (by Trevor Hastie, Robert Tibshirani and Jerome Friedman) available online. It is definitely worth having a look at it.