# Advantages of a back-propagation neural network over other function approximation methods

Hello.

Let's say I have a set of input vectors $I = \{\mathbf{x_1}, \dots, \mathbf{x_k}\} \subset \mathcal{R}^m$ and a set of output vectors $O = \{\mathbf{y_1}, \dots, \mathbf{y_k}\} \subset \mathcal{R}^n$, and I want to obtain a mapping $f : \mathcal{R}^m \to \mathcal{R}^n$ such that

$$f(\mathbf{x_i}) = \mathbf{y_i} + \epsilon_i, \forall i \in \{1, \dots, k\}$$

where $\epsilon_i$ is small, and this mapping should be continuous at least around the input/output pairs.

There are many ways of doing so.

If we suppose that the input/output pairs won't change, what are the advantages of using an Artificial Neural Network over other methods to approximate functions?

EDIT: when I say advantages, I mean the practical advantages on the use of neural networks over other function approximation methods on any domain that use the neural networks.

For example, if we think of (the canonical example of) handwriting recognition that mailing services might use to read zip codes, the neural network is nothing more than a function that maps, let's say, [0, 1]^35 (if we think of a 5x7 grid, in which the values are the "intensity" or "amount" of the ink on each cell) to [0, 1]^10 (corresponding to each digit between 0 and 9, the value being the probability of the digit). So, in this case, if we think that we will write a software to do this recognition, and that the patterns will never change, we could simply have used another technique to map the input to the output. If we use any method that produces a continuous mapping, small "variations" on the handwritting wouldn't affect too much the output of the function.

Thanks.

-
What other methods of approximation did you have in mind? –  Gjergji Zaimi Dec 27 '09 at 1:18
What is a "mapping" for you? Linear? Differentiable? Continuous? –  Qiaochu Yuan Dec 27 '09 at 1:19
@Qiaochu Yuan: I've edited the question. I want a mapping that is at least continuous around the input/output pairs. –  Bruno Reis Dec 27 '09 at 1:29
I guess what I really want you to clarify is what you want f for. "Advantage" has no meaning except with respect to some application. –  Qiaochu Yuan Dec 27 '09 at 1:50
@Qiaochu Yuan: I've tried to clarify it a bit. –  Bruno Reis Dec 27 '09 at 2:04

Artificial neural networks are mainly useful when:

1. There is no information on the form of the function f(x) in advance and the task of specifying the functional form of f(x) from the data is computationally complex.
2. And, on the other hand there is a representative sample of inputs and outputs to be used as a training set.

The main advantage of the neural network method is that it can fairly approximate a large class of functions f(x). On the other hand if additional information on the function f(x) is known, then other estimation techniques are likely to work better. For example if it is known that f(x) is linear, then linear regression would surely give better estimation errors.

The problem presented in the question seems to be well suited to neural networks.

-

The nice thing about an ANN is that you can "overfit without overfitting". That is, you've got a huge parameter space but a good way of winnowing it down as well.

Also, regarding your particular example of handwriting numbers: you may be interested in Geoff Hinton's Google talk at http://www.youtube.com/watch?v=AyzOUbkUf3M

-

have you tried Vector Machines approaches ? There is a wonderful book on machine learning (by Trevor Hastie, Robert Tibshirani and Jerome Friedman) available online. It is definitely worth having a look at it.

-