A cross post from on AI StackExchange.
So I have this function let call her $F:[0,1]^n \rightarrow \mathbb{R}$ and say $10 \le n \le 100$. I want to find some $x_0 \in [0,1]^n$ such that $F(x_0)$ is as small as possible. I don't think there is any hope of getting the global minimum. I just want a reasonably good $x_0$.
AFAIK the standard approach is to run an (accelerated) gradient decent a bunch of times and take the best result. But in my case values of $F$ are computed algorithmically and I don't have a way to compute gradients for $F$.
So I want to do something like this.
(A) We create a neural network which takes an $n$-dimensional vector as input and returns a real number as result. We want the NN to "predict" values of $F$ but at this point it is untrained.
(B) We take bunch of random points in $[0,1]^n$. We compute values of $F$ at those points. And we train NN using this data.
(C1) Now the neural net provides us with a reasonably smooth function $F_1:[0,1]^n \rightarrow \mathbb{R}$ approximating $F$. We run a gradient decent a bunch of times on $F_1$. We take the final points of those decent and compute $F$ on them to see if we caught any small values. Then we take whole paths of those gradient decent, compute $F$ on them and use this as data to retrain our neural net.
(C2) The retrained neural net provides us with a new function $F_2$ and we repeat the previous step
(C3) ...
Does this approach have a name? Is it used somewhere? Should I indeed use neural nets or there are better ways of constructing smooth approximations for my needs?
