Optimize an unknown function which can be evaluated only?

Question

Given an unknown function $f:\mathbb R^d \to \mathbb R$, we can evaluate its value at any point in its domain, but we don't have its expression. In other words, $f$ is like a black box to us.

What is the name for the problem of finding the minimizer of $f$? What are some methods out there?

What is the name for the problem of finding the solution to the equation $f(x)=0$? What are some methods out there?

In the above two problems, is it a good idea to interpolate or fit to some evaluations of f: $(x_i, f(x_i)), i=1, \dots, n$ using a function $g_\theta$ with known form and parameter $\theta$ to be determined, and then minimize $g_\theta$ or find its root?

Thanks and regards!

@chaohuang: There are two cases: you may or may not evaluate its gradient, depending on assumptions. — Tim, Apr 08 '13 at 00:58
If gradient is available, the tasks you're asking can be accomplished by gradient-based algorithms. For example, the minimum, or at least a local minimum, can be computed by steepest descent method, and the roots can be found by Newton's method. — chaohuang, Apr 08 '13 at 01:05
And if the gradient is unknown, there are metaheuristic methods, which are also called derivative-free or black-box methods and usually in the form of stochastic optimization. — chaohuang, Apr 08 '13 at 01:40
Do you know whether the function is smooth (even if you can't evaluate the gradient)? Do you know whether the function is convex? If it isn't convex, do you know whether or not it's at least Lipschitz continuous? If the function is completely general, then this is a hopeless problem. — Brian Borchers, Apr 08 '13 at 02:24
Your first question is addressed in this answer. For your second question, you could try a bisection method. — Christian Clason, Apr 08 '13 at 06:54
@ChristianClason: Note that this Question allows for higher dimension $d$ than 1, so it's not clear how bisection should apply. — hardmath, Apr 10 '13 at 04:44
@hardmath: There are multidimensional variants such as the one described in G. Wood, The bisection method in higher dimensions. — Christian Clason, Apr 10 '13 at 06:45
what if the global optimization problem is unbounded or we want to solve it like that? — , Feb 07 '14 at 14:56

score 15 · Accepted Answer · answered Apr 08 '13 at 02:03

15

The methods you are looking for -- i.e., that only use function evaluations but not derivatives -- are called derivative free optimization methods. There is a large body of literature on them, and you can find a chapter on such methods in most books on optimization. Typical approaches include

Approximating the gradient by finite differences if one can reasonably expect the function to be smooth and, possibly, convex;
Monte Carlo methods such as Simulated Annealing;
Genetic Algorithms.

answered Apr 08 '13 at 02:03

Wolfgang Bangerth

55,373
59
119

2

Can I just add "Surrogate Modelling" to that list? They are very applicable for black-box optimization, in particular if the function is costly to evaluate. – OscarB Apr 08 '13 at 08:13
Yes, you can :-) Certainly a great addition. – Wolfgang Bangerth Apr 09 '13 at 02:41
One could also use the Nelder-Mead method if good estimates of the optima are known. – J. M. May 19 '13 at 14:56
1

Yes, you could use Nelder-Mead, but it's a terrible algorithm compared to most every other one. – Wolfgang Bangerth May 20 '13 at 03:19
2

@WolfgangBangerth: Your comment on Nelder-Mead is valid only in dimension d>2. In two dimensions, it is on many problems an excellent and very hard to beat method. – Arnold Neumaier Sep 23 '13 at 15:49
I will believe that. Thanks for pointing it out. – Wolfgang Bangerth Sep 23 '13 at 16:58
method='powell' in SciPy is also derivative-free, but if you can formulate your problem in convex-optimization-problem, further using explicit derivative func will speed-up calculations/optimization – JeeyCi Dec 04 '23 at 09:00

score 2 · Answer 2 · answered May 19 '16 at 13:15

I think you should start with: GECCO Workshop on Real-Parameter Black-Box Optimization Benchmarking (BBOB 2016) http://numbbo.github.io/workshops/index.html

You will find many different algorithms that have been used in previous competitions, and that have been compared on a common basis. If you start elsewhere, you will soon drown in the hundreds of papers that claim their methods and algorithms perform better than others with little actual evidence for those claims.

Until recently, it was, to be frank, a disgraceful state of affairs and all power to INRIA, GECCO and many others for the effort they have made in establishing a framework for rational comparisons.

score -1 · Answer 3 · edited May 19 '16 at 19:15

-1

I'd just add that one of the keys here is being able to scale optimization method on multicore CPUs. If you can perform several function evaluations simultaneously, it gives you a speedup equal to a number of cores involved. Compare this to just using slightly more accurate response model, which makes you 10% more efficient or so.

I'd recommend to look at this code, it can be useful for people having access to many cores. A mathematics behind it is described in this paper.

edited May 19 '16 at 19:15

Community

1

answered May 18 '16 at 23:32

Paul

11

1

This answer is too short to be useful (and stay useful, since links may go away at any point). Also, please mention that you are the author of this software. – Christian Clason May 19 '16 at 06:50

Optimize an unknown function which can be evaluated only?

3 Answers3

Linked