Abstract:

Bayesian optimization (BO) is a data-efficient approach for optimizing expensive-to-evaluate black-box functions that suffer from noisy evaluations. Traditional BO algorithms ignore the relationship between consecutive input values, which is known to lead to significant “jumps” in the search space that cannot be implemented in practice, especially in online experimental systems. For example, in performance-driven control applications, large changes in the chosen setpoint parameters may trigger fail-safe mechanisms or lead to violation of critical safety constraints. In such applications, it is necessary to limit the allowable search space at each BO iteration, which can be done by incorporating local search constraints into the original problem setting. In this paper, we show how this novel BO setting can be cast as a Markov decision process (MDP) for which the optimal policy is characterized by an intractable dynamic programming (DP) problem. To overcome this challenge, we take advantage of approximate DP methods, particularly rollout with fast policy search, to derive an efficient multi-step lookahead BO policy. We also propose a novel base policy needed for the rollout algorithm, which explicitly incorporates the local search restrictions in an efficient and intuitive manner. Lastly, we empirically show that our proposed multi-step lookahead BO policy outperforms existing methods on a well-known benchmark problem.