Efficient Bayesian Hyperparameter Optimization

Download or read book Efficient Bayesian Hyperparameter Optimization written by Aaron Klein and published by . This book was released on 2020 with total page pages. Available in PDF, EPUB and Kindle. Book excerpt: Abstract: Automated machine learning emerged as a new research field inside of machine learning that tries to progressively automate different steps of common machine learning pipelines which are traditionally executed by humans. One of its core tasks is the automated search for the right hyperparameters of a given machine learning algorithm which in practice is often essential to achieve good performance. Compare to other optimization problems, hyperparameter optimization is usually particularly expensive, since in each iteration, it requires to train and validate the underlying algorithm. One of the most successful approaches for hyperparameter optimization is Bayesian optimization. At its core, Bayesian optimization fits a probabilistic model of the objective function, which together with an additional acquisition function is used to guide the search towards the global optimum. In this thesis we present several extensions to standard Bayesian optimization to improve its performance for hyperparameter optimization problems. First, we introduce a new probabilistic model based on Bayesian neural networks, that allows to model the performance of hyperparameter configurations across different tasks and thereby scales much better with the number of data points and dimensions than Gaussian processes which are traditionally used inside Bayesian optimization. In hyperparameter optimization, often approximations, so-called fidelities, of the objective function are available which are much cheaper to evaluate. We present two new Bayesian optimization methods that can leverage such fidelities, such as learning curves or dataset subsets, to improve the overall search process in terms of wall-clock time by orders of magnitude. Furthermore, based on our proposed Bayesian neural network model, we present a new neural network architecture which models the learning curve of iterative machine learning methods, such as neural networks. Finally, due to the high computational cost of hyperparameter optimization, thorough benchmarking and evaluation of new developed methods is often prohibitively expensive. We show that one can approximate continuous and discrete benchmarks by surrogate benchmarks that capture the characteristics of the original benchmark but take only milliseconds to evaluate. This allows us to performa rigorous analysis and comparison of various different hyperparameter optimization methods from the literature