hyperparameter_hunter.optimization.backends.skopt package

Submodules

hyperparameter_hunter.optimization.backends.skopt.engine module

This module contains various modified SKOpt assets that are used to support the other hyperparameter_hunter.optimization.backends.skopt modules

Notes

Many of the tools defined herein (although substantially modified) are based on those provided by the excellent [Scikit-Optimize](https://github.com/scikit-optimize/scikit-optimize) library. See hyperparameter_hunter.optimization.backends.skopt for a copy of SKOpt’s license.

What follows is a record of the first few commits to this file in order to clearly define what code was taken from the original Scikit-Optimize source, and how it was modified thereafter.

  • 81a70ddfa0270495f0ed39127adbac4eb1f4fa59: The content of this module (less module docstring) is identical to SKOpt’s module skopt.optimizer.optimizer at the time of SKOpt commit 6740876a6f9ad92c732d394e8534a5236a8d3f84

  • 744043d09f11cf90609cbef6ca8ab43515958feb: Add SKOpt’s skopt.utils.cook_estimator at the time of the above SKOpt commit, as well as the original import statements required by the function

  • XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: [Diverging from SKOpt] Fix broken imports, and (substantially) refactor code and documentation to follow HH conventions or for readability - Changes on and after this point are originally authored by the contributors of HyperparameterHunter and are, therefore, subject to the HyperparameterHunter license

class hyperparameter_hunter.optimization.backends.skopt.engine.Optimizer(dimensions, base_estimator='gp', n_initial_points=10, acq_func='gp_hedge', acq_optimizer='auto', random_state=None, acq_func_kwargs=None, acq_optimizer_kwargs=None, warn_on_re_ask=False)

Bases: object

Run bayesian optimisation loop

An Optimizer represents the steps of a bayesian optimisation loop. To use it you need to provide your own loop mechanism. The various optimisers provided by skopt use this class under the hood. Use this class directly if you want to control the iterations of your bayesian optimisation loop

Parameters
dimensions: List

List of shape (n_dims,) containing search space dimensions. Each search dimension can be defined as any of the following:

  • Instance of a Dimension object (Real, Integer or Categorical)

  • (<lower_bound>, <upper_bound>) tuple (for Real or Integer dimensions)

  • (<lower_bound>, <upper_bound>, <prior>) tuple (for Real dimensions)

  • List of categories (for Categorical dimensions)

base_estimator: {SKLearn Regressor, “GP”, “RF”, “ET”, “GBRT”, “DUMMY”}, default=”GP”

If not string, should inherit from sklearn.base.RegressorMixin. In addition, the predict method should have an optional return_std argument, which returns std(Y | x), along with E[Y | x].

If base_estimator is a string in {“GP”, “RF”, “ET”, “GBRT”, “DUMMY”}, a surrogate model corresponding to the relevant X_minimize function is created

n_initial_points: Int, default=10

Number of evaluations of func with initialization points before approximating it with base_estimator. Points provided as x0 count as initialization points. If len(x0) < n_initial_points, additional points are sampled at random

acq_func: {“LCB”, “EI”, “PI”, “gp_hedge”, “EIps”, “PIps”}, default=”gp_hedge”

Function to minimize over the posterior distribution. Can be any of the following:

  • “LCB”: Lower confidence bound

  • “EI”: Negative expected improvement

  • “PI”: Negative probability of improvement

  • “gp_hedge”: Probabilistically choose one of the above three acquisition functions at every iteration

    • The gains g_i are initialized to zero

    • At every iteration,

      • Each acquisition function is optimised independently to propose a candidate point X_i

      • Out of all these candidate points, the next point X_best is chosen by softmax(eta g_i)

      • After fitting the surrogate model with (X_best, y_best), the gains are updated such that g_i -= mu(X_i)

  • “EIps”: Negated expected improvement per second to take into account the function compute time. Then, the objective function is assumed to return two values, the first being the objective value and the second being the time taken in seconds

  • “PIps”: Negated probability of improvement per second. The return type of the objective function is identical to that of “EIps”

acq_optimizer: {“sampling”, “lbfgs”, “auto”}, default=”auto”

Method to minimize the acquisition function. The fit model is updated with the optimal value obtained by optimizing acq_func with acq_optimizer

  • “sampling”: acq_func is optimized by computing acq_func at n_initial_points randomly sampled points.

  • “lbfgs”: acq_func is optimized by

    • Randomly sampling n_restarts_optimizer (from acq_optimizer_kwargs) points

    • “lbfgs” is run for 20 iterations with these initial points to find local minima

    • The optimal of these local minima is used to update the prior

  • “auto”: acq_optimizer is configured on the basis of the base_estimator and the search space. If the space is Categorical or if the provided estimator is based on tree-models, then this is set to “sampling”

random_state: Int, or RandomState instance (optional)

Set random state to something other than None for reproducible results

acq_func_kwargs: Dict (optional)

Additional arguments to be passed to the acquisition function.

acq_optimizer_kwargs: Dict (optional)

Additional arguments to be passed to the acquisition optimizer

warn_on_re_ask: Boolean, default=False

If True, and the internal optimizer recommends a point that has already been evaluated on invocation of ask, a warning is logged before recommending a random point. Either way, a random point is used instead of already-evaluated recommendations. However, logging the fact that this has taken place can be useful to indicate that the optimizer may be stalling, especially if it repeatedly recommends the same point. In these cases, if the suggested point is not optimal, it can be helpful to switch a different OptPro (especially DummyOptPro), which will suggest points using different criteria

Attributes
Xi: List

Points at which objective has been evaluated

yi: List

Values of objective at corresponding points in Xi

models: List

Regression models used to fit observations and compute acquisition function

space: `hyperparameter_hunter.space.space_core.Space`

Stores parameter search space used to sample points, bounds, and type of parameters

n_initial_points_: Int

Original value passed through the n_initial_points kwarg. The value of this attribute remains unchanged along the lifespan of Optimizer, unlike _n_initial_points

_n_initial_points: Int

Number of remaining points that must be evaluated before fitting a surrogate estimator and using it to recommend incumbent search points. Initially, _n_initial_points is set to the value of the n_initial_points kwarg, like n_initial_points_. However, _n_initial_points is decremented for each point tell-ed to Optimizer

Methods

ask(self[, n_points, strategy])

Request point (or points) at which objective should be evaluated next

copy(self[, random_state])

Create a shallow copy of an instance of the optimizer

run(self, func[, n_iter])

Execute ask() + tell() loop for n_iter iterations

tell(self, x, y[, fit])

Record an observation (or several) of the objective function
property base_estimator
property acq_optimizer

Method to minimize the acquisition function. See documentation for the acq_optimizer kwarg in Optimizer.__init__() for additional information

Returns
{“lbfgs”, “sampling”}

String in {“lbfgs”, “sampling”}. If originally “auto”, one of the two aforementioned strings is selected based on base_estimator

ask(self, n_points=None, strategy='cl_min')

Request point (or points) at which objective should be evaluated next

Parameters
n_points: Int (optional)

Number of points returned by the ask method. If n_points not given, a single point to evaluate is returned. Otherwise, a list of points to evaluate is returned of size n_points. This is useful if you can evaluate your objective in parallel, and thus obtain more objective function evaluations per unit of time

strategy: {“cl_min”, “cl_mean”, “cl_max”}, default=”cl_min”

Method used to sample multiple points if n_points is an integer. If n_points is not given, strategy is ignored.

If set to “cl_min”, then “Constant Liar” strategy (see reference) is used with lie objective value being minimum of observed objective values. “cl_mean” and “cl_max” correspond to the mean and max of values, respectively.

With this strategy, a copy of optimizer is created, which is then asked for a point, and the point is told to the copy of optimizer with some fake objective (lie), the next point is asked from copy, it is also told to the copy with fake objective and so on. The type of lie defines different flavours of “cl…” strategies

Returns
List

Point (or points) recommended to be evaluated next

References

1

Chevalier, C.; Ginsbourger, D.: “Fast Computation of the Multi-points Expected Improvement with Applications in Batch Selection”. https://hal.archives-ouvertes.fr/hal-00732512/document

tell(self, x, y, fit=True)

Record an observation (or several) of the objective function

Provide values of the objective function at points suggested by ask(), or arbitrary points. By default, a new model will be fit to all observations. The new model is used to suggest the next point at which to evaluate the objective. This point can be retrieved by calling ask().

To add multiple observations in a batch, pass a list-of-lists for x, and a list of scalars for y

Parameters
x: List, or list-of-lists

Point(s) at which objective was evaluated

y: Scalar, or list

Value(s) of objective at x

fit: Boolean, default=True

Whether to fit a model to observed evaluations of the objective. A model will only be fitted after n_initial_points points have been tell-ed to the optimizer, irrespective of the value of fit. To add observations without fitting a new model, set fit to False

copy(self, random_state=None)

Create a shallow copy of an instance of the optimizer

Parameters
random_state: Int, or RandomState instance (optional)

Set random state of the copy

Returns
Optimizer

Shallow copy of self

run(self, func, n_iter=1)

Execute ask() + tell() loop for n_iter iterations

Parameters
func: Callable

Function that returns the objective value y, when given a search point x

n_iter: Int, default=1

Number of ask/tell sequences to execute

Returns
OptimizeResult

scipy.optimize.OptimizeResult instance

hyperparameter_hunter.optimization.backends.skopt.engine.is_list_like(x)

Determine whether a point is list-like

Parameters
x: List

Some point to check for list-likeness

Returns
Boolean

True if x is list-like. Else False

hyperparameter_hunter.optimization.backends.skopt.engine.is_2d_list_like(x)

Determine whether a point is 2-dimensional list-like

Parameters
x: List

Some point to check for 2D list-likeness

Returns
Boolean

True if x is 2D list-like. Else False

hyperparameter_hunter.optimization.backends.skopt.engine.check_x_in_space(x, space)

Check that an arbitrary point, or list of points, fits within the bounds of space

Parameters
x: List

Some point (or list of points), whose compatibility with space will be checked. If x is a collection of multiple points, it should be a list of lists

space: Space

Instance of hyperparameter_hunter.space.space_core.Space that defines the dimensions and bounds within which x should fit

Raises
ValueError

If x is incompatible with space for any reason

hyperparameter_hunter.optimization.backends.skopt.engine.cook_estimator(base_estimator, space=None, **kwargs)

Cook a default estimator

For the special base_estimator called “DUMMY”, the return value is None. This corresponds to sampling points at random, hence there is no need for an estimator

Parameters
base_estimator: {SKLearn Regressor, “GP”, “RF”, “ET”, “GBRT”, “DUMMY”}, default=”GP”

If not string, should inherit from sklearn.base.RegressorMixin. In addition, the predict method should have an optional return_std argument, which returns std(Y | x), along with E[Y | x].

If base_estimator is a string in {“GP”, “RF”, “ET”, “GBRT”, “DUMMY”}, a surrogate model corresponding to the relevant X_minimize function is created

space: `hyperparameter_hunter.space.space_core.Space`

Required only if the base_estimator is a Gaussian Process. Ignored otherwise

**kwargs: Dict

Extra parameters provided to the base_estimator at initialization time

Returns
SKLearn Regressor

Regressor instance cooked up according to base_estimator and kwargs

hyperparameter_hunter.optimization.backends.skopt.protocols module

This module defines the OptPro (Optimization Protocol) classes that are intended for direct use. All classes defined herein should be descendants of one of the base classes defined in hyperparameter_hunter.optimization.protocol_core

Related

hyperparameter_hunter.optimization.protocol_core

Defines the base Optimization Protocol classes from which the classes in hyperparameter_hunter.optimization.backends.skopt.protocols are descendants

class hyperparameter_hunter.optimization.backends.skopt.protocols.BayesianOptPro(target_metric=None, iterations=1, verbose=1, read_experiments=True, reporter_parameters=None, warn_on_re_ask=False, base_estimator='GP', n_initial_points=10, acquisition_function='gp_hedge', acquisition_optimizer='auto', random_state=32, acquisition_function_kwargs=None, acquisition_optimizer_kwargs=None, n_random_starts='DEPRECATED', callbacks=None, base_estimator_kwargs=None)

Bases: hyperparameter_hunter.optimization.protocol_core.SKOptPro

Bayesian optimization with Gaussian Processes

Attributes
search_space_size

The number of different hyperparameter permutations possible given the current

source_script

Methods

forge_experiment(self, model_initializer[, …])

Define hyperparameter search scaffold for building Experiments during optimization

get_ready(self)

Prepare for optimization by finalizing hyperparameter space and identifying similar Experiments.

go(self[, force_ready])

Execute hyperparameter optimization, building an Experiment for each iteration

set_dimensions(self)

Locate given hyperparameters that are space choice declarations and add them to dimensions

set_experiment_guidelines(self, \*args, …)

Deprecated since version 3.0.0a2.
source_script = None
class hyperparameter_hunter.optimization.backends.skopt.protocols.GradientBoostedRegressionTreeOptPro(target_metric=None, iterations=1, verbose=1, read_experiments=True, reporter_parameters=None, warn_on_re_ask=False, base_estimator='GBRT', n_initial_points=10, acquisition_function='EI', acquisition_optimizer='sampling', random_state=32, acquisition_function_kwargs=None, acquisition_optimizer_kwargs=None, n_random_starts='DEPRECATED', callbacks=None, base_estimator_kwargs=None)

Bases: hyperparameter_hunter.optimization.protocol_core.SKOptPro

Sequential optimization with gradient boosted regression trees

Attributes
search_space_size

The number of different hyperparameter permutations possible given the current

source_script

Methods

forge_experiment(self, model_initializer[, …])

Define hyperparameter search scaffold for building Experiments during optimization

get_ready(self)

Prepare for optimization by finalizing hyperparameter space and identifying similar Experiments.

go(self[, force_ready])

Execute hyperparameter optimization, building an Experiment for each iteration

set_dimensions(self)

Locate given hyperparameters that are space choice declarations and add them to dimensions

set_experiment_guidelines(self, \*args, …)

Deprecated since version 3.0.0a2.
source_script = None
class hyperparameter_hunter.optimization.backends.skopt.protocols.RandomForestOptPro(target_metric=None, iterations=1, verbose=1, read_experiments=True, reporter_parameters=None, warn_on_re_ask=False, base_estimator='RF', n_initial_points=10, acquisition_function='EI', acquisition_optimizer='sampling', random_state=32, acquisition_function_kwargs=None, acquisition_optimizer_kwargs=None, n_random_starts='DEPRECATED', callbacks=None, base_estimator_kwargs=None)

Bases: hyperparameter_hunter.optimization.protocol_core.SKOptPro

Sequential optimization with random forest regressor decision trees

Attributes
search_space_size

The number of different hyperparameter permutations possible given the current

source_script

Methods

forge_experiment(self, model_initializer[, …])

Define hyperparameter search scaffold for building Experiments during optimization

get_ready(self)

Prepare for optimization by finalizing hyperparameter space and identifying similar Experiments.

go(self[, force_ready])

Execute hyperparameter optimization, building an Experiment for each iteration

set_dimensions(self)

Locate given hyperparameters that are space choice declarations and add them to dimensions

set_experiment_guidelines(self, \*args, …)

Deprecated since version 3.0.0a2.
source_script = None
class hyperparameter_hunter.optimization.backends.skopt.protocols.ExtraTreesOptPro(target_metric=None, iterations=1, verbose=1, read_experiments=True, reporter_parameters=None, warn_on_re_ask=False, base_estimator='ET', n_initial_points=10, acquisition_function='EI', acquisition_optimizer='sampling', random_state=32, acquisition_function_kwargs=None, acquisition_optimizer_kwargs=None, n_random_starts='DEPRECATED', callbacks=None, base_estimator_kwargs=None)

Bases: hyperparameter_hunter.optimization.protocol_core.SKOptPro

Sequential optimization with extra trees regressor decision trees

Attributes
search_space_size

The number of different hyperparameter permutations possible given the current

source_script

Methods

forge_experiment(self, model_initializer[, …])

Define hyperparameter search scaffold for building Experiments during optimization

get_ready(self)

Prepare for optimization by finalizing hyperparameter space and identifying similar Experiments.

go(self[, force_ready])

Execute hyperparameter optimization, building an Experiment for each iteration

set_dimensions(self)

Locate given hyperparameters that are space choice declarations and add them to dimensions

set_experiment_guidelines(self, \*args, …)

Deprecated since version 3.0.0a2.
source_script = None
class hyperparameter_hunter.optimization.backends.skopt.protocols.DummyOptPro(target_metric=None, iterations=1, verbose=1, read_experiments=True, reporter_parameters=None, warn_on_re_ask=False, base_estimator='DUMMY', n_initial_points=10, acquisition_function='EI', acquisition_optimizer='sampling', random_state=32, acquisition_function_kwargs=None, acquisition_optimizer_kwargs=None, n_random_starts='DEPRECATED', callbacks=None, base_estimator_kwargs=None)

Bases: hyperparameter_hunter.optimization.protocol_core.SKOptPro

Random search by uniform sampling

Attributes
search_space_size

The number of different hyperparameter permutations possible given the current

source_script

Methods

forge_experiment(self, model_initializer[, …])

Define hyperparameter search scaffold for building Experiments during optimization

get_ready(self)

Prepare for optimization by finalizing hyperparameter space and identifying similar Experiments.

go(self[, force_ready])

Execute hyperparameter optimization, building an Experiment for each iteration

set_dimensions(self)

Locate given hyperparameters that are space choice declarations and add them to dimensions

set_experiment_guidelines(self, \*args, …)

Deprecated since version 3.0.0a2.
source_script = None
hyperparameter_hunter.optimization.backends.skopt.protocols.GBRT

alias of hyperparameter_hunter.optimization.backends.skopt.protocols.GradientBoostedRegressionTreeOptPro

hyperparameter_hunter.optimization.backends.skopt.protocols.RF

alias of hyperparameter_hunter.optimization.backends.skopt.protocols.RandomForestOptPro

hyperparameter_hunter.optimization.backends.skopt.protocols.ET

alias of hyperparameter_hunter.optimization.backends.skopt.protocols.ExtraTreesOptPro

class hyperparameter_hunter.optimization.backends.skopt.protocols.BayesianOptimization(**kwargs)

Bases: hyperparameter_hunter.optimization.backends.skopt.protocols.BayesianOptPro

Deprecated since version 3.0.0a2: Will be removed in 3.2.0. Renamed to BayesianOptPro

Attributes
search_space_size

The number of different hyperparameter permutations possible given the current

source_script

Methods

forge_experiment(self, model_initializer[, …])

Define hyperparameter search scaffold for building Experiments during optimization

get_ready(self)

Prepare for optimization by finalizing hyperparameter space and identifying similar Experiments.

go(self[, force_ready])

Execute hyperparameter optimization, building an Experiment for each iteration

set_dimensions(self)

Locate given hyperparameters that are space choice declarations and add them to dimensions

set_experiment_guidelines(self, \*args, …)

Deprecated since version 3.0.0a2.
source_script = None
class hyperparameter_hunter.optimization.backends.skopt.protocols.GradientBoostedRegressionTreeOptimization(**kwargs)

Bases: hyperparameter_hunter.optimization.backends.skopt.protocols.GradientBoostedRegressionTreeOptPro

Deprecated since version 3.0.0a2: Will be removed in 3.2.0. Renamed to GradientBoostedRegressionTreeOptPro

Attributes
search_space_size

The number of different hyperparameter permutations possible given the current

source_script

Methods

forge_experiment(self, model_initializer[, …])

Define hyperparameter search scaffold for building Experiments during optimization

get_ready(self)

Prepare for optimization by finalizing hyperparameter space and identifying similar Experiments.

go(self[, force_ready])

Execute hyperparameter optimization, building an Experiment for each iteration

set_dimensions(self)

Locate given hyperparameters that are space choice declarations and add them to dimensions

set_experiment_guidelines(self, \*args, …)

Deprecated since version 3.0.0a2.
source_script = None
class hyperparameter_hunter.optimization.backends.skopt.protocols.RandomForestOptimization(**kwargs)

Bases: hyperparameter_hunter.optimization.backends.skopt.protocols.RandomForestOptPro

Deprecated since version 3.0.0a2: Will be removed in 3.2.0. Renamed to RandomForestOptPro

Attributes
search_space_size

The number of different hyperparameter permutations possible given the current

source_script

Methods

forge_experiment(self, model_initializer[, …])

Define hyperparameter search scaffold for building Experiments during optimization

get_ready(self)

Prepare for optimization by finalizing hyperparameter space and identifying similar Experiments.

go(self[, force_ready])

Execute hyperparameter optimization, building an Experiment for each iteration

set_dimensions(self)

Locate given hyperparameters that are space choice declarations and add them to dimensions

set_experiment_guidelines(self, \*args, …)

Deprecated since version 3.0.0a2.
source_script = None
class hyperparameter_hunter.optimization.backends.skopt.protocols.ExtraTreesOptimization(**kwargs)

Bases: hyperparameter_hunter.optimization.backends.skopt.protocols.ExtraTreesOptPro

Deprecated since version 3.0.0a2: Will be removed in 3.2.0. Renamed to ExtraTreesOptPro

Attributes
search_space_size

The number of different hyperparameter permutations possible given the current

source_script

Methods

forge_experiment(self, model_initializer[, …])

Define hyperparameter search scaffold for building Experiments during optimization

get_ready(self)

Prepare for optimization by finalizing hyperparameter space and identifying similar Experiments.

go(self[, force_ready])

Execute hyperparameter optimization, building an Experiment for each iteration

set_dimensions(self)

Locate given hyperparameters that are space choice declarations and add them to dimensions

set_experiment_guidelines(self, \*args, …)

Deprecated since version 3.0.0a2.
source_script = None
class hyperparameter_hunter.optimization.backends.skopt.protocols.DummySearch(**kwargs)

Bases: hyperparameter_hunter.optimization.backends.skopt.protocols.DummyOptPro

Deprecated since version 3.0.0a2: Will be removed in 3.2.0. Renamed to DummyOptPro

Attributes
search_space_size

The number of different hyperparameter permutations possible given the current

source_script

Methods

forge_experiment(self, model_initializer[, …])

Define hyperparameter search scaffold for building Experiments during optimization

get_ready(self)

Prepare for optimization by finalizing hyperparameter space and identifying similar Experiments.

go(self[, force_ready])

Execute hyperparameter optimization, building an Experiment for each iteration

set_dimensions(self)

Locate given hyperparameters that are space choice declarations and add them to dimensions

set_experiment_guidelines(self, \*args, …)

Deprecated since version 3.0.0a2.
source_script = None

Module contents