hyperparameter_hunter.callbacks package

Submodules

hyperparameter_hunter.callbacks.aggregators module

class hyperparameter_hunter.callbacks.aggregators.AggregatorTimes

Bases: hyperparameter_hunter.callbacks.bases.BaseAggregatorCallback

Methods

on_exp_end(self)

Perform tasks when an Experiment ends

on_exp_start(self)

Perform tasks when an Experiment is started

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase

stat_aggregates: dict = None
on_exp_start(self)

Perform tasks when an Experiment is started

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_exp_end(self)

Perform tasks when an Experiment ends

class hyperparameter_hunter.callbacks.aggregators.AggregatorEvaluations

Bases: hyperparameter_hunter.callbacks.bases.BaseAggregatorCallback

Methods

on_exp_end(self)

Perform tasks when an Experiment ends

on_exp_start(self)

Perform tasks when an Experiment is started

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase

stat_aggregates: dict = None
last_evaluation_results: dict = None
on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_exp_end(self)

Perform tasks when an Experiment ends

class hyperparameter_hunter.callbacks.aggregators.AggregatorOOF

Bases: hyperparameter_hunter.callbacks.bases.BaseAggregatorCallback

Methods

on_exp_end(self)

Perform tasks when an Experiment ends

on_exp_start(self)

Perform tasks when an Experiment is started

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase

class hyperparameter_hunter.callbacks.aggregators.AggregatorHoldout

Bases: hyperparameter_hunter.callbacks.bases.BaseAggregatorCallback

Methods

on_exp_end(self)

Perform tasks when an Experiment ends

on_exp_start(self)

Perform tasks when an Experiment is started

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase

class hyperparameter_hunter.callbacks.aggregators.AggregatorTest

Bases: hyperparameter_hunter.callbacks.bases.BaseAggregatorCallback

Methods

on_exp_end(self)

Perform tasks when an Experiment ends

on_exp_start(self)

Perform tasks when an Experiment is started

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase

class hyperparameter_hunter.callbacks.aggregators.AggregatorLosses

Bases: hyperparameter_hunter.callbacks.bases.BaseAggregatorCallback

Methods

on_exp_end(self)

Perform tasks when an Experiment ends

on_exp_start(self)

Perform tasks when an Experiment is started

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase

hyperparameter_hunter.callbacks.bases module

This module defines the base callback classes, from which all other callback classes in hyperparameter_hunter.callbacks are descendants. Importantly, the specific base callback classes contained herein are all descendants of hyperparameter_hunter.callbacks.bases.BaseCallback, ensuring all callbacks descend from the same base class. This module also defines hyperparameter_hunter.callbacks.bases.lambda_callback(), which can be used to define custom callbacks to be executed during Experiments when passed to hyperparameter_hunter.environment.Environment.__init__() via the experiment_callbacks argument

hyperparameter_hunter.callbacks.evaluators module

This module defines Evaluator callbacks to score predictions generated during the different time divisions of the BaseExperiment by invoking hyperparameter_hunter.metrics.ScoringMixIn.evaluate()

Related

hyperparameter_hunter.metrics

Defines ScoringMixIn, which is inherited by BaseExperiment, and provides the evaluate method that is called by the classes in evaluators

Notes

Regarding evaluation when G.Env.save_transformed_metrics is False, target data will be either fold (for on_run_end/on_fold_end) or d (for on_rep_end/on_exp_end). Prediction data used for evaluation in this case does not follow this abnormal pattern. Target data is limited to either the fold or d data_chunks when G.Env.save_transformed_metrics is False because targets for run and rep are identical to the targets for fold and d, respectively. This is still the case even if performing inverse target transformation via EngineerStep. Because the target values do not change between these two pairs of divisions, their values may be unset, so the targets for the division immediately above are used instead. As noted in hyperparameter_hunter.data.data_chunks.target_chunks, both itself and hyperparameter_hunter.callback.wranglers.target_wranglers are concerned only with transformed targets–not with original targets (or inverted targets). That is because original targets and inverted targets should be identical. Original targets are updated only in hyperparameter_hunter.experiments.BaseExperiment.on_exp_start() (through hyperparameter_hunter.data.data_core.BaseDataset initialization) and in hyperparameter_hunter.experiments.BaseCVExperiment.on_fold_start()

class hyperparameter_hunter.callbacks.evaluators.EvaluatorOOF

Bases: hyperparameter_hunter.callbacks.bases.BaseEvaluatorCallback

Methods

on_exp_end(self)

Evaluate final (run/repetition-averaged) out-of-fold predictions

on_exp_start(self)

Perform tasks when an Experiment is started

on_fold_end(self)

Evaluate (run-averaged) out-of-fold predictions for the fold

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_rep_end(self)

Evaluate (run-averaged) out-of-fold predictions for the repetition

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_run_end(self)

Evaluate out-of-fold predictions for the run

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase

data_oof: BaseDataset = None
validation_index: list = None
on_run_end(self)

Evaluate out-of-fold predictions for the run

on_fold_end(self)

Evaluate (run-averaged) out-of-fold predictions for the fold

on_rep_end(self)

Evaluate (run-averaged) out-of-fold predictions for the repetition

on_exp_end(self)

Evaluate final (run/repetition-averaged) out-of-fold predictions

class hyperparameter_hunter.callbacks.evaluators.EvaluatorHoldout

Bases: hyperparameter_hunter.callbacks.bases.BaseEvaluatorCallback

Methods

on_exp_end(self)

Evaluate final (run/repetition-averaged) holdout predictions

on_exp_start(self)

Perform tasks when an Experiment is started

on_fold_end(self)

Evaluate (run-averaged) holdout predictions for the fold

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_rep_end(self)

Evaluate (run-averaged) holdout predictions for the repetition

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_run_end(self)

Evaluate holdout predictions for the run

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase

data_holdout: BaseDataset = None
on_run_end(self)

Evaluate holdout predictions for the run

on_fold_end(self)

Evaluate (run-averaged) holdout predictions for the fold

on_rep_end(self)

Evaluate (run-averaged) holdout predictions for the repetition

on_exp_end(self)

Evaluate final (run/repetition-averaged) holdout predictions

hyperparameter_hunter.callbacks.loggers module

class hyperparameter_hunter.callbacks.loggers.LoggerFitStatus

Bases: hyperparameter_hunter.callbacks.bases.BaseLoggerCallback

Methods

on_exp_end(self)

Perform tasks when an Experiment ends

on_exp_start(self)

Perform tasks when an Experiment is started

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase

stat_aggregates: dict = None
last_evaluation_results: dict = None
current_seed: int = None
float_format = '{:.5f}'
log_separator = ' | '
on_exp_start(self)

Perform tasks when an Experiment is started

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_exp_end(self)

Perform tasks when an Experiment ends

class hyperparameter_hunter.callbacks.loggers.LoggerOOF

Bases: hyperparameter_hunter.callbacks.bases.BaseLoggerCallback

Methods

on_exp_end(self)

Perform tasks when an Experiment ends

on_exp_start(self)

Perform tasks when an Experiment is started

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase

class hyperparameter_hunter.callbacks.loggers.LoggerHoldout

Bases: hyperparameter_hunter.callbacks.bases.BaseLoggerCallback

Methods

on_exp_end(self)

Perform tasks when an Experiment ends

on_exp_start(self)

Perform tasks when an Experiment is started

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase

class hyperparameter_hunter.callbacks.loggers.LoggerTest

Bases: hyperparameter_hunter.callbacks.bases.BaseLoggerCallback

Methods

on_exp_end(self)

Perform tasks when an Experiment ends

on_exp_start(self)

Perform tasks when an Experiment is started

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase

class hyperparameter_hunter.callbacks.loggers.LoggerEvaluation

Bases: hyperparameter_hunter.callbacks.bases.BaseLoggerCallback

Methods

on_exp_end(self)

Perform tasks when an Experiment ends

on_exp_start(self)

Perform tasks when an Experiment is started

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase

hyperparameter_hunter.callbacks.recipes module

This module contains extra callbacks that can add commonly-used functionality to Experiments. This module also serves as an example for how users can properly construct their own custom callbacks using hyperparameter_hunter.callbacks.bases.lambda_callback()

Related

hyperparameter_hunter.callbacks.bases

This module defines hyperparameter_hunter.callbacks.bases.lambda_callback(), which is how all extra callbacks created in hyperparameter_hunter.callbacks.recipes are created

hyperparameter_hunter.environment

This module provides the means to use custom callbacks made by hyperparameter_hunter.callbacks.bases.lambda_callback() through the experiment_callbacks argument of hyperparameter_hunter.environment.Environment.__init__()

Notes

For the purposes of aggregating additional Experiment information, this module describes two methods outlined in hyperparameter_hunter.callbacks.recipes.confusion_matrix_oof(), and hyperparameter_hunter.callbacks.recipes.confusion_matrix_holdout(). The first automatically handles aggregating new values; whereas, the second provides an example for manually aggregating new values, which offers greater customization at the cost of slightly more overhead

hyperparameter_hunter.callbacks.recipes.confusion_matrix_oof(on_run=True, on_fold=True, on_rep=True, on_exp=True)

Callback function to produce confusion matrices for out-of-fold predictions at each stage of the Experiment

Parameters
on_run: Boolean, default=True

If False, skip making confusion matrices for individual Experiment runs

on_fold: Boolean, default=True

If False, skip making confusion matrices for individual Experiment folds

on_rep: Boolean, default=True

If False, skip making confusion matrices for individual Experiment repetitions

on_exp: Boolean, default=True

If False, skip making final confusion matrix for the Experiment

Returns
LambdaCallback

An uninitialized LambdaCallback to generate confusion matrices, produced by hyperparameter_hunter.callbacks.bases.lambda_callback()

Notes

Unlike hyperparameter_hunter.callbacks.recipes.confusion_matrix_holdout(), this callback function allows lambda_callback to automatically aggregate the stats returned by each of the “on…” functions given to lambda_callback

If the size of this lambda_callback implementation is daunting, minimize the helper functions’ docstrings. It’s surprisingly simple

hyperparameter_hunter.callbacks.recipes.confusion_matrix_holdout(on_run=True, on_fold=True, on_rep=True, on_exp=True)

Callback function to produce confusion matrices for holdout predictions at each stage of the Experiment

Parameters
on_run: Boolean, default=True

If False, skip making confusion matrices for individual Experiment runs

on_fold: Boolean, default=True

If False, skip making confusion matrices for individual Experiment folds

on_rep: Boolean, default=True

If False, skip making confusion matrices for individual Experiment repetitions

on_exp: Boolean, default=True

If False, skip making final confusion matrix for the Experiment

Returns
LambdaCallback

An uninitialized LambdaCallback to generate confusion matrices, produced by hyperparameter_hunter.callbacks.bases.lambda_callback()

Notes

Unlike hyperparameter_hunter.callbacks.recipes.confusion_matrix_oof(), this callback bypasses lambda_callback’s ability to automatically aggregate stats returned by the “on…” functions. It does this simply by not returning values in the “on…” functions, and manually aggregating the stats in hyperparameter_hunter.experiments.BaseExperiment.stat_aggregates. This offers greater control over how your values are collected, but also requires additional overhead, namely, instantiating a dict to collect the values via _on_exp_start(). Note also that each of the “on…” functions must append their values to an explicitly named container in hyperparameter_hunter.experiments.BaseExperiment.stat_aggregates when using this method as opposed to hyperparameter_hunter.callbacks.recipes.confusion_matrix_oof()’s

If the size of this lambda_callback implementation is daunting, minimize the helper functions’ docstrings. It’s surprisingly simple

hyperparameter_hunter.callbacks.recipes.dataset_recorder()

Build a LambdaCallback that records the current state of all datasets on_fold_start in order to validate modifications made by feature_engineering.FeatureEngineer/feature_engineering.EngineerStep

Returns
LambdaCallback

Aggregator-like LambdaCallback whose values are aggregated under the name “_datasets” and whose keys are named after the corresponding callback methods

hyperparameter_hunter.callbacks.recipes.lambda_check_train_targets(on_exp_start:Union[List[hyperparameter_hunter.data.data_chunks.target_chunks.TrainTargetChunk], NoneType]=None, on_rep_start:Union[List[hyperparameter_hunter.data.data_chunks.target_chunks.TrainTargetChunk], NoneType]=None, on_fold_start:Union[List[hyperparameter_hunter.data.data_chunks.target_chunks.TrainTargetChunk], NoneType]=None, on_run_start:Union[List[hyperparameter_hunter.data.data_chunks.target_chunks.TrainTargetChunk], NoneType]=None, on_run_end:Union[List[hyperparameter_hunter.data.data_chunks.target_chunks.TrainTargetChunk], NoneType]=None, on_fold_end:Union[List[hyperparameter_hunter.data.data_chunks.target_chunks.TrainTargetChunk], NoneType]=None, on_rep_end:Union[List[hyperparameter_hunter.data.data_chunks.target_chunks.TrainTargetChunk], NoneType]=None, on_exp_end:Union[List[hyperparameter_hunter.data.data_chunks.target_chunks.TrainTargetChunk], NoneType]=None)

LambdaCallback to check the values of an experiment’s data_train.target attribute

The list of TrainTargetChunk instances given to each parameter represents the expected value of hyperparameter_hunter.experiments.CVExperiment.data_train.target for each call of that particular callback method. In other words, the number of items in each parameter’s list should correspond to the number of times that callback method is expected to be invoked.

This means that on_exp_start and on_exp_end should both contain only a single TrainTargetChunk (because they are only ever invoked once by an experiment), and their values should be the expected states of data_train.target on experiment start and end, respectively.

Parameters
on_exp_start: List[TrainTargetChunk], or None, default=None

Expected value of train targets when on_exp_start is invoked. Should contain only a single TrainTargetChunk instance

on_rep_start: List[TrainTargetChunk], or None, default=None

Expected value of train targets on each invocation of on_rep_start. Should contain as many TrainTargetChunk instances as repetitions will be conducted during the experiment. Should contain only a single value if the number or repetitions is one, or if hyperparameter_hunter.environment.Environment.cv_type is not a repeated CV scheme

on_fold_start: List[TrainTargetChunk], or None, default=None

Expected value of train targets on each invocation of on_fold_start. The values to provide are not as straight-forward, as they depend on the number of repetitions as well. If only a single repetition will be conducted, then on_fold_start should simply contain as many TrainTargetChunk instances as folds will be conducted. However, if multiple repetitions will be conducted, then the length of on_fold_start should be (<# of reps> * <# of folds>). For example, if performing RepeatedKFold cross validation with 2 repetitions, and 3 folds/splits, then on_fold_start should contain 6 values

on_run_start: List[TrainTargetChunk], or None, default=None

Expected value of train targets on each invocation of on_run_start. Similarly to on_fold_start, the length/values of on_run_start depends on the number of repetitions, as well as the number of folds that will be conducted. The length of on_run_start should be (<# of reps> * <# of folds> * <# of runs>). If performing standard, non-repeated KFold-like cross validation, with 3 folds, and only a single run, then on_run_start should contain 3 values. Just as in the on_fold_start description example, if performing RepeatedKFold CV with 2 repetitions, and 3 folds, and 1 run, then on_run_start should contain 6 values. On the extreme end, if performing RepeatedKFold CV with 2 repetitions, and 3 folds, and 4 runs, then on_run_start should contain 24 values

on_run_end: List[TrainTargetChunk], or None, default=None

See `on_run_start` description

on_fold_end: List[TrainTargetChunk], or None, default=None

See `on_fold_start` description

on_rep_end: List[TrainTargetChunk], or None, default=None

See `on_rep_start` description

on_exp_end: List[TrainTargetChunk], or None, default=None

See `on_exp_start` description

Notes

As is always the case, on_run_start and on_run_end will still be invoked even if hyperparameter_hunter.environment.Environment.runs is 1. In this case, they will be invoked as many times as on_fold_start and on_fold_end are invoked; however, this does not mean that the values of data_train.target are identical between fold and run divisions

hyperparameter_hunter.callbacks.recipes.aggregator_epochs_elapsed(on_run=True, on_fold=True, on_rep=True, on_exp=True)

Callback function to aggregate and average the number of epochs elapsed during model training at each stage of the Experiment

Parameters
on_run: Boolean, default=True

If False, skip recording epochs elapsed for individual Experiment runs

on_fold: Boolean, default=True

If False, skip making epochs-elapsed averages for individual Experiment folds

on_rep: Boolean, default=True

If False, skip making epochs-elapsed averages for individual Experiment repetitions

on_exp: Boolean, default=True

If False, skip making epochs-elapsed average for the Experiment

Returns
LambdaCallback

An uninitialized LambdaCallback to aggregate the number of epochs elapsed during training, produced by hyperparameter_hunter.callbacks.bases.lambda_callback()

Module contents