hyperparameter_hunter.callbacks package

Subpackages

• hyperparameter_hunter.callbacks.wranglers package
  • Submodules
  • hyperparameter_hunter.callbacks.wranglers.input_wranglers module
    • Related
  • hyperparameter_hunter.callbacks.wranglers.predictors module
    • Related
  • hyperparameter_hunter.callbacks.wranglers.target_wranglers module
    • Related
  • Module contents
    • Related

Submodules

hyperparameter_hunter.callbacks.aggregators module

class hyperparameter_hunter.callbacks.aggregators.AggregatorTimes

Bases: hyperparameter_hunter.callbacks.bases.BaseAggregatorCallback

Methods

on_exp_end(self)	Perform tasks when an Experiment ends
on_exp_start(self)	Perform tasks when an Experiment is started
on_fold_end(self)	Perform tasks on fold end in an Experiment’s cross-validation scheme
on_fold_start(self)	Perform tasks on fold start in an Experiment’s cross-validation scheme
on_rep_end(self)	Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme
on_rep_start(self)	Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme
on_run_end(self)	Perform tasks on run end in an Experiment’s multiple-run-averaging phase
on_run_start(self)	Perform tasks on run start in an Experiment’s multiple-run-averaging phase

on_exp_start(self)

Perform tasks when an Experiment is started

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_exp_end(self)

Perform tasks when an Experiment ends

class hyperparameter_hunter.callbacks.aggregators.AggregatorEvaluations

Bases: hyperparameter_hunter.callbacks.bases.BaseAggregatorCallback

Methods

on_exp_end(self)	Perform tasks when an Experiment ends
on_exp_start(self)	Perform tasks when an Experiment is started
on_fold_end(self)	Perform tasks on fold end in an Experiment’s cross-validation scheme
on_fold_start(self)	Perform tasks on fold start in an Experiment’s cross-validation scheme
on_rep_end(self)	Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme
on_rep_start(self)	Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme
on_run_end(self)	Perform tasks on run end in an Experiment’s multiple-run-averaging phase
on_run_start(self)	Perform tasks on run start in an Experiment’s multiple-run-averaging phase

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_exp_end(self)

Perform tasks when an Experiment ends

class hyperparameter_hunter.callbacks.aggregators.AggregatorOOF

Bases: hyperparameter_hunter.callbacks.bases.BaseAggregatorCallback

Methods

on_exp_end(self)	Perform tasks when an Experiment ends
on_exp_start(self)	Perform tasks when an Experiment is started
on_fold_end(self)	Perform tasks on fold end in an Experiment’s cross-validation scheme
on_fold_start(self)	Perform tasks on fold start in an Experiment’s cross-validation scheme
on_rep_end(self)	Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme
on_rep_start(self)	Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme
on_run_end(self)	Perform tasks on run end in an Experiment’s multiple-run-averaging phase
on_run_start(self)	Perform tasks on run start in an Experiment’s multiple-run-averaging phase

class hyperparameter_hunter.callbacks.aggregators.AggregatorHoldout

Bases: hyperparameter_hunter.callbacks.bases.BaseAggregatorCallback

Methods

on_exp_end(self)	Perform tasks when an Experiment ends
on_exp_start(self)	Perform tasks when an Experiment is started
on_fold_end(self)	Perform tasks on fold end in an Experiment’s cross-validation scheme
on_fold_start(self)	Perform tasks on fold start in an Experiment’s cross-validation scheme
on_rep_end(self)	Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme
on_rep_start(self)	Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme
on_run_end(self)	Perform tasks on run end in an Experiment’s multiple-run-averaging phase
on_run_start(self)	Perform tasks on run start in an Experiment’s multiple-run-averaging phase

class hyperparameter_hunter.callbacks.aggregators.AggregatorTest

Bases: hyperparameter_hunter.callbacks.bases.BaseAggregatorCallback

Methods

on_exp_end(self)	Perform tasks when an Experiment ends
on_exp_start(self)	Perform tasks when an Experiment is started
on_fold_end(self)	Perform tasks on fold end in an Experiment’s cross-validation scheme
on_fold_start(self)	Perform tasks on fold start in an Experiment’s cross-validation scheme
on_rep_end(self)	Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme
on_rep_start(self)	Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme
on_run_end(self)	Perform tasks on run end in an Experiment’s multiple-run-averaging phase
on_run_start(self)	Perform tasks on run start in an Experiment’s multiple-run-averaging phase

class hyperparameter_hunter.callbacks.aggregators.AggregatorLosses

Bases: hyperparameter_hunter.callbacks.bases.BaseAggregatorCallback

Methods

on_exp_end(self)	Perform tasks when an Experiment ends
on_exp_start(self)	Perform tasks when an Experiment is started
on_fold_end(self)	Perform tasks on fold end in an Experiment’s cross-validation scheme
on_fold_start(self)	Perform tasks on fold start in an Experiment’s cross-validation scheme
on_rep_end(self)	Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme
on_rep_start(self)	Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme
on_run_end(self)	Perform tasks on run end in an Experiment’s multiple-run-averaging phase
on_run_start(self)	Perform tasks on run start in an Experiment’s multiple-run-averaging phase

hyperparameter_hunter.callbacks.bases module

This module defines the base callback classes, from which all other callback classes in hyperparameter_hunter.callbacks are descendants. Importantly, the specific base callback classes contained herein are all descendants of hyperparameter_hunter.callbacks.bases.BaseCallback, ensuring all callbacks descend from the same base class. This module also defines hyperparameter_hunter.callbacks.bases.lambda_callback(), which can be used to define custom callbacks to be executed during Experiments when passed to hyperparameter_hunter.environment.Environment.__init__() via the experiment_callbacks argument

Related

hyperparameter_hunter.callbacks

The rest of the submodules within this module should define classes which all descend from the base callback classes defined in hyperparameter_hunter.callbacks.bases

hyperparameter_hunter.experiment_core

This is where callback classes are added as bases inherited by hyperparameter_hunter.experiments.BaseExperiment. This module is the path that links hyperparameter_hunter.callbacks to hyperparameter_hunter.experiments

hyperparameter_hunter.environment

This module provides the means to use custom callbacks made by hyperparameter_hunter.callbacks.bases.lambda_callback() through the experiment_callbacks argument of hyperparameter_hunter.environment.Environment.__init__()

class hyperparameter_hunter.callbacks.bases.BaseCallback

Bases: object

The base class from which all callbacks and all intermediate base callbacks are descendants. Callback classes’ __init__() will not be called, so any tasks that must be performed at the onset of an experiment should be placed in on_exp_start()

Notes

__init__(): Some classes that inherit BaseCallback may implement __init__(); however, callback classes are NEVER INITIALIZED. Callback classes should be regarded as extensions to the Experiment class that is inheriting them. Because they are inherited, callback classes have access to all attributes of the inheriting Experiment class. If any callback class does implement __init__(), it is simply a convention to clearly declare the attributes required by the other methods of that callback class. Again, __init__() of classes that inherit BaseCallback will not be called

The methods below each call settings.G.debug() to signal that the dynamic callback inheritance organized in experiment_core.ExperimentMeta has proceeded at least partially successfully. If all callback methods end by executing the method of the same name in their parent classes, then the below debug messages should be visible in the “Heartbeat.log” file. Conversely, if any of the below debug messages are not printed to “Heartbeat.log”, it is likely that a callback class’s implementation of the corresponding method does not end with “super().<method_name>()”. Such cases should be remedied immediately, as the callback stream could be skipping any number of other callbacks

Methods

on_exp_end(self)	Perform tasks when an Experiment ends
on_exp_start(self)	Perform tasks when an Experiment is started
on_fold_end(self)	Perform tasks on fold end in an Experiment’s cross-validation scheme
on_fold_start(self)	Perform tasks on fold start in an Experiment’s cross-validation scheme
on_rep_end(self)	Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme
on_rep_start(self)	Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme
on_run_end(self)	Perform tasks on run end in an Experiment’s multiple-run-averaging phase
on_run_start(self)	Perform tasks on run start in an Experiment’s multiple-run-averaging phase

on_exp_start(self)

Perform tasks when an Experiment is started

on_exp_end(self)

Perform tasks when an Experiment ends

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

hyperparameter_hunter.callbacks.bases.lambda_callback(on_exp_start=None, on_exp_end=None, on_rep_start=None, on_rep_end=None, on_fold_start=None, on_fold_end=None, on_run_start=None, on_run_end=None, agg_name=None, do_reshape_aggs=True, method_agg_keys=False, on_experiment_start=<object object at 0x7fe8726edbf0>, on_experiment_end=<object object at 0x7fe8726edbf0>, on_repetition_start=<object object at 0x7fe8726edbf0>, on_repetition_end=<object object at 0x7fe8726edbf0>)

Utility for creating custom callbacks to be declared by Environment and used by Experiments. The callable “on_<…>_<start/end>” parameters provided will receive as input whichever attributes of the Experiment are included in the signature of the given callable. If **kwargs is given in the callable’s signature, a dict of all of the Experiment’s attributes will be provided. This can be helpful for trying to figure out how to build a custom callback, but should not be used unless absolutely necessary. If the Experiment does not have an attribute specified in the callable’s signature, the following placeholder will be given: “INVALID KWARG”

Parameters

on_exp_start: Callable, or None, default=None

Callable that receives Experiment’s values for parameters in the signature at Experiment start

on_exp_end: Callable, or None, default=None

Callable that receives Experiment’s values for parameters in the signature at Experiment end

on_rep_start: Callable, or None, default=None

Callable that receives Experiment’s values for parameters in the signature at repetition start

on_rep_end: Callable, or None, default=None

Callable that receives Experiment’s values for parameters in the signature at repetition end

on_fold_start: Callable, or None, default=None

Callable that receives Experiment’s values for parameters in the signature at fold start

on_fold_end: Callable, or None, default=None

Callable that receives Experiment’s values for parameters in the signature at fold end

on_run_start: Callable, or None, default=None

Callable that receives Experiment’s values for parameters in the signature at run start

on_run_end: Callable, or None, default=None

Callable that receives Experiment’s values for parameters in the signature at run end

agg_name: Str, default=uuid.uuid4

This parameter is only used if the callables are behaving like AggregatorCallbacks by returning values (see the “Notes” section below for details on this). If the callables do return values, they will be stored under a key named (“_” + agg_name) in a dict in hyperparameter_hunter.experiments.BaseExperiment.stat_aggregates. The purpose of this parameter is to make it easier to understand an Experiment’s description file, as agg_name will default to a UUID if it is not given

do_reshape_aggs: Boolean, default=True

Whether to reshape the aggregated values to reflect the nested repetitions/folds/runs structure used for other aggregated values. If False, lists of aggregated values are left in their original shapes. This parameter is only used if the callables are behaving like AggregatorCallbacks (see the “Notes” section below and agg_name for details on this)

method_agg_keys: Boolean, default=False

If True, the aggregate keys for the items added to the dict at agg_name are equivalent to the names of the “on_<…>_<start/end>” pseudo-methods whose values are being aggregated. In other words, the pool of all possible aggregate keys goes from [“runs”, “folds”, “reps”, “final”] to the names of the eight “on_<…>_<start/end>” kwargs of lambda_callback(). See the “Notes” section below for further details and a rough outline

on_experiment_start: …

Deprecated since version 3.0.0: Renamed to on_exp_start. Will be removed in 3.2.0

on_experiment_end: …

Deprecated since version 3.0.0: Renamed to on_exp_end. Will be removed in 3.2.0

on_repetition_start: …

Deprecated since version 3.0.0: Renamed to on_rep_start. Will be removed in 3.2.0

on_repetition_end: …

Deprecated since version 3.0.0: Renamed to on_rep_end. Will be removed in 3.2.0

Returns

LambdaCallback: LambdaCallback

Uninitialized class, whose methods are the callables of the corresponding “on…” kwarg

Notes

For all of the “on_<…>_<start/end>” callables provided as input to lambda_callback, consider the following guidelines (for example function “f”, which can represent any of the callables):

• All input parameters in the signature of “f” are attributes of the Experiment being executed

  • If “**kwargs” is a parameter, a dict of all the Experiment’s attributes will be provided

• “f” will be treated as a method of a parent class of the Experiment

  • Take care when modifying attributes, as changes are reflected in the Experiment itself

• If “f” returns something, it will automatically behave like an AggregatorCallback (see hyperparameter_hunter.callbacks.aggregators). Specifically, the following will occur:

  • A new key (named by agg_name if given, else a UUID) with a dict value is added to hyperparameter_hunter.experiments.BaseExperiment.stat_aggregates

    • This new dict can have up to four keys: “runs” (list), “folds” (list), “reps” (list), and “final” (object)

  • If “f” is an “on_run…” function, the returned value is appended to the “runs” list in the new dict

  • Similarly, if “f” is an “on_fold…” or “on_rep…” function, the returned value is appended to the “folds”, or “reps” list, respectively

  • If “f” is an “on_exp…” function, the “final” key in the new dict is set to the returned value

  • If values were aggregated in the aforementioned manner, the lists of collected values will be reshaped according to runs/folds/reps on Experiment end

  • The aggregated values will be saved in the Experiment’s description file

    • This is because hyperparameter_hunter.experiments.BaseExperiment.stat_aggregates is saved in its entirety

What follows is a rough outline of the structure produced when using an aggregator-like callback that automatically populates experiments.BaseExperiment.stat_aggregates with results of the functions used as arguments to lambda_callback():

BaseExperiment.stat_aggregates = dict(
    ...,
    <`agg_name`>=dict(
        <agg_key "runs">  = [...],
        <agg_key "folds"> = [...],
        <agg_key "reps">  = [...],
        <agg_key "final"> = object(),
        ...
    ),
    ...
)

In the above outline, the actual agg_key`s included in the dict at `agg_name depend on which “on_<…>_<start/end>” callables are behaving like aggregators. For example, if neither on_run_start nor on_run_end explicitly returns something, then the “runs” agg_key is not included in the agg_name dict. Similarly, if, for example, neither on_exp_start nor on_exp_end is provided, then the “final” agg_key is not included. If method_agg_keys=True, then the agg keys used in the dict are modified to be named after the method called. For example, if method_agg_keys=True and on_fold_start and on_fold_end are both callables returning values to be aggregated, then the agg_key`s used for each will be “on_fold_start” and “on_fold_end”, respectively. In this example, if `method_agg_keys=False (default) and do_reshape_aggs=False, then the single “folds” agg_key would contain the combined contents returned by both methods in the order in which they were returned

For examples using lambda_callback to create custom callbacks, see hyperparameter_hunter.callbacks.recipes

Examples

>>> from hyperparameter_hunter.environment import Environment
>>> def printer_helper(_rep, _fold, _run, last_evaluation_results):
...     print(f"{_rep}.{_fold}.{_run}   {last_evaluation_results}")
>>> my_lambda_callback = lambda_callback(
...     on_exp_end=printer_helper,
...     on_rep_end=printer_helper,
...     on_fold_end=printer_helper,
...     on_run_end=printer_helper,
... )
... # env = Environment(
... #     train_dataset="i am a dataset",
... #     results_path="path/to/HyperparameterHunterAssets",
... #     metrics=["roc_auc_score"],
... #     experiment_callbacks=[my_lambda_callback]
... # )
... # ... Now execute an Experiment, or an Optimization Protocol...

See hyperparameter_hunter.examples.lambda_callback_example for more information

class hyperparameter_hunter.callbacks.bases.BaseWranglerCallback

Bases: hyperparameter_hunter.callbacks.bases.BaseCallback

Base class from which all callbacks in hyperparameter_hunter.callbacks.wranglers are descendants

Methods

on_exp_end(self)	Perform tasks when an Experiment ends
on_exp_start(self)	Perform tasks when an Experiment is started
on_fold_end(self)	Perform tasks on fold end in an Experiment’s cross-validation scheme
on_fold_start(self)	Perform tasks on fold start in an Experiment’s cross-validation scheme
on_rep_end(self)	Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme
on_rep_start(self)	Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme
on_run_end(self)	Perform tasks on run end in an Experiment’s multiple-run-averaging phase
on_run_start(self)	Perform tasks on run start in an Experiment’s multiple-run-averaging phase

class hyperparameter_hunter.callbacks.bases.BaseInputWranglerCallback

Bases: hyperparameter_hunter.callbacks.bases.BaseWranglerCallback

Methods

on_exp_end(self)	Perform tasks when an Experiment ends
on_exp_start(self)	Perform tasks when an Experiment is started
on_fold_end(self)	Perform tasks on fold end in an Experiment’s cross-validation scheme
on_fold_start(self)	Perform tasks on fold start in an Experiment’s cross-validation scheme
on_rep_end(self)	Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme
on_rep_start(self)	Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme
on_run_end(self)	Perform tasks on run end in an Experiment’s multiple-run-averaging phase
on_run_start(self)	Perform tasks on run start in an Experiment’s multiple-run-averaging phase

class hyperparameter_hunter.callbacks.bases.BaseTargetWranglerCallback

Bases: hyperparameter_hunter.callbacks.bases.BaseWranglerCallback

Methods

on_exp_end(self)	Perform tasks when an Experiment ends
on_exp_start(self)	Perform tasks when an Experiment is started
on_fold_end(self)	Perform tasks on fold end in an Experiment’s cross-validation scheme
on_fold_start(self)	Perform tasks on fold start in an Experiment’s cross-validation scheme
on_rep_end(self)	Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme
on_rep_start(self)	Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme
on_run_end(self)	Perform tasks on run end in an Experiment’s multiple-run-averaging phase
on_run_start(self)	Perform tasks on run start in an Experiment’s multiple-run-averaging phase

class hyperparameter_hunter.callbacks.bases.BasePredictorCallback

Bases: hyperparameter_hunter.callbacks.bases.BaseWranglerCallback

Methods

on_exp_end(self)	Perform tasks when an Experiment ends
on_exp_start(self)	Perform tasks when an Experiment is started
on_fold_end(self)	Perform tasks on fold end in an Experiment’s cross-validation scheme
on_fold_start(self)	Perform tasks on fold start in an Experiment’s cross-validation scheme
on_rep_end(self)	Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme
on_rep_start(self)	Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme
on_run_end(self)	Perform tasks on run end in an Experiment’s multiple-run-averaging phase
on_run_start(self)	Perform tasks on run start in an Experiment’s multiple-run-averaging phase

class hyperparameter_hunter.callbacks.bases.BaseLoggerCallback

Bases: hyperparameter_hunter.callbacks.bases.BaseCallback

Base class from which all callbacks in hyperparameter_hunter.callbacks.loggers are descendants

Methods

on_exp_end(self)	Perform tasks when an Experiment ends
on_exp_start(self)	Perform tasks when an Experiment is started
on_fold_end(self)	Perform tasks on fold end in an Experiment’s cross-validation scheme
on_fold_start(self)	Perform tasks on fold start in an Experiment’s cross-validation scheme
on_rep_end(self)	Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme
on_rep_start(self)	Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme
on_run_end(self)	Perform tasks on run end in an Experiment’s multiple-run-averaging phase
on_run_start(self)	Perform tasks on run start in an Experiment’s multiple-run-averaging phase

class hyperparameter_hunter.callbacks.bases.BaseAggregatorCallback

Bases: hyperparameter_hunter.callbacks.bases.BaseCallback

Base class from which all callbacks in hyperparameter_hunter.callbacks.aggregators are descendants

Methods

on_exp_end(self)	Perform tasks when an Experiment ends
on_exp_start(self)	Perform tasks when an Experiment is started
on_fold_end(self)	Perform tasks on fold end in an Experiment’s cross-validation scheme
on_fold_start(self)	Perform tasks on fold start in an Experiment’s cross-validation scheme
on_rep_end(self)	Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme
on_rep_start(self)	Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme
on_run_end(self)	Perform tasks on run end in an Experiment’s multiple-run-averaging phase
on_run_start(self)	Perform tasks on run start in an Experiment’s multiple-run-averaging phase

class hyperparameter_hunter.callbacks.bases.BaseEvaluatorCallback

Bases: hyperparameter_hunter.callbacks.bases.BaseCallback

Base class from which all callbacks in hyperparameter_hunter.callbacks.evaluators are descendants

Methods

on_exp_end(self)	Perform tasks when an Experiment ends
on_exp_start(self)	Perform tasks when an Experiment is started
on_fold_end(self)	Perform tasks on fold end in an Experiment’s cross-validation scheme
on_fold_start(self)	Perform tasks on fold start in an Experiment’s cross-validation scheme
on_rep_end(self)	Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme
on_rep_start(self)	Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme
on_run_end(self)	Perform tasks on run end in an Experiment’s multiple-run-averaging phase
on_run_start(self)	Perform tasks on run start in an Experiment’s multiple-run-averaging phase

hyperparameter_hunter.callbacks.evaluators module

This module defines Evaluator callbacks to score predictions generated during the different time divisions of the BaseExperiment by invoking hyperparameter_hunter.metrics.ScoringMixIn.evaluate()

Related

hyperparameter_hunter.metrics

Defines ScoringMixIn, which is inherited by BaseExperiment, and provides the evaluate method that is called by the classes in evaluators

Notes

Regarding evaluation when G.Env.save_transformed_metrics is False, target data will be either fold (for on_run_end/on_fold_end) or d (for on_rep_end/on_exp_end). Prediction data used for evaluation in this case does not follow this abnormal pattern. Target data is limited to either the fold or d data_chunks when G.Env.save_transformed_metrics is False because targets for run and rep are identical to the targets for fold and d, respectively. This is still the case even if performing inverse target transformation via EngineerStep. Because the target values do not change between these two pairs of divisions, their values may be unset, so the targets for the division immediately above are used instead. As noted in hyperparameter_hunter.data.data_chunks.target_chunks, both itself and hyperparameter_hunter.callback.wranglers.target_wranglers are concerned only with transformed targets–not with original targets (or inverted targets). That is because original targets and inverted targets should be identical. Original targets are updated only in hyperparameter_hunter.experiments.BaseExperiment.on_exp_start() (through hyperparameter_hunter.data.data_core.BaseDataset initialization) and in hyperparameter_hunter.experiments.BaseCVExperiment.on_fold_start()

class hyperparameter_hunter.callbacks.evaluators.EvaluatorOOF

Bases: hyperparameter_hunter.callbacks.bases.BaseEvaluatorCallback

Methods

on_exp_end(self)	Evaluate final (run/repetition-averaged) out-of-fold predictions
on_exp_start(self)	Perform tasks when an Experiment is started
on_fold_end(self)	Evaluate (run-averaged) out-of-fold predictions for the fold
on_fold_start(self)	Perform tasks on fold start in an Experiment’s cross-validation scheme
on_rep_end(self)	Evaluate (run-averaged) out-of-fold predictions for the repetition
on_rep_start(self)	Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme
on_run_end(self)	Evaluate out-of-fold predictions for the run
on_run_start(self)	Perform tasks on run start in an Experiment’s multiple-run-averaging phase

on_run_end(self)

Evaluate out-of-fold predictions for the run

on_fold_end(self)

Evaluate (run-averaged) out-of-fold predictions for the fold

on_rep_end(self)

Evaluate (run-averaged) out-of-fold predictions for the repetition

on_exp_end(self)

Evaluate final (run/repetition-averaged) out-of-fold predictions

class hyperparameter_hunter.callbacks.evaluators.EvaluatorHoldout

Bases: hyperparameter_hunter.callbacks.bases.BaseEvaluatorCallback

Methods

on_exp_end(self)	Evaluate final (run/repetition-averaged) holdout predictions
on_exp_start(self)	Perform tasks when an Experiment is started
on_fold_end(self)	Evaluate (run-averaged) holdout predictions for the fold
on_fold_start(self)	Perform tasks on fold start in an Experiment’s cross-validation scheme
on_rep_end(self)	Evaluate (run-averaged) holdout predictions for the repetition
on_rep_start(self)	Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme
on_run_end(self)	Evaluate holdout predictions for the run
on_run_start(self)	Perform tasks on run start in an Experiment’s multiple-run-averaging phase

on_run_end(self)

Evaluate holdout predictions for the run

on_fold_end(self)

Evaluate (run-averaged) holdout predictions for the fold

on_rep_end(self)

Evaluate (run-averaged) holdout predictions for the repetition

on_exp_end(self)

Evaluate final (run/repetition-averaged) holdout predictions

hyperparameter_hunter.callbacks.loggers module

class hyperparameter_hunter.callbacks.loggers.LoggerFitStatus

Bases: hyperparameter_hunter.callbacks.bases.BaseLoggerCallback

Methods

on_exp_end(self)	Perform tasks when an Experiment ends
on_exp_start(self)	Perform tasks when an Experiment is started
on_fold_end(self)	Perform tasks on fold end in an Experiment’s cross-validation scheme
on_fold_start(self)	Perform tasks on fold start in an Experiment’s cross-validation scheme
on_rep_end(self)	Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme
on_rep_start(self)	Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme
on_run_end(self)	Perform tasks on run end in an Experiment’s multiple-run-averaging phase
on_run_start(self)	Perform tasks on run start in an Experiment’s multiple-run-averaging phase

float_format = '{:.5f}'

log_separator = ' | '

on_exp_start(self)

Perform tasks when an Experiment is started

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_exp_end(self)

Perform tasks when an Experiment ends

class hyperparameter_hunter.callbacks.loggers.LoggerOOF

Bases: hyperparameter_hunter.callbacks.bases.BaseLoggerCallback

Methods

on_exp_end(self)	Perform tasks when an Experiment ends
on_exp_start(self)	Perform tasks when an Experiment is started
on_fold_end(self)	Perform tasks on fold end in an Experiment’s cross-validation scheme
on_fold_start(self)	Perform tasks on fold start in an Experiment’s cross-validation scheme
on_rep_end(self)	Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme
on_rep_start(self)	Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme
on_run_end(self)	Perform tasks on run end in an Experiment’s multiple-run-averaging phase
on_run_start(self)	Perform tasks on run start in an Experiment’s multiple-run-averaging phase

class hyperparameter_hunter.callbacks.loggers.LoggerHoldout

Bases: hyperparameter_hunter.callbacks.bases.BaseLoggerCallback

Methods

on_exp_end(self)	Perform tasks when an Experiment ends
on_exp_start(self)	Perform tasks when an Experiment is started
on_fold_end(self)	Perform tasks on fold end in an Experiment’s cross-validation scheme
on_fold_start(self)	Perform tasks on fold start in an Experiment’s cross-validation scheme
on_rep_end(self)	Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme
on_rep_start(self)	Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme
on_run_end(self)	Perform tasks on run end in an Experiment’s multiple-run-averaging phase
on_run_start(self)	Perform tasks on run start in an Experiment’s multiple-run-averaging phase

class hyperparameter_hunter.callbacks.loggers.LoggerTest

Bases: hyperparameter_hunter.callbacks.bases.BaseLoggerCallback

Methods

on_exp_end(self)	Perform tasks when an Experiment ends
on_exp_start(self)	Perform tasks when an Experiment is started
on_fold_end(self)	Perform tasks on fold end in an Experiment’s cross-validation scheme
on_fold_start(self)	Perform tasks on fold start in an Experiment’s cross-validation scheme
on_rep_end(self)	Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme
on_rep_start(self)	Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme
on_run_end(self)	Perform tasks on run end in an Experiment’s multiple-run-averaging phase
on_run_start(self)	Perform tasks on run start in an Experiment’s multiple-run-averaging phase

class hyperparameter_hunter.callbacks.loggers.LoggerEvaluation

Bases: hyperparameter_hunter.callbacks.bases.BaseLoggerCallback

Methods

on_exp_end(self)	Perform tasks when an Experiment ends
on_exp_start(self)	Perform tasks when an Experiment is started
on_fold_end(self)	Perform tasks on fold end in an Experiment’s cross-validation scheme
on_fold_start(self)	Perform tasks on fold start in an Experiment’s cross-validation scheme
on_rep_end(self)	Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme
on_rep_start(self)	Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme
on_run_end(self)	Perform tasks on run end in an Experiment’s multiple-run-averaging phase
on_run_start(self)	Perform tasks on run start in an Experiment’s multiple-run-averaging phase

hyperparameter_hunter.callbacks.recipes module

This module contains extra callbacks that can add commonly-used functionality to Experiments. This module also serves as an example for how users can properly construct their own custom callbacks using hyperparameter_hunter.callbacks.bases.lambda_callback()

Related

hyperparameter_hunter.callbacks.bases

This module defines hyperparameter_hunter.callbacks.bases.lambda_callback(), which is how all extra callbacks created in hyperparameter_hunter.callbacks.recipes are created

hyperparameter_hunter.environment

This module provides the means to use custom callbacks made by hyperparameter_hunter.callbacks.bases.lambda_callback() through the experiment_callbacks argument of hyperparameter_hunter.environment.Environment.__init__()

Notes

For the purposes of aggregating additional Experiment information, this module describes two methods outlined in hyperparameter_hunter.callbacks.recipes.confusion_matrix_oof(), and hyperparameter_hunter.callbacks.recipes.confusion_matrix_holdout(). The first automatically handles aggregating new values; whereas, the second provides an example for manually aggregating new values, which offers greater customization at the cost of slightly more overhead

hyperparameter_hunter.callbacks.recipes.confusion_matrix_oof(on_run=True, on_fold=True, on_rep=True, on_exp=True)

Callback function to produce confusion matrices for out-of-fold predictions at each stage of the Experiment

Parameters

on_run: Boolean, default=True

If False, skip making confusion matrices for individual Experiment runs

on_fold: Boolean, default=True

If False, skip making confusion matrices for individual Experiment folds

on_rep: Boolean, default=True

If False, skip making confusion matrices for individual Experiment repetitions

on_exp: Boolean, default=True

If False, skip making final confusion matrix for the Experiment

Returns

LambdaCallback

An uninitialized LambdaCallback to generate confusion matrices, produced by hyperparameter_hunter.callbacks.bases.lambda_callback()

Notes

Unlike hyperparameter_hunter.callbacks.recipes.confusion_matrix_holdout(), this callback function allows lambda_callback to automatically aggregate the stats returned by each of the “on…” functions given to lambda_callback

If the size of this lambda_callback implementation is daunting, minimize the helper functions’ docstrings. It’s surprisingly simple

hyperparameter_hunter.callbacks.recipes.confusion_matrix_holdout(on_run=True, on_fold=True, on_rep=True, on_exp=True)

Callback function to produce confusion matrices for holdout predictions at each stage of the Experiment

Parameters

on_run: Boolean, default=True

If False, skip making confusion matrices for individual Experiment runs

on_fold: Boolean, default=True

If False, skip making confusion matrices for individual Experiment folds

on_rep: Boolean, default=True

If False, skip making confusion matrices for individual Experiment repetitions

on_exp: Boolean, default=True

If False, skip making final confusion matrix for the Experiment

Returns

LambdaCallback

An uninitialized LambdaCallback to generate confusion matrices, produced by hyperparameter_hunter.callbacks.bases.lambda_callback()

Notes

Unlike hyperparameter_hunter.callbacks.recipes.confusion_matrix_oof(), this callback bypasses lambda_callback’s ability to automatically aggregate stats returned by the “on…” functions. It does this simply by not returning values in the “on…” functions, and manually aggregating the stats in hyperparameter_hunter.experiments.BaseExperiment.stat_aggregates. This offers greater control over how your values are collected, but also requires additional overhead, namely, instantiating a dict to collect the values via _on_exp_start(). Note also that each of the “on…” functions must append their values to an explicitly named container in hyperparameter_hunter.experiments.BaseExperiment.stat_aggregates when using this method as opposed to hyperparameter_hunter.callbacks.recipes.confusion_matrix_oof()’s

If the size of this lambda_callback implementation is daunting, minimize the helper functions’ docstrings. It’s surprisingly simple

hyperparameter_hunter.callbacks.recipes.dataset_recorder()

Build a LambdaCallback that records the current state of all datasets on_fold_start in order to validate modifications made by feature_engineering.FeatureEngineer/feature_engineering.EngineerStep

Returns

LambdaCallback

Aggregator-like LambdaCallback whose values are aggregated under the name “_datasets” and whose keys are named after the corresponding callback methods

hyperparameter_hunter.callbacks.recipes.lambda_check_train_targets(on_exp_start:Union[List[hyperparameter_hunter.data.data_chunks.target_chunks.TrainTargetChunk], NoneType]=None, on_rep_start:Union[List[hyperparameter_hunter.data.data_chunks.target_chunks.TrainTargetChunk], NoneType]=None, on_fold_start:Union[List[hyperparameter_hunter.data.data_chunks.target_chunks.TrainTargetChunk], NoneType]=None, on_run_start:Union[List[hyperparameter_hunter.data.data_chunks.target_chunks.TrainTargetChunk], NoneType]=None, on_run_end:Union[List[hyperparameter_hunter.data.data_chunks.target_chunks.TrainTargetChunk], NoneType]=None, on_fold_end:Union[List[hyperparameter_hunter.data.data_chunks.target_chunks.TrainTargetChunk], NoneType]=None, on_rep_end:Union[List[hyperparameter_hunter.data.data_chunks.target_chunks.TrainTargetChunk], NoneType]=None, on_exp_end:Union[List[hyperparameter_hunter.data.data_chunks.target_chunks.TrainTargetChunk], NoneType]=None)

LambdaCallback to check the values of an experiment’s data_train.target attribute

The list of TrainTargetChunk instances given to each parameter represents the expected value of hyperparameter_hunter.experiments.CVExperiment.data_train.target for each call of that particular callback method. In other words, the number of items in each parameter’s list should correspond to the number of times that callback method is expected to be invoked.

This means that on_exp_start and on_exp_end should both contain only a single TrainTargetChunk (because they are only ever invoked once by an experiment), and their values should be the expected states of data_train.target on experiment start and end, respectively.

Parameters

on_exp_start: List[TrainTargetChunk], or None, default=None

Expected value of train targets when on_exp_start is invoked. Should contain only a single TrainTargetChunk instance

on_rep_start: List[TrainTargetChunk], or None, default=None

Expected value of train targets on each invocation of on_rep_start. Should contain as many TrainTargetChunk instances as repetitions will be conducted during the experiment. Should contain only a single value if the number or repetitions is one, or if hyperparameter_hunter.environment.Environment.cv_type is not a repeated CV scheme

on_fold_start: List[TrainTargetChunk], or None, default=None

Expected value of train targets on each invocation of on_fold_start. The values to provide are not as straight-forward, as they depend on the number of repetitions as well. If only a single repetition will be conducted, then on_fold_start should simply contain as many TrainTargetChunk instances as folds will be conducted. However, if multiple repetitions will be conducted, then the length of on_fold_start should be (<# of reps> * <# of folds>). For example, if performing RepeatedKFold cross validation with 2 repetitions, and 3 folds/splits, then on_fold_start should contain 6 values

on_run_start: List[TrainTargetChunk], or None, default=None

Expected value of train targets on each invocation of on_run_start. Similarly to on_fold_start, the length/values of on_run_start depends on the number of repetitions, as well as the number of folds that will be conducted. The length of on_run_start should be (<# of reps> * <# of folds> * <# of runs>). If performing standard, non-repeated KFold-like cross validation, with 3 folds, and only a single run, then on_run_start should contain 3 values. Just as in the on_fold_start description example, if performing RepeatedKFold CV with 2 repetitions, and 3 folds, and 1 run, then on_run_start should contain 6 values. On the extreme end, if performing RepeatedKFold CV with 2 repetitions, and 3 folds, and 4 runs, then on_run_start should contain 24 values

on_run_end: List[TrainTargetChunk], or None, default=None

See `on_run_start` description

on_fold_end: List[TrainTargetChunk], or None, default=None

See `on_fold_start` description

on_rep_end: List[TrainTargetChunk], or None, default=None

See `on_rep_start` description

on_exp_end: List[TrainTargetChunk], or None, default=None

See `on_exp_start` description

Notes

As is always the case, on_run_start and on_run_end will still be invoked even if hyperparameter_hunter.environment.Environment.runs is 1. In this case, they will be invoked as many times as on_fold_start and on_fold_end are invoked; however, this does not mean that the values of data_train.target are identical between fold and run divisions

hyperparameter_hunter.callbacks.recipes.aggregator_epochs_elapsed(on_run=True, on_fold=True, on_rep=True, on_exp=True)

Callback function to aggregate and average the number of epochs elapsed during model training at each stage of the Experiment

Parameters

on_run: Boolean, default=True

If False, skip recording epochs elapsed for individual Experiment runs

on_fold: Boolean, default=True

If False, skip making epochs-elapsed averages for individual Experiment folds

on_rep: Boolean, default=True

If False, skip making epochs-elapsed averages for individual Experiment repetitions

on_exp: Boolean, default=True

If False, skip making epochs-elapsed average for the Experiment

Returns

LambdaCallback

An uninitialized LambdaCallback to aggregate the number of epochs elapsed during training, produced by hyperparameter_hunter.callbacks.bases.lambda_callback()

Module contents