hyperparameter_hunter.callbacks.wranglers package

Submodules

hyperparameter_hunter.callbacks.wranglers.input_wranglers module

This module defines callbacks that descend from BaseInputWranglerCallback. Input wrangler callbacks are concerned with managing the input data chunks of an experiment’s datasets. This module acts as a liaison between :class:~hyperparameter_hunter.experiments.BaseCVExperiment` and the input chunk classes defined in hyperparameter_hunter.data.data_chunks.input_chunks. Each callback defined herein is responsible for ensuring the proper execution of precisely one descendant of BaseInputChunk, defined in input_chunks.

Input wranglers are quite a bit less interesting than the other wranglers because they kinda “stop caring” after a while. Input data exists entirely for the purposes of fitting a model and making predictions - between on_run_start and on_run_end. For essential operations, we don’t need input data once we hit on_run_end, so none of the “…end” methods of input wranglers do anything.

hyperparameter_hunter.callbacks.wranglers.predictors module

This module defines callbacks that descend from BasePredictorCallback. Predictor wrangler callbacks are concerned with managing the prediction chunks of an experiment’s datasets. This module acts as a liaison between :class:~hyperparameter_hunter.experiments.BaseCVExperiment` and the prediction chunk classes defined in hyperparameter_hunter.data.data_chunks.prediction_chunks. Each callback defined herein is responsible for ensuring the proper execution of precisely one descendant of BasePredictorChunk, defined in prediction_chunks.

Predictors are the busiest of all three wrangler callbacks. While we only actually get predictions when we first hit on_run_end, we need to keep track of them across runs, folds and reps, so predictions need to be cleared out during the “…start” callback methods. There are two mission-critical tasks for which we need predictions: 1) Evaluation against targets, and 2) Recording - not only to ensure our model is behaving as expected, but also for ensembling. Ensembling is a real pain if you’re trying to do it, using only evaluation metrics as a guide, and re-running selected experiments so you can save the predictions this time, just to figure out if the ensemble actually performs in the end.

Once again, feature engineering throws a monkey-wrench into our expectations for the predictor callbacks. If we’re performing any kind of target transformation (which is often the case), then evaluations need to be made using transformed predictions and targets. Calculating f1-score would not go well if we tried to give the metric function the stringified iris dataset labels of “setosa”, “versicolor”, or “virginica”. It’s gonna want the transformed, numerical representation of the targets. Similarly, averaging predictions across divisions uses transformed predictions because it requires values that can actually be averaged. For the purposes of recording, we may want either transformed or inverted (original form) prediction - or both. Lots of weird things start misbehaving in lots of confusing ways if our predictor wranglers aren’t carefully managing predictions across all the experiment’s divisions, and in both forms: transformed, and inverted (original form).

Related

hyperparameter_hunter.data.data_chunks.prediction_chunks

Defines the prediction data chunk classes, each of which has one counterpart/handler defined in predictors

class hyperparameter_hunter.callbacks.wranglers.predictors.PredictorOOF

Bases: hyperparameter_hunter.callbacks.bases.BasePredictorCallback

Methods

on_exp_end(self)

Perform tasks when an Experiment ends

on_exp_start(self)

Perform tasks when an Experiment is started

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase
on_exp_start(self)

Perform tasks when an Experiment is started

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_exp_end(self)

Perform tasks when an Experiment ends

class hyperparameter_hunter.callbacks.wranglers.predictors.PredictorHoldout

Bases: hyperparameter_hunter.callbacks.bases.BasePredictorCallback

Methods

on_exp_end(self)

Perform tasks when an Experiment ends

on_exp_start(self)

Perform tasks when an Experiment is started

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase
on_exp_start(self)

Perform tasks when an Experiment is started

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_exp_end(self)

Perform tasks when an Experiment ends

class hyperparameter_hunter.callbacks.wranglers.predictors.PredictorTest

Bases: hyperparameter_hunter.callbacks.bases.BasePredictorCallback

Methods

on_exp_end(self)

Perform tasks when an Experiment ends

on_exp_start(self)

Perform tasks when an Experiment is started

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase
on_exp_start(self)

Perform tasks when an Experiment is started

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_exp_end(self)

Perform tasks when an Experiment ends

hyperparameter_hunter.callbacks.wranglers.target_wranglers module

This module defines callbacks that descend from BaseTargetWranglerCallback. Target wrangler callbacks are concerned with managing the target chunks of an experiment’s datasets. This module acts as a liaison between :class:~hyperparameter_hunter.experiments.BaseCVExperiment` and the target chunk classes defined in hyperparameter_hunter.data.data_chunks.target_chunks. Each callback defined herein is responsible for ensuring the proper execution of precisely one descendant of BaseTargetChunk, defined in target_chunks.

Target wranglers are a stark contrast to the relatively boring input wranglers. We need target data for fitting models and for evaluating predictions (which takes place during every “…end” method). Therefore, target wranglers have some mission-critical task to perform on every callback method, especially when feature engineering gets thrown in.

Related

hyperparameter_hunter.data.data_chunks.target_chunks

Defines the target data chunk classes, each of which has one counterpart/handler defined in target_wranglers

class hyperparameter_hunter.callbacks.wranglers.target_wranglers.WranglerTargetTrain

Bases: hyperparameter_hunter.callbacks.bases.BaseTargetWranglerCallback

Methods

on_exp_end(self)

Perform tasks when an Experiment ends

on_exp_start(self)

Perform tasks when an Experiment is started

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase
class hyperparameter_hunter.callbacks.wranglers.target_wranglers.WranglerTargetOOF

Bases: hyperparameter_hunter.callbacks.bases.BaseTargetWranglerCallback

Methods

on_exp_end(self)

Perform tasks when an Experiment ends

on_exp_start(self)

Perform tasks when an Experiment is started

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase
on_exp_start(self)

Perform tasks when an Experiment is started

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_exp_end(self)

Perform tasks when an Experiment ends

class hyperparameter_hunter.callbacks.wranglers.target_wranglers.WranglerTargetHoldout

Bases: hyperparameter_hunter.callbacks.bases.BaseTargetWranglerCallback

Methods

on_exp_end(self)

Perform tasks when an Experiment ends

on_exp_start(self)

Perform tasks when an Experiment is started

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase
on_exp_start(self)

Perform tasks when an Experiment is started

on_rep_start(self)

Perform tasks on repetition start in an Experiment’s repeated cross-validation scheme

on_fold_start(self)

Perform tasks on fold start in an Experiment’s cross-validation scheme

on_run_start(self)

Perform tasks on run start in an Experiment’s multiple-run-averaging phase

on_run_end(self)

Perform tasks on run end in an Experiment’s multiple-run-averaging phase

on_fold_end(self)

Perform tasks on fold end in an Experiment’s cross-validation scheme

on_rep_end(self)

Perform tasks on repetition end in an Experiment’s repeated cross-validation scheme

on_exp_end(self)

Perform tasks when an Experiment ends

Module contents

This module contains the final implementations of the three types of BaseWranglerCallback descendants. The callbacks defined herein act as liaisons between the experiment and its datasets (the datasets’ data chunks). Each callback in the module is expected to be responsible for a specific descendant of hyperparameter_hunter.data.data_core.BaseDataChunk, which can be seen from the type annotation at the forefront of each callback class for its “data”-prefixed attribute

Each callback in the module is actually pulling its dataset (and the appropriate data chunk) from the experiment via its four dataset attributes:

Specifically, each callback herein is responsible for the data chunk denoted by the name of that callback’s immediate parent callback, which is one of the following:

Related

hyperparameter_hunter.data

This module defines the data chunks (attributes of datasets), for which each callback defined in wranglers is responsible. This responsibility is usually satisfied by simply invoking the correct callback method. However, occasionally a data chunk’s callback method will require additional inputs. In these cases, the wrangler callbacks must ensure the proper arguments are provided