ax.early_stopping

Strategies

Base Strategies

class ax.early_stopping.strategies.base.BaseEarlyStoppingStrategy(metric_names: Optional[Iterable[str]] = None, seconds_between_polls: int = 60, true_objective_metric_name: Optional[str] = None)[source]

Bases: abc.ABC, ax.utils.common.base.Base

Interface for heuristics that halt trials early, typically based on early results from that trial.

abstract should_stop_trials_early(trial_indices: Set[int], experiment: ax.core.experiment.Experiment, **kwargs: Dict[str, Any]) → Dict[int, Optional[str]][source]

Decide whether to complete trials before evaluation is fully concluded.

Typical examples include stopping a machine learning model’s training, or halting the gathering of samples before some planned number are collected.

Parameters

  • trial_indices – Indices of candidate trials to stop early.

  • experiment – Experiment that contains the trials and other contextual data.

Returns

A dictionary mapping trial indices that should be early stopped to (optional) messages with the associated reason.

class ax.early_stopping.strategies.base.EarlyStoppingTrainingData(X: numpy.ndarray, Y: numpy.ndarray, Yvar: numpy.ndarray, arm_names: List[Optional[str]])[source]

Bases: object

Dataclass for keeping data arrays related to model training and arm names together.

X: numpy.ndarray
Y: numpy.ndarray
Yvar: numpy.ndarray
arm_names: List[Optional[str]]
class ax.early_stopping.strategies.base.ModelBasedEarlyStoppingStrategy(metric_names: Optional[Iterable[str]] = None, seconds_between_polls: int = 60, true_objective_metric_name: Optional[str] = None)[source]

Bases: ax.early_stopping.strategies.base.BaseEarlyStoppingStrategy

A base class for model based early stopping strategies. Includes a helper function for processing MapData into arrays.

get_training_data(experiment: ax.core.experiment.Experiment, map_data: ax.core.map_data.MapData, keep_every_k_per_arm: Optional[int] = None) → ax.early_stopping.strategies.base.EarlyStoppingTrainingData[source]

Processes the raw (untransformed) training data into arrays for use in modeling.

Parameters

  • experiment – Experiment that contains the data.

  • map_data – The MapData from the experiment, as can be obtained by via _check_validity_and_get_data.

  • Subsample the learning curve by keeping every (keep_every_k_per_arm) – kth entry. Useful for limiting training data for modeling.

Returns

An EarlyStoppingTrainingData that contains training data arrays X, Y,

and Yvar + a list of arm names.

Logical Strategies

class ax.early_stopping.strategies.logical.AndEarlyStoppingStrategy(left: ax.early_stopping.strategies.base.BaseEarlyStoppingStrategy, right: ax.early_stopping.strategies.base.BaseEarlyStoppingStrategy, seconds_between_polls: int = 60, true_objective_metric_name: Optional[str] = None)[source]

Bases: ax.early_stopping.strategies.logical.LogicalEarlyStoppingStrategy

should_stop_trials_early(trial_indices: Set[int], experiment: ax.core.experiment.Experiment, **kwargs: Dict[str, Any]) → Dict[int, Optional[str]][source]

Decide whether to complete trials before evaluation is fully concluded.

Typical examples include stopping a machine learning model’s training, or halting the gathering of samples before some planned number are collected.

Parameters

  • trial_indices – Indices of candidate trials to stop early.

  • experiment – Experiment that contains the trials and other contextual data.

Returns

A dictionary mapping trial indices that should be early stopped to (optional) messages with the associated reason.

class ax.early_stopping.strategies.logical.LogicalEarlyStoppingStrategy(left: ax.early_stopping.strategies.base.BaseEarlyStoppingStrategy, right: ax.early_stopping.strategies.base.BaseEarlyStoppingStrategy, seconds_between_polls: int = 60, true_objective_metric_name: Optional[str] = None)[source]

Bases: ax.early_stopping.strategies.base.BaseEarlyStoppingStrategy

class ax.early_stopping.strategies.logical.OrEarlyStoppingStrategy(left: ax.early_stopping.strategies.base.BaseEarlyStoppingStrategy, right: ax.early_stopping.strategies.base.BaseEarlyStoppingStrategy, seconds_between_polls: int = 60, true_objective_metric_name: Optional[str] = None)[source]

Bases: ax.early_stopping.strategies.logical.LogicalEarlyStoppingStrategy

should_stop_trials_early(trial_indices: Set[int], experiment: ax.core.experiment.Experiment, **kwargs: Dict[str, Any]) → Dict[int, Optional[str]][source]

Decide whether to complete trials before evaluation is fully concluded.

Typical examples include stopping a machine learning model’s training, or halting the gathering of samples before some planned number are collected.

Parameters

  • trial_indices – Indices of candidate trials to stop early.

  • experiment – Experiment that contains the trials and other contextual data.

Returns

A dictionary mapping trial indices that should be early stopped to (optional) messages with the associated reason.

PercentileEarlyStoppingStrategy

class ax.early_stopping.strategies.percentile.PercentileEarlyStoppingStrategy(metric_names: Optional[Iterable[str]] = None, seconds_between_polls: int = 60, true_objective_metric_name: Optional[str] = None, percentile_threshold: float = 50.0, min_progression: float = 0.1, min_curves: float = 5, trial_indices_to_ignore: Optional[List[int]] = None)[source]

Bases: ax.early_stopping.strategies.base.BaseEarlyStoppingStrategy

Implements the strategy of stopping a trial if its performance falls below that of other trials at the same step.

should_stop_trial_early(trial_index: int, experiment: ax.core.experiment.Experiment, df: pandas.DataFrame, minimize: bool) → Tuple[bool, Optional[str]][source]

Stop a trial if its performance is in the bottom percentile_threshold of the trials at the same step.

Parameters

  • trial_index – Indices of candidate trial to stop early.

  • experiment – Experiment that contains the trials and other contextual data.

  • df – Dataframe of partial results after applying interpolation, filtered to objective metric.

  • minimize – Whether objective value is being minimized.

Returns

A tuple (should_stop, reason), where should_stop is True iff the trial should be stopped, and reason is an (optional) string providing information on why the trial should or should not be stopped.

should_stop_trials_early(trial_indices: Set[int], experiment: ax.core.experiment.Experiment, **kwargs: Dict[str, Any]) → Dict[int, Optional[str]][source]

Stop a trial if its performance is in the bottom percentile_threshold of the trials at the same step.

Parameters

  • trial_indices – Indices of candidate trials to consider for early stopping.

  • experiment – Experiment that contains the trials and other contextual data.

Returns

A dictionary mapping trial indices that should be early stopped to (optional) messages with the associated reason. An empty dictionary means no suggested updates to any trial’s status.

ThresholdEarlyStoppingStrategy

class ax.early_stopping.strategies.threshold.ThresholdEarlyStoppingStrategy(metric_names: Optional[Iterable[str]] = None, seconds_between_polls: int = 60, true_objective_metric_name: Optional[str] = None, metric_threshold: float = 0.2, min_progression: float = 10, trial_indices_to_ignore: Optional[List[int]] = None)[source]

Bases: ax.early_stopping.strategies.base.BaseEarlyStoppingStrategy

Implements the strategy of stopping a trial if its performance doesn’t reach a pre-specified threshold by a certain progression.

should_stop_trial_early(trial_index: int, experiment: ax.core.experiment.Experiment, df: pandas.DataFrame, map_key: str, minimize: bool) → Tuple[bool, Optional[str]][source]

Stop a trial if its performance doesn’t reach a pre-specified threshold by min_progression.

Parameters

  • trial_index – Indices of candidate trial to stop early.

  • experiment – Experiment that contains the trials and other contextual data.

  • df – Dataframe of partial results for the objective metric.

  • map_key – Name of the column of the dataset that indicates progression.

  • minimize – Whether objective value is being minimized.

Returns

A tuple (should_stop, reason), where should_stop is True iff the trial should be stopped, and reason is an (optional) string providing information on why the trial should or should not be stopped.

should_stop_trials_early(trial_indices: Set[int], experiment: ax.core.experiment.Experiment, **kwargs: Dict[str, Any]) → Dict[int, Optional[str]][source]

Stop a trial if its performance doesn’t reach a pre-specified threshold by min_progression.

Parameters

  • trial_indices – Indices of candidate trials to consider for early stopping.

  • experiment – Experiment that contains the trials and other contextual data.

Returns

A dictionary mapping trial indices that should be early stopped to (optional) messages with the associated reason. An empty dictionary means no suggested updates to any trial’s status.

Utils

ax.early_stopping.utils.align_partial_results(df: pandas.DataFrame, progr_key: str, metrics: List[str], interpolation: str = 'slinear', do_forward_fill: bool = False) → Tuple[Dict[str, pandas.DataFrame], Dict[str, pandas.DataFrame]][source]

Helper function to align partial results with heterogeneous index

Parameters

  • df – The DataFrame containing the raw data (in long format).

  • progr_key – The key of the column indexing progression (such as the number of training examples, timestamps, etc.).

  • metrics – The names of the metrics to consider.

  • interpolation – The interpolation method used to fill missing values (if applicable). See pandas.DataFrame.interpolate for available options. Limit area is inside.

  • forward_fill – If True, performs a forward fill after interpolation. This is useful for scalarizing learning curves when some data is missing. For instance, suppose we obtain a curve for task_1 for progression in [a, b] and task_2 for progression in [c, d] where b < c. Performing the forward fill on task_1 is a possible solution.

Returns

A two-tuple containing a dict mapping the provided metric names to the index-normalized and interpolated mean (sem).