ax.early_stopping

Strategies

class ax.early_stopping.strategies.BaseEarlyStoppingStrategy(true_objective_metric_name: Optional[str] = None)[source]

Bases: abc.ABC, ax.utils.common.base.Base

Interface for heuristics that halt trials early, typically based on early results from that trial.

abstract should_stop_trials_early(trial_indices: Set[int], experiment: ax.core.experiment.Experiment, **kwargs: Dict[str, Any]) → Dict[int, Optional[str]][source]

Decide whether to complete trials before evaluation is fully concluded.

Typical examples include stopping a machine learning model’s training, or halting the gathering of samples before some planned number are collected.

Parameters

  • trial_indices – Indices of candidate trials to stop early.

  • experiment – Experiment that contains the trials and other contextual data.

Returns

A dictionary mapping trial indices that should be early stopped to (optional) messages with the associated reason.

property true_objective_metric_name
class ax.early_stopping.strategies.PercentileEarlyStoppingStrategy(true_objective_metric_name: Optional[str] = None, percentile_threshold: float = 50.0, min_progression: float = 0.1, min_curves: float = 5, trial_indices_to_ignore: Optional[List[int]] = None)[source]

Bases: ax.early_stopping.strategies.BaseEarlyStoppingStrategy

Implements the strategy of stopping a trial if its performance falls below that of other trials at the same step.

should_stop_trial_early(trial_index: int, experiment: ax.core.experiment.Experiment, df: pandas.DataFrame, percentile_threshold: float, map_key: str, minimize: bool) → Tuple[bool, Optional[str]][source]

Stop a trial if its performance is in the bottom percentile_threshold of the trials at the same step.

Parameters

  • trial_index – Indices of candidate trial to stop early.

  • experiment – Experiment that contains the trials and other contextual data.

  • df – Dataframe of partial results after applying interpolation, filtered to objective metric.

  • percentile_threshold – Falling below this threshold compared to other trials at the same step will stop the run. Must be between 0.0 and 100.0. e.g. if percentile_threshold=25.0, the bottom 25% of trials are stopped. Note that “bottom” here is determined based on performance, not absolute values; if minimize is False, then “bottom” actually refers to the top trials in terms of metric value.

  • map_key – Name of the column of the dataset that indicates progression.

  • minimize – Whether objective value is being minimized.

Returns

A tuple (should_stop, reason), where should_stop is True iff the trial should be stopped, and reason is an (optional) string providing information on why the trial should or should not be stopped.

should_stop_trials_early(trial_indices: Set[int], experiment: ax.core.experiment.Experiment, **kwargs: Dict[str, Any]) → Dict[int, Optional[str]][source]

Stop a trial if its performance is in the bottom percentile_threshold of the trials at the same step.

Parameters

  • trial_indices – Indices of candidate trials to consider for early stopping.

  • experiment – Experiment that contains the trials and other contextual data.

Returns

A dictionary mapping trial indices that should be early stopped to (optional) messages with the associated reason. An empty dictionary means no suggested updates to any trial’s status.

Utils

ax.early_stopping.utils.align_partial_results(df: pandas.DataFrame, progr_key: str, metrics: List[str], interpolation: str = 'slinear', do_forward_fill: bool = False) → Tuple[Dict[str, pandas.DataFrame], Dict[str, pandas.DataFrame]][source]

Helper function to align partial results with heterogeneous index

Parameters

  • df – The DataFrame containing the raw data (in long format).

  • progr_key – The key of the column indexing progression (such as the number of training examples, timestamps, etc.).

  • metrics – The names of the metrics to consider.

  • interpolation – The interpolation method used to fill missing values (if applicable). See pandas.DataFrame.interpolate for available options. Limit area is inside.

  • forward_fill – If True, performs a forward fill after interpolation. This is useful for scalarizing learning curves when some data is missing. For instance, suppose we obtain a curve for task_1 for progression in [a, b] and task_2 for progression in [c, d] where b < c. Performing the forward fill on task_1 is a possible solution.

Returns

A two-tuple containing a dict mapping the provided metric names to the index-normalized and interpolated mean (sem).