ax.early_stopping¶
Strategies¶
-
class
ax.early_stopping.strategies.
BaseEarlyStoppingStrategy
[source]¶ Bases:
abc.ABC
Interface for heuristics that halt trials early, typically based on early results from that trial.
-
abstract
should_stop_trials_early
(trial_indices: Set[int], experiment: ax.core.experiment.Experiment, **kwargs: Dict[str, Any]) → Dict[int, Optional[str]][source]¶ Decide whether to complete trials before evaluation is fully concluded.
Typical examples include stopping a machine learning model’s training, or halting the gathering of samples before some planned number are collected.
- Parameters
trial_indices – Indices of candidate trials to stop early.
experiment – Experiment that contains the trials and other contextual data.
- Returns
A dictionary mapping trial indices that should be early stopped to (optional) messages with the associated reason.
-
abstract
-
class
ax.early_stopping.strategies.
PercentileEarlyStoppingStrategy
(percentile_threshold: float = 50.0, min_progression: float = 0.1, min_curves: float = 5)[source]¶ Bases:
ax.early_stopping.strategies.BaseEarlyStoppingStrategy
Implements the strategy of stopping a trial if its performance falls below that of other trials at the same step.
-
should_stop_trial_early
(trial_index: int, experiment: ax.core.experiment.Experiment, df: pandas.DataFrame, percentile_threshold: float, map_key: str, minimize: bool) → Tuple[bool, Optional[str]][source]¶ Stop a trial if its performance is in the bottom percentile_threshold of the trials at the same step.
- Parameters
trial_index – Indices of candidate trial to stop early.
experiment – Experiment that contains the trials and other contextual data.
df – Dataframe of partial results after applying interpolation, filtered to objective metric.
percentile_threshold – Falling below this threshold compared to other trials at the same step will stop the run. Must be between 0.0 and 100.0. e.g. if percentile_threshold=25.0, the bottom 25% of trials are stopped. Note that “bottom” here is determined based on performance, not absolute values; if minimize is False, then “bottom” actually refers to the top trials in terms of metric value.
map_key – Name of the column of the dataset that indicates progression.
minimize – Whether objective value is being minimized.
- Returns
A tuple (should_stop, reason), where should_stop is True iff the trial should be stopped, and reason is an (optional) string providing information on why the trial should or should not be stopped.
-
should_stop_trials_early
(trial_indices: Set[int], experiment: ax.core.experiment.Experiment, **kwargs: Dict[str, Any]) → Dict[int, Optional[str]][source]¶ Stop a trial if its performance is in the bottom percentile_threshold of the trials at the same step.
- Parameters
trial_indices – Indices of candidate trials to consider for early stopping.
experiment – Experiment that contains the trials and other contextual data.
- Returns
A dictionary mapping trial indices that should be early stopped to (optional) messages with the associated reason. An empty dictionary means no suggested updates to any trial’s status.
-
Utils¶
-
ax.early_stopping.utils.
align_partial_results
(df: pandas.DataFrame, progr_key: str, metrics: List[str], interpolation: str = 'slinear') → Tuple[Dict[str, pandas.DataFrame], Dict[str, pandas.DataFrame]][source]¶ Helper function to align partial results with heterogeneous index
- Parameters
df – The DataFrame containining the raw data (in long format).
progr_key – The key of the column indexing progression (such as the number of training examples, timestamps, etc.).
metrics – The names of the metrics to consider.
interpolation – The interpolation method used to fill missing values (if applicable). See pandas.DataFrame.interpolate for available options.
- Returns
A two-tuple containig a dict mapping the provided metric names to the index-normalized and interpolated mean (sem).