Version: Next

Trial-level early stopping

Trial-level early stopping aims to monitor the results of expensive evaluations with timeseries-like data and terminate those that are unlikely to produce promising results prior to completing that evaluation. This reduces computational waste, and enables the same amount of resources to explore more configurations. Early stopping is useful for expensive to evaluate problems where stepwise information is available on the way to the final measurement.

Like the Getting Started tutorial we'll be minimizing the Hartmann6 function, but this time we've modified it to incorporate a new parameter $t$ which allows the function to produce timeseries-like data where the value returned is closer and closer to Hartmann6's true value as $t$ increases. At $t = 100$ the function will simply return Hartmann6's unaltered value.

f(x, t) = hartmann6(x) - log_2(t/100)

While the function is synthetic, the workflow captures the intended principles for this tutorial and is similar to the process of training typical machine learning models.

Learning Objectives

Understand when time-series-like data can be used in an optimization experiment
Run a simple optimization experiment with early stopping
Configure details of an early stopping strategy
Analyze the results of the optimization

Prerequisites

Familiarity with Python and basic programming concepts
Understanding of adaptive experimentation and Bayesian optimization
Getting Started with Ax

Step 1: Import Necessary Modules

First, ensure you have all the necessary imports:

import numpy as np
from ax.api.client import Client
from ax.api.configs import  RangeParameterConfig

Step 2: Initialize the Client

Create an instance of the Client to manage the state of your experiment.

client = Client()

Step 3: Configure the Experiment

The Client instance can be configured with a series of Configs that define how the experiment will be run.

The Hartmann6 problem is usually evaluated on the hypercube $x_i \in (0, 1)$ , so we will define six identical RangeParameterConfigs with these bounds.

You may specify additional features like parameter constraints to further refine the search space and parameter scaling to help navigate parameters with nonuniform effects.

# Define six float parameters for the Hartmann6 function
parameters = [
    RangeParameterConfig(
        name=f"x{i + 1}", parameter_type="float", bounds=(0, 1)
    )
    for i in range(6)
]

client.configure_experiment(parameters=parameters)

Step 4: Configure Optimization

Now, we must configure the objective for this optimization, which we do using Client.configure_optimization. This method expects a string objective, an expression containing either a single metric to maximize, a linear combination of metrics to maximize, or a tuple of multiple metrics to jointly maximize. These expressions are parsed using SymPy. For example:

"score" would direct Ax to maximize a metric named score
"-loss" would direct Ax to Ax to minimize a metric named loss
"task_0 + 0.5 * task_1" would direct Ax to maximize the sum of two task scores, downweighting task_1 by a factor of 0.5
"score, -flops" would direct Ax to simultaneously maximize score while minimizing flops

See these recipes for more information on configuring objectives and outcome constraints.

client.configure_optimization(objective="-hartmann6")

Step 5: Run Trials with early stopping

Here, we will configure the ask-tell loop.

We begin by defining our Hartmann6 function as written above. Remember, this is just an example problem and any Python function can be substituted here.

Then we will iteratively do the following:

Call client.get_next_trials to "ask" Ax for a parameterization to evaluate
Evaluate hartmann6_curve using those parameters in an inner loop to simulate the generation of timeseries data
"Tell" Ax the partial result using client.attach_data
Query whether the trial should be stopped via client.should_stop_trial_early
Stop the underperforming trial and report back to Ax that is has been stopped

This loop will run multiple trials to optimize the function.

Ax will configure an EarlyStoppingStrategy when should_stop_trial_early is called for the first time. By default Ax uses a Percentile early stopping strategy which will terminate a trial early if its performance falls below a percentile threshold when compared to other trials at the same step. Early stopping can only occur after a minimum number of progressions to prevent premature early stopping. This validates that both enough data is gathered to make a decision and there is a minimum number of completed trials with curve data; these completed trials establish a baseline.

# Hartmann6 function
def hartmann6(x1, x2, x3, x4, x5, x6):
    alpha = np.array([1.0, 1.2, 3.0, 3.2])
    A = np.array(
        [
            [10, 3, 17, 3.5, 1.7, 8],
            [0.05, 10, 17, 0.1, 8, 14],
            [3, 3.5, 1.7, 10, 17, 8],
            [17, 8, 0.05, 10, 0.1, 14],
        ]
    )
    P = 10**-4 * np.array(
        [
            [1312, 1696, 5569, 124, 8283, 5886],
            [2329, 4135, 8307, 3736, 1004, 9991],
            [2348, 1451, 3522, 2883, 3047, 6650],
            [4047, 8828, 8732, 5743, 1091, 381],
        ]
    )

    outer = 0.0
    for i in range(4):
        inner = 0.0
        for j, x in enumerate([x1, x2, x3, x4, x5, x6]):
            inner += A[i, j] * (x - P[i, j]) ** 2
        outer += alpha[i] * np.exp(-inner)
    return -outer


# Hartmann6 function with additional t term such that
# hartmann6(X) == hartmann6_curve(X, t=100)
def hartmann6_curve(x1, x2, x3, x4, x5, x6, t):
    return hartmann6(x1, x2, x3, x4, x5, x6) - np.log2(t / 100)


(
    hartmann6(0.1, 0.45, 0.8, 0.25, 0.552, 1.0),
    hartmann6_curve(0.1, 0.45, 0.8, 0.25, 0.552, 1.0, 100),
)

Output:
(np.float64(-0.4878737485613134), np.float64(-0.4878737485613134))

maximum_progressions = 100  # Observe hartmann6_curve over 100 progressions

for _ in range(30):  # Run 30 rounds of trials
    trials = client.get_next_trials(max_trials=3)
    for trial_index, parameters in trials.items():
        for t in range(1, maximum_progressions + 1):
            raw_data = {"hartmann6": hartmann6_curve(t=t, **parameters)}

            # On the final reading call complete_trial and break, else call attach_data
            if t == maximum_progressions:
                client.complete_trial(
                    trial_index=trial_index, raw_data=raw_data, progression=t
                )
                break

            client.attach_data(
                trial_index=trial_index, raw_data=raw_data, progression=t
            )

            # If the trial is underperforming, stop it
            if client.should_stop_trial_early(trial_index=trial_index):
                client.mark_trial_early_stopped(trial_index=trial_index)
                break

Output:
[WARNING 06-11 05:07:27] ax.api.client: 3 trials requested but only 2 could be generated.
[WARNING 06-11 05:07:31] ax.api.client: 3 trials requested but only 1 could be generated.
[INFO 06-11 05:07:31] ax.early_stopping.strategies.percentile: Early stoppinging trial 5: Trial objective value 3.133675076022489 is worse than 50.0-th percentile (3.00844581154208) across comparable trials..
[INFO 06-11 05:07:34] ax.early_stopping.strategies.percentile: Early stoppinging trial 6: Trial objective value 3.146091314642227 is worse than 50.0-th percentile (3.133675076022489) across comparable trials..
[INFO 06-11 05:08:16] ax.early_stopping.strategies.percentile: Early stoppinging trial 15: Trial objective value 2.7590402855248315 is worse than 50.0-th percentile (2.7385766858604805) across comparable trials..
[INFO 06-11 05:08:16] ax.early_stopping.strategies.percentile: Early stoppinging trial 16: Trial objective value 2.913451790268876 is worse than 50.0-th percentile (2.7590402855248315) across comparable trials..
[INFO 06-11 05:08:16] ax.early_stopping.strategies.percentile: Early stoppinging trial 17: Trial objective value 3.2319239220935336 is worse than 50.0-th percentile (2.7878266943549805) across comparable trials..
[INFO 06-11 05:08:21] ax.early_stopping.strategies.percentile: Early stoppinging trial 18: Trial objective value 2.6254749293688815 is worse than 50.0-th percentile (2.437486867851624) across comparable trials..
[INFO 06-11 05:08:21] ax.early_stopping.strategies.percentile: Early stoppinging trial 19: Trial objective value 3.2106268245772838 is worse than 50.0-th percentile (2.789795778151973) across comparable trials..
[INFO 06-11 05:08:21] ax.early_stopping.strategies.percentile: Early stoppinging trial 20: Trial objective value 2.9135648301822563 is worse than 50.0-th percentile (2.816613103185129) across comparable trials..
[INFO 06-11 05:08:26] ax.early_stopping.strategies.percentile: Early stoppinging trial 21: Trial objective value 2.6296742944348668 is worse than 50.0-th percentile (2.5806095624461953) across comparable trials..
[INFO 06-11 05:08:26] ax.early_stopping.strategies.percentile: Early stoppinging trial 22: Trial objective value 3.2035875610002518 is worse than 50.0-th percentile (2.816613103185129) across comparable trials..
[INFO 06-11 05:08:27] ax.early_stopping.strategies.percentile: Early stoppinging trial 23: Trial objective value 2.913149142970136 is worse than 50.0-th percentile (2.8462960854153083) across comparable trials..
[INFO 06-11 05:08:30] ax.early_stopping.strategies.percentile: Early stoppinging trial 24: Trial objective value 2.6314748593285056 is worse than 50.0-th percentile (2.603042245907538) across comparable trials..
[INFO 06-11 05:08:31] ax.early_stopping.strategies.percentile: Early stoppinging trial 25: Trial objective value 3.3076597021246235 is worse than 50.0-th percentile (2.8462960854153083) across comparable trials..
[INFO 06-11 05:08:31] ax.early_stopping.strategies.percentile: Early stoppinging trial 26: Trial objective value 3.2228828104318135 is worse than 50.0-th percentile (2.875979067645487) across comparable trials..
[INFO 06-11 05:08:35] ax.early_stopping.strategies.percentile: Early stoppinging trial 27: Trial objective value 2.4994246180969624 is worse than 50.0-th percentile (2.3119559857677654) across comparable trials..
[INFO 06-11 05:08:35] ax.early_stopping.strategies.percentile: Early stoppinging trial 28: Trial objective value 3.2196935248140486 is worse than 50.0-th percentile (2.875979067645487) across comparable trials..
[INFO 06-11 05:08:36] ax.early_stopping.strategies.percentile: Early stoppinging trial 29: Trial objective value 2.927972280724743 is worse than 50.0-th percentile (2.879597807353579) across comparable trials..
[INFO 06-11 05:08:39] ax.early_stopping.strategies.percentile: Early stoppinging trial 30: Trial objective value 2.498325977293253 is worse than 50.0-th percentile (2.4550786803623366) across comparable trials..
[INFO 06-11 05:08:39] ax.early_stopping.strategies.percentile: Early stoppinging trial 31: Trial objective value 3.2259259924101222 is worse than 50.0-th percentile (2.879597807353579) across comparable trials..
[INFO 06-11 05:08:40] ax.early_stopping.strategies.percentile: Early stoppinging trial 32: Trial objective value 2.9211172930153757 is worse than 50.0-th percentile (2.8832165470616706) across comparable trials..
[INFO 06-11 05:08:43] ax.early_stopping.strategies.percentile: Early stoppinging trial 33: Trial objective value 2.6275775004340662 is worse than 50.0-th percentile (2.624955500180821) across comparable trials..
[INFO 06-11 05:08:43] ax.early_stopping.strategies.percentile: Early stoppinging trial 34: Trial objective value 3.201361745867972 is worse than 50.0-th percentile (2.8832165470616706) across comparable trials..
[INFO 06-11 05:08:44] ax.early_stopping.strategies.percentile: Early stoppinging trial 35: Trial objective value 3.3060837278931174 is worse than 50.0-th percentile (2.8981828450159033) across comparable trials..
[INFO 06-11 05:08:49] ax.early_stopping.strategies.percentile: Early stoppinging trial 36: Trial objective value 2.4976376335745725 is worse than 50.0-th percentile (2.4763581569684545) across comparable trials..
[INFO 06-11 05:08:49] ax.early_stopping.strategies.percentile: Early stoppinging trial 37: Trial objective value 3.200695392898968 is worse than 50.0-th percentile (2.8981828450159033) across comparable trials..
[INFO 06-11 05:08:49] ax.early_stopping.strategies.percentile: Early stoppinging trial 38: Trial objective value 2.9201304650596076 is worse than 50.0-th percentile (2.913149142970136) across comparable trials..
[INFO 06-11 05:08:55] ax.early_stopping.strategies.percentile: Early stoppinging trial 39: Trial objective value 2.6328000814815073 is worse than 50.0-th percentile (2.624955500180821) across comparable trials..
[INFO 06-11 05:08:56] ax.early_stopping.strategies.percentile: Early stoppinging trial 40: Trial objective value 3.221177862731472 is worse than 50.0-th percentile (2.913149142970136) across comparable trials..
[INFO 06-11 05:08:56] ax.early_stopping.strategies.percentile: Early stoppinging trial 41: Trial objective value 2.9185512884341644 is worse than 50.0-th percentile (2.913300466619506) across comparable trials..
[INFO 06-11 05:09:00] ax.early_stopping.strategies.percentile: Early stoppinging trial 42: Trial objective value 2.6288797193867537 is worse than 50.0-th percentile (2.625215214774851) across comparable trials..
[INFO 06-11 05:09:01] ax.early_stopping.strategies.percentile: Early stoppinging trial 43: Trial objective value 3.195905875526558 is worse than 50.0-th percentile (2.913300466619506) across comparable trials..
[INFO 06-11 05:09:01] ax.early_stopping.strategies.percentile: Early stoppinging trial 44: Trial objective value 2.922232300065867 is worse than 50.0-th percentile (2.913451790268876) across comparable trials..
[INFO 06-11 05:09:06] ax.early_stopping.strategies.percentile: Early stoppinging trial 45: Trial objective value 2.6269654847275454 is worse than 50.0-th percentile (2.6254749293688815) across comparable trials..
[INFO 06-11 05:09:06] ax.early_stopping.strategies.percentile: Early stoppinging trial 46: Trial objective value 3.204634139376904 is worse than 50.0-th percentile (2.913451790268876) across comparable trials..
[INFO 06-11 05:09:07] ax.early_stopping.strategies.percentile: Early stoppinging trial 47: Trial objective value 2.7190971387427476 is worse than 50.0-th percentile (2.6262202070482132) across comparable trials..
[INFO 06-11 05:09:10] ax.early_stopping.strategies.percentile: Early stoppinging trial 48: Trial objective value 2.379673461714165 is worse than 50.0-th percentile (2.1964787683478293) across comparable trials..
[INFO 06-11 05:09:11] ax.early_stopping.strategies.percentile: Early stoppinging trial 49: Trial objective value 3.230747974223732 is worse than 50.0-th percentile (2.913300466619506) across comparable trials..
[INFO 06-11 05:09:11] ax.early_stopping.strategies.percentile: Early stoppinging trial 50: Trial objective value 2.9179953993120824 is worse than 50.0-th percentile (2.913451790268876) across comparable trials..
[INFO 06-11 05:09:15] ax.early_stopping.strategies.percentile: Early stoppinging trial 51: Trial objective value 2.627192275848562 is worse than 50.0-th percentile (2.6262202070482132) across comparable trials..
[INFO 06-11 05:09:15] ax.early_stopping.strategies.percentile: Early stoppinging trial 52: Trial objective value 3.22994713999275 is worse than 50.0-th percentile (2.913451790268876) across comparable trials..
[INFO 06-11 05:09:15] ax.early_stopping.strategies.percentile: Early stoppinging trial 53: Trial objective value 2.913503105688045 is worse than 50.0-th percentile (2.9134774479784604) across comparable trials..
[INFO 06-11 05:09:19] ax.early_stopping.strategies.percentile: Early stoppinging trial 54: Trial objective value 2.3796139230417377 is worse than 50.0-th percentile (2.3396014629424005) across comparable trials..
[INFO 06-11 05:09:20] ax.early_stopping.strategies.percentile: Early stoppinging trial 55: Trial objective value 3.2385371215726853 is worse than 50.0-th percentile (2.9134774479784604) across comparable trials..
[INFO 06-11 05:09:20] ax.early_stopping.strategies.percentile: Early stoppinging trial 56: Trial objective value 2.9152774860204627 is worse than 50.0-th percentile (2.913503105688045) across comparable trials..
[INFO 06-11 05:09:25] ax.early_stopping.strategies.percentile: Early stoppinging trial 57: Trial objective value 2.625764176854041 is worse than 50.0-th percentile (2.625619553111461) across comparable trials..
[INFO 06-11 05:09:25] ax.early_stopping.strategies.percentile: Early stoppinging trial 58: Trial objective value 3.2189295826268474 is worse than 50.0-th percentile (2.913503105688045) across comparable trials..
[INFO 06-11 05:09:25] ax.early_stopping.strategies.percentile: Early stoppinging trial 59: Trial objective value 2.9201519575774584 is worse than 50.0-th percentile (2.913533967935151) across comparable trials..
[INFO 06-11 05:09:30] ax.early_stopping.strategies.percentile: Early stoppinging trial 60: Trial objective value 2.6322116129606155 is worse than 50.0-th percentile (2.625764176854041) across comparable trials..
[INFO 06-11 05:09:31] ax.early_stopping.strategies.percentile: Early stoppinging trial 61: Trial objective value 3.2065135915837732 is worse than 50.0-th percentile (2.913533967935151) across comparable trials..
[INFO 06-11 05:09:31] ax.early_stopping.strategies.percentile: Early stoppinging trial 62: Trial objective value 2.918247874566295 is worse than 50.0-th percentile (2.9135648301822563) across comparable trials..
[INFO 06-11 05:09:35] ax.early_stopping.strategies.percentile: Early stoppinging trial 63: Trial objective value 2.497452088030234 is worse than 50.0-th percentile (2.495150679134101) across comparable trials..
[INFO 06-11 05:09:35] ax.early_stopping.strategies.percentile: Early stoppinging trial 64: Trial objective value 3.1885709728375353 is worse than 50.0-th percentile (2.9135648301822563) across comparable trials..
[INFO 06-11 05:09:36] ax.early_stopping.strategies.percentile: Early stoppinging trial 65: Trial objective value 2.9263255797452485 is worse than 50.0-th percentile (2.9144211581013595) across comparable trials..
[INFO 06-11 05:09:41] ax.early_stopping.strategies.percentile: Early stoppinging trial 66: Trial objective value 2.499726974675024 is worse than 50.0-th percentile (2.4963013835821677) across comparable trials..
[INFO 06-11 05:09:41] ax.early_stopping.strategies.percentile: Early stoppinging trial 67: Trial objective value 3.2310192059000014 is worse than 50.0-th percentile (2.9144211581013595) across comparable trials..
[INFO 06-11 05:09:41] ax.early_stopping.strategies.percentile: Early stoppinging trial 68: Trial objective value 2.919057609602588 is worse than 50.0-th percentile (2.9152774860204627) across comparable trials..
[INFO 06-11 05:09:47] ax.early_stopping.strategies.percentile: Early stoppinging trial 69: Trial objective value 2.627874480969324 is worse than 50.0-th percentile (2.625619553111461) across comparable trials..
[INFO 06-11 05:09:47] ax.early_stopping.strategies.percentile: Early stoppinging trial 70: Trial objective value 3.2189281276614237 is worse than 50.0-th percentile (2.9152774860204627) across comparable trials..
[INFO 06-11 05:09:47] ax.early_stopping.strategies.percentile: Early stoppinging trial 71: Trial objective value 2.775804017862453 is worse than 50.0-th percentile (2.625764176854041) across comparable trials..
[INFO 06-11 05:09:52] ax.early_stopping.strategies.percentile: Early stoppinging trial 72: Trial objective value 2.628822246028083 is worse than 50.0-th percentile (2.626364830790793) across comparable trials..
[INFO 06-11 05:09:52] ax.early_stopping.strategies.percentile: Early stoppinging trial 73: Trial objective value 2.736798917122897 is worse than 50.0-th percentile (2.6269654847275454) across comparable trials..
[INFO 06-11 05:09:52] ax.early_stopping.strategies.percentile: Early stoppinging trial 74: Trial objective value 3.2544259272274214 is worse than 50.0-th percentile (2.9135648301822563) across comparable trials..
[INFO 06-11 05:09:56] ax.early_stopping.strategies.percentile: Early stoppinging trial 75: Trial objective value 2.630661593142655 is worse than 50.0-th percentile (2.627078880288054) across comparable trials..
[INFO 06-11 05:09:57] ax.early_stopping.strategies.percentile: Early stoppinging trial 76: Trial objective value 3.296641996002944 is worse than 50.0-th percentile (2.9135648301822563) across comparable trials..
[INFO 06-11 05:09:57] ax.early_stopping.strategies.percentile: Early stoppinging trial 77: Trial objective value 3.212104356548028 is worse than 50.0-th percentile (2.9144211581013595) across comparable trials..
[INFO 06-11 05:10:03] ax.early_stopping.strategies.percentile: Early stoppinging trial 78: Trial objective value 2.4999748706888 is worse than 50.0-th percentile (2.497452088030234) across comparable trials..
[INFO 06-11 05:10:03] ax.early_stopping.strategies.percentile: Early stoppinging trial 79: Trial objective value 3.214079507274765 is worse than 50.0-th percentile (2.9144211581013595) across comparable trials..
[INFO 06-11 05:10:03] ax.early_stopping.strategies.percentile: Early stoppinging trial 80: Trial objective value 2.91612000622411 is worse than 50.0-th percentile (2.9152774860204627) across comparable trials..
[INFO 06-11 05:10:07] ax.early_stopping.strategies.percentile: Early stoppinging trial 81: Trial objective value 2.4991222227501875 is worse than 50.0-th percentile (2.497544860802403) across comparable trials..
[INFO 06-11 05:10:07] ax.early_stopping.strategies.percentile: Early stoppinging trial 82: Trial objective value 3.218961694880014 is worse than 50.0-th percentile (2.9152774860204627) across comparable trials..
[INFO 06-11 05:10:07] ax.early_stopping.strategies.percentile: Early stoppinging trial 83: Trial objective value 3.3188351760503783 is worse than 50.0-th percentile (2.9156987461222865) across comparable trials..
[INFO 06-11 05:10:12] ax.early_stopping.strategies.percentile: Early stoppinging trial 84: Trial objective value 2.6280487666149464 is worse than 50.0-th percentile (2.6269654847275454) across comparable trials..
[INFO 06-11 05:10:12] ax.early_stopping.strategies.percentile: Early stoppinging trial 85: Trial objective value 3.2241255227807826 is worse than 50.0-th percentile (2.9156987461222865) across comparable trials..
[INFO 06-11 05:10:13] ax.early_stopping.strategies.percentile: Early stoppinging trial 86: Trial objective value 2.7780477759441573 is worse than 50.0-th percentile (2.627078880288054) across comparable trials..

Step 6: Analyze Results

After running trials, you can analyze the results. Most commonly this means extracting the parameterization from the best performing trial you conducted.

best_parameters, prediction, index, name = client.get_best_parameterization()
print("Best Parameters:", best_parameters)
print("Prediction (mean, variance):", prediction)

Output:
Best Parameters: {'x1': 0.07267899489395542, 'x2': 0.07520388736930421, 'x3': 0.005006849767324244, 'x4': 0.3408162283572103, 'x5': 0.3212367688019611, 'x6': 0.5262769438863542}
Prediction (mean, variance): {'hartmann6': (np.float64(-1.8428209499963972), np.float64(0.0021192078903913077))}

Step 7: Compute Analyses

Ax can also produce a number of analyses to help interpret the results of the experiment via client.compute_analyses. Users can manually select which analyses to run, or can allow Ax to select which would be most relevant. In this case Ax selects the following:

Parrellel Coordinates Plot shows which parameterizations were evaluated and what metric values were observed -- this is useful for getting a high level overview of how thoroughly the search space was explored and which regions tend to produce which outcomes
Progression Plot shows each partial observation observed by Ax for each trial in a timeseries
Sensitivity Analysis Plot shows which parameters have the largest affect on the objective using Sobol Indicies
Slice Plot shows how the model predicts a single parameter effects the objective along with a confidence interval
Contour Plot shows how the model predicts a pair of parameters effects the objective as a 2D surface
Summary lists all trials generated along with their parameterizations, observations, and miscellaneous metadata
Cross Validation helps to visualize how well the surrogate model is able to predict out of sample points

# display=True instructs Ax to sort then render the resulting analyses
cards = client.compute_analyses(display=True)

Parallel Coordinates for hartmann6

The parallel coordinates plot displays multi-dimensional data by representing each parameter as a parallel axis. This plot helps in assessing how thoroughly the search space has been explored and in identifying patterns or clusterings associated with high-performing (good) or low-performing (bad) arms. By tracing lines across the axes, one can observe correlations and interactions between parameters, gaining insights into the relationships that contribute to the success or failure of different configurations within the experiment.

Summary for Experiment

High-level summary of the Trial-s in this Experiment

	trial_index	arm_name	trial_status	generation_node	hartmann6	x1	x2	x3	x4	x5	x6
0	0	0_0	COMPLETED	CenterOfSearchSpace	-0.505315	0.500000	0.500000	0.500000	0.500000	0.500000	0.500000
1	1	1_0	COMPLETED	Sobol	-0.438712	0.250056	0.554518	0.629723	0.636566	0.087562	0.755922
2	2	2_0	COMPLETED	Sobol	-0.166733	0.601908	0.342159	0.344932	0.082071	0.635074	0.460801
3	3	3_0	COMPLETED	Sobol	-0.037721	0.819525	0.837016	0.923633	0.933356	0.832043	0.144020
4	4	4_0	COMPLETED	Sobol	-1.860616	0.046286	0.047450	0.082211	0.347767	0.382790	0.565148
...	...	...	...	...	...	...	...	...	...	...	...
82	82	82_0	EARLY_STOPPED	MBM	3.218962	0.000000	0.000000	0.033168	0.351509	0.604359	0.244938
83	83	83_0	EARLY_STOPPED	MBM	3.318835	1.000000	1.000000	0.853346	0.308222	1.000000	0.000000
84	84	84_0	EARLY_STOPPED	MBM	2.628049	0.227504	0.000000	0.050082	0.353779	0.205367	0.269705
85	85	85_0	EARLY_STOPPED	MBM	3.224126	0.000000	0.000000	0.034414	0.352595	0.587825	0.214274
86	86	86_0	EARLY_STOPPED	MBM	2.778048	0.000000	0.000000	0.046085	0.334341	0.000000	0.691660

hartmann6 by progression

The progression plot tracks the evolution of each metric over the course of the experiment. This visualization is typically used to monitor the improvement of metrics over Trial iterations, but can also be useful in informing decisions about early stopping for Trials.

Sensitivity Analysis for hartmann6

Understand how each parameter affects hartmann6 according to a second-order sensitivity analysis.

x3, x4 vs. hartmann6

The contour plot visualizes the predicted outcomes for hartmann6 across a two-dimensional parameter space, with other parameters held fixed at their status_quo value (or mean value if status_quo is unavailable). This plot helps in identifying regions of optimal performance and understanding how changes in the selected parameters influence the predicted outcomes. Contour lines represent levels of constant predicted values, providing insights into the gradient and potential optima within the parameter space.

x3 vs. hartmann6

The slice plot provides a one-dimensional view of predicted outcomes for hartmann6 as a function of a single parameter, while keeping all other parameters fixed at their status_quo value (or mean value if status_quo is unavailable). This visualization helps in understanding the sensitivity and impact of changes in the selected parameter on the predicted metric outcomes.

x4 vs. hartmann6

x2, x3 vs. hartmann6

x2, x4 vs. hartmann6

Cross Validation for hartmann6

The cross-validation plot displays the model fit for each metric in the experiment. It employs a leave-one-out approach, where the model is trained on all data except one sample, which is used for validation. The plot shows the predicted outcome for the validation set on the y-axis against its actual value on the x-axis. Points that align closely with the dotted diagonal line indicate a strong model fit, signifying accurate predictions. Additionally, the plot includes 95% confidence intervals that provide insight into the noise in observations and the uncertainty in model predictions. A horizontal, flat line of predictions indicates that the model has not picked up on sufficient signal in the data, and instead is just predicting the mean.

Conclusion

This tutorial demonstates Ax's early stopping capabilities, which utilize timeseries-like data to monitor the results of expensive evaluations and terminate those that are unlikely to produce promising results, freeing up resources to explore more configurations. This can be used in a number of applications, and is especially useful in machine learning contexts.

Learning Objectives​

Prerequisites​

Step 1: Import Necessary Modules​

Step 2: Initialize the Client​

Step 3: Configure the Experiment​

Step 4: Configure Optimization​

Step 5: Run Trials with early stopping​

Step 6: Analyze Results​

Step 7: Compute Analyses​

Conclusion​