Open in GitHubRun in Google Colab

Trial-level early stopping

Trial-level early stopping aims to monitor the results of expensive evaluations with timeseries-like data and terminate those that are unlikely to produce promising results prior to completing that evaluation. This reduces computational waste, and enables the same amount of resources to explore more configurations. Early stopping is useful for expensive to evaluate problems where stepwise information is available on the way to the final measurement.

Like the Getting Started tutorial we'll be minimizing the Hartmann6 function, but this time we've modified it to incorporate a new parameter $t$ which allows the function to produce timeseries-like data where the value returned is closer and closer to Hartmann6's true value as $t$ increases. At $t = 100$ the function will simply return Hartmann6's unaltered value.

$$f(x, t) = hartmann6(x) - log_2(t/100)$$

While the function is synthetic, the workflow captures the intended principles for this tutorial and is similar to the process of training typical machine learning models.

Learning Objectives

• Understand when time-series-like data can be used in an optimization experiment
• Run a simple optimization experiment with early stopping
• Configure details of an early stopping strategy
• Analyze the results of the optimization

Prerequisites

• Familiarity with Python and basic programming concepts
• Understanding of adaptive experimentation and Bayesian optimization
• Getting Started with Ax

Step 1: Import Necessary Modules

First, ensure you have all the necessary imports:

import numpy as np
from ax.api.client import Client
from ax.api.configs import  RangeParameterConfig

Step 2: Initialize the Client

Create an instance of the Client to manage the state of your experiment.

client = Client()

Step 3: Configure the Experiment

The Client instance can be configured with a series of Configs that define how the experiment will be run.

The Hartmann6 problem is usually evaluated on the hypercube $x_i \in (0, 1)$, so we will define six identical RangeParameterConfigs with these bounds.

You may specify additional features like parameter constraints to further refine the search space and parameter scaling to help navigate parameters with nonuniform effects.

# Define six float parameters for the Hartmann6 function
parameters = [
    RangeParameterConfig(
        name=f"x{i + 1}", parameter_type="float", bounds=(0, 1)
    )
    for i in range(6)
]

client.configure_experiment(parameters=parameters)

Step 4: Configure Optimization

Now, we must configure the objective for this optimization, which we do using Client.configure_optimization. This method expects a string objective, an expression containing either a single metric to maximize, a linear combination of metrics to maximize, or a tuple of multiple metrics to jointly maximize. These expressions are parsed using SymPy. For example:

• "score" would direct Ax to maximize a metric named score
• "-loss" would direct Ax to Ax to minimize a metric named loss
• "task_0 + 0.5 * task_1" would direct Ax to maximize the sum of two task scores, downweighting task_1 by a factor of 0.5
• "score, -flops" would direct Ax to simultaneously maximize score while minimizing flops

See these recipes for more information on configuring objectives and outcome constraints.

client.configure_optimization(objective="-hartmann6")

Step 5: Run Trials with early stopping

Here, we will configure the ask-tell loop.

We begin by defining our Hartmann6 function as written above. Remember, this is just an example problem and any Python function can be substituted here.

Then we will iteratively do the following:

• Call client.get_next_trials to "ask" Ax for a parameterization to evaluate
• Evaluate hartmann6_curve using those parameters in an inner loop to simulate the generation of timeseries data
• "Tell" Ax the partial result using client.attach_data
• Query whether the trial should be stopped via client.should_stop_trial_early
• Stop the underperforming trial and report back to Ax that is has been stopped

This loop will run multiple trials to optimize the function.

Ax will configure an EarlyStoppingStrategy when should_stop_trial_early is called for the first time. By default Ax uses a Percentile early stopping strategy which will terminate a trial early if its performance falls below a percentile threshold when compared to other trials at the same step. Early stopping can only occur after a minimum number of progressions to prevent premature early stopping. This validates that both enough data is gathered to make a decision and there is a minimum number of completed trials with curve data; these completed trials establish a baseline.

# Hartmann6 function
def hartmann6(x1, x2, x3, x4, x5, x6):
    alpha = np.array([1.0, 1.2, 3.0, 3.2])
    A = np.array(
        [
            [10, 3, 17, 3.5, 1.7, 8],
            [0.05, 10, 17, 0.1, 8, 14],
            [3, 3.5, 1.7, 10, 17, 8],
            [17, 8, 0.05, 10, 0.1, 14],
        ]
    )
    P = 10**-4 * np.array(
        [
            [1312, 1696, 5569, 124, 8283, 5886],
            [2329, 4135, 8307, 3736, 1004, 9991],
            [2348, 1451, 3522, 2883, 3047, 6650],
            [4047, 8828, 8732, 5743, 1091, 381],
        ]
    )

    outer = 0.0
    for i in range(4):
        inner = 0.0
        for j, x in enumerate([x1, x2, x3, x4, x5, x6]):
            inner += A[i, j] * (x - P[i, j]) ** 2
        outer += alpha[i] * np.exp(-inner)
    return -outer


# Hartmann6 function with additional t term such that
# hartmann6(X) == hartmann6_curve(X, t=100)
def hartmann6_curve(x1, x2, x3, x4, x5, x6, t):
    return hartmann6(x1, x2, x3, x4, x5, x6) - np.log2(t / 100)


(
    hartmann6(0.1, 0.45, 0.8, 0.25, 0.552, 1.0),
    hartmann6_curve(0.1, 0.45, 0.8, 0.25, 0.552, 1.0, 100),
)

Output:

(np.float64(-0.4878737485613134), np.float64(-0.4878737485613134))

maximum_progressions = 100  # Observe hartmann6_curve over 100 progressions

for _ in range(30):  # Run 30 rounds of trials
    trials = client.get_next_trials(max_trials=3)
    for trial_index, parameters in trials.items():
        for t in range(1, maximum_progressions + 1):
            raw_data = {"hartmann6": hartmann6_curve(t=t, **parameters)}

            # On the final reading call complete_trial and break, else call attach_data
            if t == maximum_progressions:
                client.complete_trial(
                    trial_index=trial_index, raw_data=raw_data, progression=t
                )
                break

            client.attach_data(
                trial_index=trial_index, raw_data=raw_data, progression=t
            )

            # If the trial is underperforming, stop it
            if client.should_stop_trial_early(trial_index=trial_index):
                client.mark_trial_early_stopped(trial_index=trial_index)
                break

Output:

[INFO 01-22 17:46:55] ax.api.client: GenerationStrategy(name='Center+Sobol+MBM:fast', nodes=[CenterGenerationNode(next_node_name='Sobol'), GenerationNode(name='Sobol', generator_specs=[GeneratorSpec(generator_enum=Sobol, generator_key_override=None)], transition_criteria=[MinTrials(transition_to='MBM'), MinTrials(transition_to='MBM')]), GenerationNode(name='MBM', generator_specs=[GeneratorSpec(generator_enum=BoTorch, generator_key_override=None)], transition_criteria=[])]) chosen based on user input and problem structure.
[INFO 01-22 17:46:55] ax.api.client: Generated new trial 0 with parameters {'x1': 0.5, 'x2': 0.5, 'x3': 0.5, 'x4': 0.5, 'x5': 0.5, 'x6': 0.5} using GenerationNode CenterOfSearchSpace.
[INFO 01-22 17:46:55] ax.api.client: Generated new trial 1 with parameters {'x1': 0.718403, 'x2': 0.132347, 'x3': 0.207968, 'x4': 0.886665, 'x5': 0.284014, 'x6': 0.974865} using GenerationNode Sobol.
[INFO 01-22 17:46:55] ax.api.client: Generated new trial 2 with parameters {'x1': 0.061872, 'x2': 0.889394, 'x3': 0.729777, 'x4': 0.336118, 'x5': 0.853096, 'x6': 0.025046} using GenerationNode Sobol.
[INFO 01-22 17:46:57] ax.api.client: Trial 0 marked COMPLETED.
[INFO 01-22 17:47:00] ax.api.client: Trial 1 marked COMPLETED.
[INFO 01-22 17:47:02] ax.api.client: Trial 2 marked COMPLETED.
[INFO 01-22 17:47:02] ax.api.client: Generated new trial 3 with parameters {'x1': 0.334068, 'x2': 0.355383, 'x3': 0.33944, 'x4': 0.545887, 'x5': 0.683735, 'x6': 0.442011} using GenerationNode Sobol.
[INFO 01-22 17:47:02] ax.api.client: Generated new trial 4 with parameters {'x1': 0.928634, 'x2': 0.596501, 'x3': 0.847813, 'x4': 0.2472, 'x5': 0.241579, 'x6': 0.558082} using GenerationNode Sobol.
[WARNING 01-22 17:47:02] ax.api.client: 3 trials requested but only 2 could be generated.
[INFO 01-22 17:47:05] ax.api.client: Trial 3 marked COMPLETED.
[INFO 01-22 17:47:07] ax.api.client: Trial 4 marked COMPLETED.
[INFO 01-22 17:47:08] ax.api.client: Generated new trial 5 with parameters {'x1': 0.531099, 'x2': 0.640472, 'x3': 0.632354, 'x4': 0.400578, 'x5': 0.460304, 'x6': 0.429904} using GenerationNode MBM.
[WARNING 01-22 17:47:08] ax.api.client: 3 trials requested but only 1 could be generated.
[INFO 01-22 17:47:11] ax.api.client: Trial 5 marked COMPLETED.
[INFO 01-22 17:47:13] ax.api.client: Generated new trial 6 with parameters {'x1': 0.750356, 'x2': 0.329126, 'x3': 0.627461, 'x4': 0.298141, 'x5': 0.500433, 'x6': 0.50161} using GenerationNode MBM.
[INFO 01-22 17:47:13] ax.api.client: Generated new trial 7 with parameters {'x1': 0.275347, 'x2': 0.749004, 'x3': 0.452404, 'x4': 0.641738, 'x5': 0.473609, 'x6': 0.460634} using GenerationNode MBM.
[INFO 01-22 17:47:13] ax.api.client: Generated new trial 8 with parameters {'x1': 0.432701, 'x2': 0.40719, 'x3': 0.170885, 'x4': 0.245036, 'x5': 0.467963, 'x6': 0.380129} using GenerationNode MBM.
[INFO 01-22 17:47:16] ax.api.client: Trial 6 marked COMPLETED.
[INFO 01-22 17:47:19] ax.api.client: Trial 7 marked COMPLETED.
[INFO 01-22 17:47:22] ax.api.client: Trial 8 marked COMPLETED.
[INFO 01-22 17:47:23] ax.api.client: Generated new trial 9 with parameters {'x1': 0.407051, 'x2': 0.442102, 'x3': 0.001216, 'x4': 0.092865, 'x5': 0.461547, 'x6': 0.510672} using GenerationNode MBM.
[INFO 01-22 17:47:23] ax.api.client: Generated new trial 10 with parameters {'x1': 0.425484, 'x2': 0.3216, 'x3': 0.163153, 'x4': 0.278378, 'x5': 0.44742, 'x6': 0.193543} using GenerationNode MBM.
[INFO 01-22 17:47:23] ax.api.client: Generated new trial 11 with parameters {'x1': 0.260434, 'x2': 0.320317, 'x3': 0.41841, 'x4': 0.147374, 'x5': 0.459015, 'x6': 0.394352} using GenerationNode MBM.
[INFO 01-22 17:47:26] ax.api.client: Trial 9 marked COMPLETED.
[INFO 01-22 17:47:30] ax.api.client: Trial 10 marked COMPLETED.
[INFO 01-22 17:47:33] ax.api.client: Trial 11 marked COMPLETED.
[INFO 01-22 17:47:34] ax.api.client: Generated new trial 12 with parameters {'x1': 0.205783, 'x2': 0.294406, 'x3': 0.402789, 'x4': 0.09192, 'x5': 0.433598, 'x6': 0.471201} using GenerationNode MBM.
[INFO 01-22 17:47:34] ax.api.client: Generated new trial 13 with parameters {'x1': 0.23622, 'x2': 0.454448, 'x3': 0.412345, 'x4': 0.034435, 'x5': 0.501543, 'x6': 0.406223} using GenerationNode MBM.
[INFO 01-22 17:47:34] ax.api.client: Generated new trial 14 with parameters {'x1': 0.209671, 'x2': 0.002943, 'x3': 0.422663, 'x4': 0.156264, 'x5': 0.406741, 'x6': 0.442104} using GenerationNode MBM.
[INFO 01-22 17:47:37] ax.api.client: Trial 12 marked COMPLETED.
[INFO 01-22 17:47:41] ax.api.client: Trial 13 marked COMPLETED.
[INFO 01-22 17:47:44] ax.api.client: Trial 14 marked COMPLETED.
[INFO 01-22 17:47:46] ax.api.client: Generated new trial 15 with parameters {'x1': 0.074579, 'x2': 0.085759, 'x3': 0.526608, 'x4': 0.06684, 'x5': 0.39016, 'x6': 0.483323} using GenerationNode MBM.
[INFO 01-22 17:47:46] ax.api.client: Generated new trial 16 with parameters {'x1': 0.074433, 'x2': 0.045077, 'x3': 0.435176, 'x4': 0.701566, 'x5': 0.407006, 'x6': 0.495897} using GenerationNode MBM.
[INFO 01-22 17:47:46] ax.api.client: Generated new trial 17 with parameters {'x1': 0.246444, 'x2': 0.111247, 'x3': 0.479566, 'x4': 0.0, 'x5': 0.370025, 'x6': 0.463757} using GenerationNode MBM.
[INFO 01-22 17:47:49] ax.api.client: Trial 15 marked COMPLETED.
[INFO 01-22 17:47:50] ax.api.client: Trial 16 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [2.9075914866283235]
- Thresholds: [2.8341269022978586]
- Number of trials: [17]
- Trial indices: [[0, 15, 14, 13, 12, 11, 9, 10, 8, 7, 6, 2, 3, 4, 5, 1, 16]]
[INFO 01-22 17:47:50] ax.api.client: Trial 16 marked EARLY_STOPPED.
[INFO 01-22 17:47:53] ax.api.client: Trial 17 marked COMPLETED.
[INFO 01-22 17:47:55] ax.api.client: Generated new trial 18 with parameters {'x1': 0.0, 'x2': 0.0, 'x3': 0.626423, 'x4': 0.0, 'x5': 0.428476, 'x6': 0.51528} using GenerationNode MBM.
[INFO 01-22 17:47:55] ax.api.client: Generated new trial 19 with parameters {'x1': 0.0, 'x2': 0.0, 'x3': 0.645295, 'x4': 0.956716, 'x5': 0.426045, 'x6': 0.523305} using GenerationNode MBM.
[INFO 01-22 17:47:55] ax.api.client: Generated new trial 20 with parameters {'x1': 0.0, 'x2': 0.0, 'x3': 0.923922, 'x4': 0.0, 'x5': 0.425133, 'x6': 0.484847} using GenerationNode MBM.
[INFO 01-22 17:47:59] ax.api.client: Trial 18 marked COMPLETED.
[INFO 01-22 17:48:00] ax.api.client: Trial 19 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.2824051125269325]
- Thresholds: [2.8253700027414936]
- Number of trials: [20]
- Trial indices: [[10, 1, 18, 15, 4, 3, 7, 5, 14, 9, 16, 2, 12, 8, 17, 11, 13, 0, 19, 6]]
[INFO 01-22 17:48:00] ax.api.client: Trial 19 marked EARLY_STOPPED.
[INFO 01-22 17:48:00] ax.api.client: Trial 20 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [2.9098210075985778]
- Thresholds: [2.8341269022978586]
- Number of trials: [21]
- Trial indices: [[19, 16, 11, 3, 9, 15, 1, 17, 4, 12, 5, 18, 8, 2, 6, 14, 10, 7, 13, 0, 20]]
[INFO 01-22 17:48:00] ax.api.client: Trial 20 marked EARLY_STOPPED.
[INFO 01-22 17:48:01] ax.api.client: Generated new trial 21 with parameters {'x1': 0.0, 'x2': 0.020417, 'x3': 0.385432, 'x4': 0.21224, 'x5': 0.326819, 'x6': 0.598123} using GenerationNode MBM.
[INFO 01-22 17:48:01] ax.api.client: Generated new trial 22 with parameters {'x1': 0.0, 'x2': 0.0, 'x3': 0.396394, 'x4': 0.226104, 'x5': 0.339068, 'x6': 0.746521} using GenerationNode MBM.
[INFO 01-22 17:48:01] ax.api.client: Generated new trial 23 with parameters {'x1': 0.0, 'x2': 0.134266, 'x3': 0.389411, 'x4': 0.204798, 'x5': 0.29105, 'x6': 0.479872} using GenerationNode MBM.
[INFO 01-22 17:48:05] ax.api.client: Trial 21 marked COMPLETED.
[INFO 01-22 17:48:09] ax.api.client: Trial 22 marked COMPLETED.
[INFO 01-22 17:48:13] ax.api.client: Trial 23 marked COMPLETED.
[INFO 01-22 17:48:14] ax.api.client: Generated new trial 24 with parameters {'x1': 0.0, 'x2': 0.0, 'x3': 0.25471, 'x4': 0.200008, 'x5': 0.293009, 'x6': 0.713689} using GenerationNode MBM.
[INFO 01-22 17:48:14] ax.api.client: Generated new trial 25 with parameters {'x1': 0.0, 'x2': 0.0, 'x3': 0.443843, 'x4': 0.294606, 'x5': 0.294598, 'x6': 0.681797} using GenerationNode MBM.
[INFO 01-22 17:48:14] ax.api.client: Generated new trial 26 with parameters {'x1': 0.0, 'x2': 0.0, 'x3': 0.274999, 'x4': 0.118765, 'x5': 0.217066, 'x6': 0.792623} using GenerationNode MBM.
[INFO 01-22 17:48:18] ax.api.client: Trial 24 marked COMPLETED.
[INFO 01-22 17:48:22] ax.api.client: Trial 25 marked COMPLETED.
[INFO 01-22 17:48:27] ax.api.client: Trial 26 marked COMPLETED.
[INFO 01-22 17:48:28] ax.api.client: Generated new trial 27 with parameters {'x1': 0.0, 'x2': 0.0, 'x3': 0.500051, 'x4': 0.415942, 'x5': 0.325056, 'x6': 0.657771} using GenerationNode MBM.
[INFO 01-22 17:48:28] ax.api.client: Generated new trial 28 with parameters {'x1': 0.0, 'x2': 0.0, 'x3': 0.704684, 'x4': 0.439055, 'x5': 0.312569, 'x6': 0.683247} using GenerationNode MBM.
[INFO 01-22 17:48:28] ax.api.client: Generated new trial 29 with parameters {'x1': 0.0, 'x2': 0.23513, 'x3': 0.519121, 'x4': 0.376485, 'x5': 0.317849, 'x6': 0.70767} using GenerationNode MBM.
[INFO 01-22 17:48:32] ax.api.client: Trial 27 marked COMPLETED.
[INFO 01-22 17:48:37] ax.api.client: Trial 28 marked COMPLETED.
[INFO 01-22 17:48:41] ax.api.client: Trial 29 marked COMPLETED.
[INFO 01-22 17:48:43] ax.api.client: Generated new trial 30 with parameters {'x1': 0.0, 'x2': 0.938468, 'x3': 0.422884, 'x4': 0.303275, 'x5': 0.309608, 'x6': 0.735559} using GenerationNode MBM.
[INFO 01-22 17:48:43] ax.api.client: Generated new trial 31 with parameters {'x1': 0.0, 'x2': 0.0, 'x3': 0.408336, 'x4': 0.301924, 'x5': 0.326915, 'x6': 0.706133} using GenerationNode MBM.
[INFO 01-22 17:48:43] ax.api.client: Generated new trial 32 with parameters {'x1': 0.0, 'x2': 1.0, 'x3': 0.379885, 'x4': 0.381209, 'x5': 0.270545, 'x6': 0.765502} using GenerationNode MBM.
[INFO 01-22 17:48:43] ax.api.client: Trial 30 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.0119969992398294]
- Thresholds: [2.63007768920506]
- Number of trials: [31]
- Trial indices: [[0, 8, 28, 5, 18, 11, 4, 22, 15, 3, 14, 17, 1, 24, 12, 25, 20, 19, 7, 29, 13, 30, 21, 9, 16, 10, 23, 26, 27, 2, 6]]
[INFO 01-22 17:48:43] ax.api.client: Trial 30 marked EARLY_STOPPED.
[INFO 01-22 17:48:48] ax.api.client: Trial 31 marked COMPLETED.
[INFO 01-22 17:48:48] ax.api.client: Trial 32 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.140452519264649]
- Thresholds: [2.63007768920506]
- Number of trials: [33]
- Trial indices: [[0, 29, 5, 4, 7, 6, 28, 8, 9, 25, 10, 11, 3, 26, 27, 20, 23, 22, 19, 18, 31, 30, 21, 16, 14, 17, 2, 15, 13, 24, 1, 12, 32]]
[INFO 01-22 17:48:48] ax.api.client: Trial 32 marked EARLY_STOPPED.
[INFO 01-22 17:48:49] ax.api.client: Generated new trial 33 with parameters {'x1': 0.0, 'x2': 1.0, 'x3': 0.347111, 'x4': 0.38655, 'x5': 0.383324, 'x6': 0.790846} using GenerationNode MBM.
[INFO 01-22 17:48:49] ax.api.client: Generated new trial 34 with parameters {'x1': 0.0, 'x2': 1.0, 'x3': 0.505958, 'x4': 0.489992, 'x5': 0.131057, 'x6': 0.905691} using GenerationNode MBM.
[INFO 01-22 17:48:49] ax.api.client: Generated new trial 35 with parameters {'x1': 0.196118, 'x2': 0.397471, 'x3': 0.423135, 'x4': 0.293296, 'x5': 0.283777, 'x6': 0.699847} using GenerationNode MBM.
[INFO 01-22 17:48:50] ax.api.client: Trial 33 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.1639702289136125]
- Thresholds: [2.647557564917772]
- Number of trials: [34]
- Trial indices: [[5, 30, 29, 15, 28, 14, 6, 3, 27, 1, 17, 4, 26, 18, 7, 16, 13, 9, 25, 24, 10, 23, 21, 19, 8, 31, 0, 11, 20, 12, 2, 22, 32, 33]]
[INFO 01-22 17:48:50] ax.api.client: Trial 33 marked EARLY_STOPPED.
[INFO 01-22 17:48:50] ax.api.client: Trial 34 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.2598870160643836]
- Thresholds: [2.6650374406304835]
- Number of trials: [35]
- Trial indices: [[0, 30, 1, 12, 31, 7, 19, 20, 8, 6, 27, 9, 5, 28, 13, 18, 14, 4, 15, 17, 22, 21, 25, 11, 3, 26, 24, 33, 23, 2, 10, 16, 29, 32, 34]]
[INFO 01-22 17:48:50] ax.api.client: Trial 34 marked EARLY_STOPPED.
[INFO 01-22 17:48:55] ax.api.client: Trial 35 marked COMPLETED.
[INFO 01-22 17:48:56] ax.api.client: Generated new trial 36 with parameters {'x1': 0.152477, 'x2': 0.053364, 'x3': 0.477882, 'x4': 0.233726, 'x5': 0.0, 'x6': 0.621152} using GenerationNode MBM.
[INFO 01-22 17:48:56] ax.api.client: Generated new trial 37 with parameters {'x1': 0.0, 'x2': 1.0, 'x3': 0.182508, 'x4': 0.55444, 'x5': 0.966674, 'x6': 0.941859} using GenerationNode MBM.
[INFO 01-22 17:48:56] ax.api.client: Generated new trial 38 with parameters {'x1': 0.0, 'x2': 1.0, 'x3': 0.505145, 'x4': 0.290498, 'x5': 0.0, 'x6': 0.671247} using GenerationNode MBM.
[INFO 01-22 17:48:57] ax.api.client: Trial 36 should be stopped early: Trial objective values at progressions in [11.00, 11.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [11.0]
- Underperforms: [True]
- Trial objective values: [2.4078069356951044]
- Thresholds: [1.9511699107179665]
- Number of trials: [30]
- Trial indices: [[0, 35, 7, 6, 1, 11, 29, 3, 10, 26, 23, 18, 31, 24, 14, 8, 27, 12, 22, 17, 4, 2, 28, 5, 21, 9, 15, 25, 36, 13]]
[INFO 01-22 17:48:57] ax.api.client: Trial 36 marked EARLY_STOPPED.
[INFO 01-22 17:48:57] ax.api.client: Trial 37 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.3206450787934823]
- Thresholds: [2.647557564917772]
- Number of trials: [38]
- Trial indices: [[0, 37, 21, 19, 15, 30, 14, 29, 22, 27, 13, 35, 4, 31, 28, 5, 6, 34, 18, 32, 12, 9, 8, 23, 24, 1, 7, 2, 10, 26, 17, 16, 25, 11, 20, 36, 3, 33]]
[INFO 01-22 17:48:57] ax.api.client: Trial 37 marked EARLY_STOPPED.
[INFO 01-22 17:48:58] ax.api.client: Trial 38 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.258438886447086]
- Thresholds: [2.6650374406304835]
- Number of trials: [39]
- Trial indices: [[0, 36, 34, 6, 35, 10, 9, 20, 22, 23, 3, 21, 32, 11, 1, 33, 19, 12, 30, 28, 15, 17, 26, 7, 25, 18, 5, 29, 24, 13, 31, 8, 4, 14, 16, 38, 37, 27, 2]]
[INFO 01-22 17:48:58] ax.api.client: Trial 38 marked EARLY_STOPPED.
[INFO 01-22 17:48:59] ax.api.client: Generated new trial 39 with parameters {'x1': 0.031161, 'x2': 0.422323, 'x3': 0.395613, 'x4': 0.323527, 'x5': 0.0, 'x6': 0.679664} using GenerationNode MBM.
[INFO 01-22 17:48:59] ax.api.client: Generated new trial 40 with parameters {'x1': 0.0, 'x2': 1.0, 'x3': 0.033512, 'x4': 0.552666, 'x5': 0.0, 'x6': 0.79887} using GenerationNode MBM.
[INFO 01-22 17:48:59] ax.api.client: Generated new trial 41 with parameters {'x1': 0.0, 'x2': 1.0, 'x3': 0.52277, 'x4': 0.793772, 'x5': 0.0, 'x6': 0.0} using GenerationNode MBM.
[INFO 01-22 17:49:00] ax.api.client: Trial 39 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [2.8010495820717614]
- Thresholds: [2.6878742498702612]
- Number of trials: [40]
- Trial indices: [[9, 25, 4, 37, 24, 3, 10, 35, 26, 22, 23, 21, 20, 27, 34, 8, 32, 33, 1, 5, 28, 29, 7, 13, 6, 17, 2, 18, 16, 12, 11, 19, 31, 30, 15, 14, 0, 38, 39, 36]]
[INFO 01-22 17:49:00] ax.api.client: Trial 39 marked EARLY_STOPPED.
[INFO 01-22 17:49:00] ax.api.client: Trial 40 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.307068701209497]
- Thresholds: [2.7107110591100385]
- Number of trials: [41]
- Trial indices: [[36, 0, 39, 8, 27, 16, 12, 4, 18, 28, 1, 9, 17, 20, 19, 6, 23, 11, 3, 21, 25, 31, 24, 7, 30, 15, 14, 10, 13, 35, 26, 29, 33, 34, 32, 2, 5, 40, 37, 38, 22]]
[INFO 01-22 17:49:00] ax.api.client: Trial 40 marked EARLY_STOPPED.
[INFO 01-22 17:49:01] ax.api.client: Trial 41 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.215181061037444]
- Thresholds: [2.7558803205909]
- Number of trials: [42]
- Trial indices: [[0, 10, 38, 4, 39, 36, 5, 37, 20, 21, 29, 1, 19, 28, 9, 27, 16, 26, 12, 13, 6, 15, 34, 22, 14, 17, 30, 2, 33, 35, 32, 3, 24, 18, 41, 40, 11, 7, 8, 23, 31, 25]]
[INFO 01-22 17:49:01] ax.api.client: Trial 41 marked EARLY_STOPPED.
[INFO 01-22 17:49:02] ax.api.client: Generated new trial 42 with parameters {'x1': 0.325519, 'x2': 1.0, 'x3': 0.614128, 'x4': 0.402341, 'x5': 1.0, 'x6': 0.971642} using GenerationNode MBM.
[INFO 01-22 17:49:02] ax.api.client: Generated new trial 43 with parameters {'x1': 0.977645, 'x2': 1.0, 'x3': 1.0, 'x4': 0.969692, 'x5': 0.0, 'x6': 0.0} using GenerationNode MBM.
[INFO 01-22 17:49:02] ax.api.client: Generated new trial 44 with parameters {'x1': 1.0, 'x2': 1.0, 'x3': 0.0, 'x4': 1.0, 'x5': 0.0, 'x6': 0.0} using GenerationNode MBM.
[INFO 01-22 17:49:03] ax.api.client: Trial 42 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.307720281315857]
- Thresholds: [2.8010495820717614]
- Number of trials: [43]
- Trial indices: [[0, 17, 41, 40, 39, 30, 4, 5, 38, 1, 29, 36, 15, 37, 28, 6, 13, 27, 20, 19, 14, 22, 12, 21, 18, 35, 33, 11, 7, 2, 3, 32, 25, 16, 34, 10, 8, 26, 24, 9, 23, 42, 31]]
[INFO 01-22 17:49:03] ax.api.client: Trial 42 marked EARLY_STOPPED.
[INFO 01-22 17:49:03] ax.api.client: Trial 43 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.3197127281268495]
- Thresholds: [2.8088313426284452]
- Number of trials: [44]
- Trial indices: [[0, 41, 40, 4, 38, 39, 1, 11, 9, 14, 27, 32, 17, 23, 30, 5, 37, 16, 13, 25, 12, 2, 18, 36, 8, 33, 31, 26, 3, 22, 7, 28, 21, 19, 15, 42, 20, 10, 29, 34, 24, 6, 35, 43]]
[INFO 01-22 17:49:03] ax.api.client: Trial 43 marked EARLY_STOPPED.
[INFO 01-22 17:49:04] ax.api.client: Trial 44 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.320860044276492]
- Thresholds: [2.816613103185129]
- Number of trials: [45]
- Trial indices: [[0, 43, 10, 12, 34, 4, 5, 33, 32, 18, 35, 23, 17, 6, 41, 19, 25, 1, 40, 39, 3, 21, 2, 31, 13, 27, 22, 15, 24, 30, 16, 20, 7, 9, 29, 26, 14, 8, 38, 37, 36, 42, 28, 11, 44]]
[INFO 01-22 17:49:04] ax.api.client: Trial 44 marked EARLY_STOPPED.
[INFO 01-22 17:49:05] ax.api.client: Generated new trial 45 with parameters {'x1': 0.379815, 'x2': 0.0, 'x3': 1.0, 'x4': 0.649887, 'x5': 0.0, 'x6': 0.0} using GenerationNode MBM.
[INFO 01-22 17:49:05] ax.api.client: Generated new trial 46 with parameters {'x1': 0.0, 'x2': 1.0, 'x3': 0.0, 'x4': 0.844752, 'x5': 0.0, 'x6': 0.0} using GenerationNode MBM.
[INFO 01-22 17:49:05] ax.api.client: Generated new trial 47 with parameters {'x1': 0.425447, 'x2': 0.0, 'x3': 1.0, 'x4': 0.164966, 'x5': 0.0, 'x6': 0.0} using GenerationNode MBM.
[INFO 01-22 17:49:06] ax.api.client: Trial 45 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.3140319242890333]
- Thresholds: [2.8253700027414936]
- Number of trials: [46]
- Trial indices: [[0, 34, 5, 43, 22, 27, 14, 9, 42, 44, 15, 1, 40, 33, 3, 8, 36, 41, 19, 35, 11, 2, 7, 23, 20, 25, 24, 29, 10, 31, 21, 30, 37, 4, 26, 16, 17, 38, 13, 39, 6, 32, 12, 28, 45, 18]]
[INFO 01-22 17:49:06] ax.api.client: Trial 45 marked EARLY_STOPPED.
[INFO 01-22 17:49:06] ax.api.client: Trial 46 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.241594018873626]
- Thresholds: [2.8341269022978586]
- Number of trials: [47]
- Trial indices: [[0, 44, 7, 32, 8, 9, 23, 45, 6, 33, 43, 34, 30, 31, 1, 35, 10, 12, 11, 5, 29, 22, 28, 26, 4, 21, 14, 13, 24, 25, 18, 20, 15, 41, 37, 3, 36, 16, 17, 38, 27, 40, 2, 46, 19, 39, 42]]
[INFO 01-22 17:49:06] ax.api.client: Trial 46 marked EARLY_STOPPED.
[INFO 01-22 17:49:07] ax.api.client: Trial 47 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.3142324971111656]
- Thresholds: [2.847823551021131]
- Number of trials: [48]
- Trial indices: [[34, 7, 16, 36, 18, 4, 17, 45, 46, 37, 28, 38, 3, 15, 8, 44, 1, 27, 19, 14, 20, 39, 35, 12, 29, 21, 31, 23, 13, 26, 24, 9, 22, 5, 6, 42, 10, 41, 11, 2, 30, 43, 40, 47, 33, 32, 25, 0]]
[INFO 01-22 17:49:07] ax.api.client: Trial 47 marked EARLY_STOPPED.
[INFO 01-22 17:49:08] ax.api.client: Generated new trial 48 with parameters {'x1': 0.0, 'x2': 0.0, 'x3': 1.0, 'x4': 0.937772, 'x5': 0.0, 'x6': 0.0} using GenerationNode MBM.
[INFO 01-22 17:49:08] ax.api.client: Generated new trial 49 with parameters {'x1': 0.0, 'x2': 1.0, 'x3': 0.359744, 'x4': 1.0, 'x5': 1.0, 'x6': 0.0} using GenerationNode MBM.
[INFO 01-22 17:49:08] ax.api.client: Generated new trial 50 with parameters {'x1': 0.0, 'x2': 1.0, 'x3': 0.3584, 'x4': 0.39651, 'x5': 0.0, 'x6': 0.741623} using GenerationNode MBM.
[INFO 01-22 17:49:09] ax.api.client: Trial 48 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.3216979429995503]
- Thresholds: [2.861520199744403]
- Number of trials: [49]
- Trial indices: [[0, 39, 24, 46, 23, 26, 33, 34, 47, 40, 36, 22, 9, 38, 1, 45, 7, 6, 31, 12, 17, 15, 16, 19, 11, 10, 32, 3, 18, 28, 20, 21, 29, 5, 30, 13, 25, 37, 4, 27, 42, 14, 35, 8, 2, 41, 44, 43, 48]]
[INFO 01-22 17:49:09] ax.api.client: Trial 48 marked EARLY_STOPPED.
[INFO 01-22 17:49:09] ax.api.client: Trial 49 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.295987703927376]
- Thresholds: [2.884555843186363]
- Number of trials: [50]
- Trial indices: [[0, 48, 26, 11, 10, 24, 44, 46, 14, 27, 1, 45, 23, 28, 13, 9, 29, 7, 8, 22, 41, 25, 12, 43, 42, 15, 2, 40, 31, 30, 39, 6, 35, 18, 16, 34, 33, 32, 21, 19, 47, 49, 4, 38, 36, 20, 37, 3, 5, 17]]
[INFO 01-22 17:49:09] ax.api.client: Trial 49 marked EARLY_STOPPED.
[INFO 01-22 17:49:10] ax.api.client: Trial 50 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.2786124653475235]
- Thresholds: [2.9075914866283235]
- Number of trials: [51]
- Trial indices: [[0, 21, 35, 5, 49, 47, 32, 34, 45, 33, 20, 6, 23, 1, 31, 46, 30, 19, 24, 7, 8, 43, 17, 4, 18, 10, 2, 36, 42, 29, 9, 44, 26, 12, 27, 11, 25, 14, 37, 13, 48, 3, 50, 22, 39, 38, 40, 41, 16, 15, 28]]
[INFO 01-22 17:49:10] ax.api.client: Trial 50 marked EARLY_STOPPED.
[INFO 01-22 17:49:12] ax.api.client: Generated new trial 51 with parameters {'x1': 0.189996, 'x2': 0.0, 'x3': 0.39214, 'x4': 0.279869, 'x5': 0.0, 'x6': 0.66503} using GenerationNode MBM.
[INFO 01-22 17:49:12] ax.api.client: Generated new trial 52 with parameters {'x1': 1.0, 'x2': 1.0, 'x3': 0.918065, 'x4': 1.0, 'x5': 1.0, 'x6': 0.0} using GenerationNode MBM.
[INFO 01-22 17:49:12] ax.api.client: Generated new trial 53 with parameters {'x1': 0.778579, 'x2': 1.0, 'x3': 0.608655, 'x4': 0.475213, 'x5': 0.0, 'x6': 1.0} using GenerationNode MBM.
[INFO 01-22 17:49:13] ax.api.client: Trial 51 should be stopped early: Trial objective values at progressions in [11.00, 11.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [11.0]
- Underperforms: [True]
- Trial objective values: [2.483957996902248]
- Thresholds: [2.053156744263546]
- Number of trials: [31]
- Trial indices: [[0, 6, 35, 1, 27, 5, 3, 22, 8, 7, 14, 10, 4, 24, 36, 23, 26, 25, 12, 28, 21, 31, 15, 2, 9, 13, 29, 18, 11, 51, 17]]
[INFO 01-22 17:49:13] ax.api.client: Trial 51 marked EARLY_STOPPED.
[INFO 01-22 17:49:13] ax.api.client: Trial 52 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.32090325079093]
- Thresholds: [2.9075914866283235]
- Number of trials: [53]
- Trial indices: [[52, 50, 38, 15, 51, 14, 36, 5, 23, 40, 4, 28, 49, 39, 1, 37, 18, 22, 30, 47, 8, 34, 31, 12, 17, 29, 13, 16, 6, 24, 41, 7, 48, 35, 27, 10, 9, 21, 19, 44, 33, 46, 11, 32, 45, 25, 26, 0, 20, 42, 3, 43, 2]]
[INFO 01-22 17:49:13] ax.api.client: Trial 52 marked EARLY_STOPPED.
[INFO 01-22 17:49:14] ax.api.client: Trial 53 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.301672364204989]
- Thresholds: [2.9087062471134506]
- Number of trials: [54]
- Trial indices: [[26, 53, 27, 25, 38, 6, 36, 8, 9, 51, 7, 39, 52, 35, 5, 41, 40, 50, 1, 33, 34, 32, 31, 43, 42, 23, 12, 44, 11, 4, 24, 10, 13, 28, 14, 30, 46, 3, 22, 15, 45, 21, 18, 16, 20, 48, 17, 29, 2, 49, 47, 19, 0, 37]]
[INFO 01-22 17:49:14] ax.api.client: Trial 53 marked EARLY_STOPPED.
[INFO 01-22 17:49:16] ax.api.client: Generated new trial 54 with parameters {'x1': 0.0, 'x2': 1.0, 'x3': 1.0, 'x4': 0.390374, 'x5': 0.0, 'x6': 1.0} using GenerationNode MBM.
[INFO 01-22 17:49:16] ax.api.client: Generated new trial 55 with parameters {'x1': 0.0, 'x2': 1.0, 'x3': 0.0, 'x4': 0.447673, 'x5': 0.0, 'x6': 1.0} using GenerationNode MBM.
[INFO 01-22 17:49:16] ax.api.client: Generated new trial 56 with parameters {'x1': 0.0, 'x2': 0.361972, 'x3': 1.0, 'x4': 0.311247, 'x5': 0.0, 'x6': 0.816885} using GenerationNode MBM.
[INFO 01-22 17:49:16] ax.api.client: Trial 54 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.2926835311550415]
- Thresholds: [2.9098210075985778]
- Number of trials: [55]
- Trial indices: [[0, 7, 52, 47, 4, 26, 35, 28, 53, 42, 46, 14, 34, 1, 51, 13, 40, 23, 8, 39, 17, 36, 15, 24, 6, 41, 38, 12, 32, 33, 25, 29, 11, 44, 9, 37, 20, 45, 10, 21, 16, 50, 3, 5, 54, 43, 19, 27, 18, 48, 49, 31, 30, 22, 2]]
[INFO 01-22 17:49:16] ax.api.client: Trial 54 marked EARLY_STOPPED.
[INFO 01-22 17:49:17] ax.api.client: Trial 55 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.3115028601822725]
- Thresholds: [2.94131395757758]
- Number of trials: [56]
- Trial indices: [[44, 42, 17, 28, 43, 8, 30, 35, 53, 4, 27, 41, 18, 54, 1, 46, 16, 45, 9, 39, 34, 10, 14, 15, 25, 19, 26, 49, 6, 31, 24, 52, 23, 33, 20, 22, 47, 48, 7, 55, 21, 12, 0, 36, 29, 5, 32, 50, 51, 3, 38, 13, 37, 2, 11, 40]]
[INFO 01-22 17:49:17] ax.api.client: Trial 55 marked EARLY_STOPPED.
[INFO 01-22 17:49:18] ax.api.client: Trial 56 should be stopped early: Trial objective values at progressions in [11.00, 11.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [11.0]
- Underperforms: [True]
- Trial objective values: [2.5853114247799334]
- Thresholds: [2.066873663892438]
- Number of trials: [32]
- Trial indices: [[0, 56, 35, 13, 14, 23, 36, 3, 31, 51, 12, 27, 25, 29, 1, 22, 15, 6, 28, 26, 9, 4, 21, 8, 24, 17, 2, 11, 5, 10, 18, 7]]
[INFO 01-22 17:49:18] ax.api.client: Trial 56 marked EARLY_STOPPED.
[INFO 01-22 17:49:19] ax.api.client: Generated new trial 57 with parameters {'x1': 0.0, 'x2': 1.0, 'x3': 0.0, 'x4': 0.430463, 'x5': 1.0, 'x6': 1.0} using GenerationNode MBM.
[INFO 01-22 17:49:19] ax.api.client: Generated new trial 58 with parameters {'x1': 0.0, 'x2': 1.0, 'x3': 1.0, 'x4': 0.377968, 'x5': 1.0, 'x6': 0.870405} using GenerationNode MBM.
[INFO 01-22 17:49:19] ax.api.client: Generated new trial 59 with parameters {'x1': 0.542099, 'x2': 0.0, 'x3': 1.0, 'x4': 0.37385, 'x5': 0.0, 'x6': 1.0} using GenerationNode MBM.
[INFO 01-22 17:49:20] ax.api.client: Trial 57 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.321840762123822]
- Thresholds: [2.94131395757758]
- Number of trials: [58]
- Trial indices: [[0, 46, 1, 37, 36, 21, 6, 45, 7, 50, 44, 49, 56, 55, 19, 20, 23, 17, 31, 35, 4, 39, 40, 8, 22, 18, 5, 16, 54, 38, 12, 11, 14, 32, 13, 10, 47, 41, 53, 43, 57, 27, 29, 51, 34, 2, 25, 48, 9, 24, 52, 30, 26, 33, 3, 42, 28, 15]]
[INFO 01-22 17:49:20] ax.api.client: Trial 57 marked EARLY_STOPPED.
[INFO 01-22 17:49:21] ax.api.client: Trial 58 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.3206828332904856]
- Thresholds: [2.972806907556582]
- Number of trials: [59]
- Trial indices: [[0, 56, 54, 53, 52, 50, 6, 9, 49, 10, 47, 48, 7, 4, 8, 51, 43, 16, 15, 14, 40, 39, 18, 41, 1, 44, 42, 12, 11, 46, 13, 45, 17, 21, 20, 32, 23, 24, 30, 29, 5, 58, 27, 28, 25, 57, 26, 55, 2, 3, 34, 19, 36, 35, 33, 31, 22, 37, 38]]
[INFO 01-22 17:49:21] ax.api.client: Trial 58 marked EARLY_STOPPED.
[INFO 01-22 17:49:21] ax.api.client: Trial 59 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.1188159563515536]
- Thresholds: [2.992401953398206]
- Number of trials: [60]
- Trial indices: [[0, 58, 55, 56, 7, 54, 8, 9, 52, 5, 10, 12, 14, 13, 53, 11, 15, 17, 16, 51, 1, 26, 22, 23, 20, 48, 4, 21, 50, 49, 18, 19, 25, 24, 46, 47, 44, 27, 28, 29, 45, 3, 41, 33, 31, 43, 6, 57, 37, 38, 39, 40, 59, 36, 42, 2, 34, 35, 30, 32]]
[INFO 01-22 17:49:21] ax.api.client: Trial 59 marked EARLY_STOPPED.
[INFO 01-22 17:49:24] ax.api.client: Generated new trial 60 with parameters {'x1': 0.0, 'x2': 1.0, 'x3': 0.0, 'x4': 0.393933, 'x5': 0.0, 'x6': 0.843897} using GenerationNode MBM.
[INFO 01-22 17:49:24] ax.api.client: Generated new trial 61 with parameters {'x1': 0.0, 'x2': 0.0, 'x3': 1.0, 'x4': 0.0, 'x5': 0.0, 'x6': 0.0} using GenerationNode MBM.
[INFO 01-22 17:49:24] ax.api.client: Generated new trial 62 with parameters {'x1': 0.484789, 'x2': 1.0, 'x3': 0.0, 'x4': 0.480251, 'x5': 0.0, 'x6': 1.0} using GenerationNode MBM.
[INFO 01-22 17:49:25] ax.api.client: Trial 60 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.2990666865931226]
- Thresholds: [3.0119969992398294]
- Number of trials: [61]
- Trial indices: [[0, 60, 7, 6, 15, 54, 13, 14, 12, 52, 51, 16, 17, 18, 53, 50, 1, 20, 19, 21, 49, 47, 23, 42, 44, 24, 25, 45, 3, 46, 22, 48, 43, 29, 27, 28, 9, 26, 11, 8, 4, 10, 39, 41, 31, 38, 30, 57, 2, 34, 58, 55, 56, 59, 5, 37, 36, 32, 35, 33, 40]]
[INFO 01-22 17:49:25] ax.api.client: Trial 60 marked EARLY_STOPPED.
[INFO 01-22 17:49:25] ax.api.client: Trial 61 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.318355664138188]
- Thresholds: [3.013214481790949]
- Number of trials: [62]
- Trial indices: [[0, 57, 7, 6, 60, 18, 15, 55, 19, 20, 17, 4, 54, 51, 53, 21, 16, 52, 1, 23, 22, 14, 49, 28, 27, 48, 13, 8, 50, 24, 25, 26, 47, 10, 30, 9, 29, 12, 3, 32, 45, 31, 43, 46, 35, 42, 58, 56, 41, 11, 5, 39, 59, 61, 36, 37, 2, 40, 38, 33, 44, 34]]
[INFO 01-22 17:49:25] ax.api.client: Trial 61 marked EARLY_STOPPED.
[INFO 01-22 17:49:26] ax.api.client: Trial 62 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.312841178050253]
- Thresholds: [3.014431964342068]
- Number of trials: [63]
- Trial indices: [[0, 57, 8, 58, 28, 55, 29, 4, 6, 10, 33, 32, 30, 54, 31, 56, 26, 13, 12, 24, 48, 14, 35, 50, 53, 11, 49, 51, 52, 9, 1, 16, 18, 36, 39, 20, 5, 19, 40, 42, 21, 60, 62, 27, 59, 7, 61, 38, 45, 17, 43, 37, 23, 41, 3, 44, 15, 46, 2, 47, 34, 25, 22]]
[INFO 01-22 17:49:26] ax.api.client: Trial 62 marked EARLY_STOPPED.
[INFO 01-22 17:49:28] ax.api.client: Generated new trial 63 with parameters {'x1': 0.0, 'x2': 0.54904, 'x3': 0.0, 'x4': 0.361587, 'x5': 0.0, 'x6': 0.810465} using GenerationNode MBM.
[INFO 01-22 17:49:28] ax.api.client: Generated new trial 64 with parameters {'x1': 0.0, 'x2': 0.75544, 'x3': 1.0, 'x4': 0.341187, 'x5': 1.0, 'x6': 0.759584} using GenerationNode MBM.
[INFO 01-22 17:49:28] ax.api.client: Generated new trial 65 with parameters {'x1': 0.0, 'x2': 1.0, 'x3': 0.0, 'x4': 0.378273, 'x5': 1.0, 'x6': 0.866953} using GenerationNode MBM.
[INFO 01-22 17:49:29] ax.api.client: Trial 63 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.1294867497617758]
- Thresholds: [3.04887227595321]
- Number of trials: [64]
- Trial indices: [[31, 4, 0, 60, 8, 58, 6, 7, 9, 22, 21, 34, 1, 56, 17, 32, 54, 52, 35, 53, 55, 14, 51, 16, 36, 29, 3, 38, 50, 23, 5, 24, 19, 18, 27, 40, 37, 28, 49, 10, 20, 47, 13, 48, 57, 15, 61, 59, 30, 26, 62, 63, 42, 43, 25, 33, 44, 12, 11, 2, 46, 45, 39, 41]]
[INFO 01-22 17:49:29] ax.api.client: Trial 63 marked EARLY_STOPPED.
[INFO 01-22 17:49:29] ax.api.client: Trial 64 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.31611777216623]
- Thresholds: [3.083312587564352]
- Number of trials: [65]
- Trial indices: [[0, 19, 62, 21, 4, 31, 59, 14, 6, 22, 33, 57, 60, 1, 32, 34, 7, 30, 54, 35, 58, 56, 55, 53, 15, 29, 37, 9, 23, 20, 13, 36, 39, 52, 28, 12, 48, 38, 51, 40, 50, 64, 8, 63, 27, 26, 44, 43, 46, 10, 45, 47, 5, 11, 18, 16, 42, 25, 17, 2, 49, 24, 3, 41, 61]]
[INFO 01-22 17:49:29] ax.api.client: Trial 64 marked EARLY_STOPPED.
[INFO 01-22 17:49:30] ax.api.client: Trial 65 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.321717502237847]
- Thresholds: [3.0834037733870696]
- Number of trials: [66]
- Trial indices: [[0, 62, 34, 65, 23, 35, 3, 57, 16, 55, 4, 32, 33, 11, 7, 22, 24, 56, 13, 37, 12, 44, 30, 31, 21, 54, 52, 38, 29, 43, 45, 25, 36, 41, 10, 42, 51, 53, 20, 17, 26, 19, 64, 14, 1, 60, 58, 61, 48, 5, 9, 28, 46, 6, 18, 27, 50, 47, 40, 39, 15, 8, 49, 2, 63, 59]]
[INFO 01-22 17:49:30] ax.api.client: Trial 65 marked EARLY_STOPPED.
[INFO 01-22 17:49:33] ax.api.client: Generated new trial 66 with parameters {'x1': 0.0, 'x2': 1.0, 'x3': 1.0, 'x4': 0.483232, 'x5': 0.0, 'x6': 1.0} using GenerationNode MBM.
[INFO 01-22 17:49:33] ax.api.client: Generated new trial 67 with parameters {'x1': 0.961044, 'x2': 1.0, 'x3': 1.0, 'x4': 0.554024, 'x5': 1.0, 'x6': 1.0} using GenerationNode MBM.
[INFO 01-22 17:49:33] ax.api.client: Generated new trial 68 with parameters {'x1': 1.0, 'x2': 0.0, 'x3': 0.0, 'x4': 1.0, 'x5': 0.0, 'x6': 0.0} using GenerationNode MBM.
[INFO 01-22 17:49:33] ax.api.client: Trial 66 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.2945153892854067]
- Thresholds: [3.0834949592097867]
- Number of trials: [67]
- Trial indices: [[0, 66, 65, 60, 43, 40, 19, 28, 41, 20, 42, 4, 37, 38, 12, 13, 44, 45, 46, 3, 16, 58, 1, 24, 14, 15, 10, 27, 36, 49, 47, 7, 51, 48, 50, 21, 35, 6, 8, 33, 5, 30, 52, 2, 53, 54, 11, 29, 9, 26, 55, 56, 25, 18, 22, 23, 57, 34, 32, 64, 61, 62, 59, 17, 31, 39, 63]]
[INFO 01-22 17:49:33] ax.api.client: Trial 66 marked EARLY_STOPPED.
[INFO 01-22 17:49:34] ax.api.client: Trial 67 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.3218909898873696]
- Thresholds: [3.10115545778067]
- Number of trials: [68]
- Trial indices: [[29, 33, 62, 23, 25, 0, 60, 63, 41, 61, 28, 1, 43, 44, 45, 47, 24, 3, 15, 9, 59, 31, 32, 34, 30, 18, 4, 42, 14, 11, 12, 46, 48, 56, 53, 27, 26, 13, 54, 2, 55, 37, 65, 36, 16, 7, 17, 21, 38, 39, 51, 52, 49, 5, 58, 57, 8, 50, 20, 19, 40, 22, 6, 10, 64, 67, 35, 66]]
[INFO 01-22 17:49:34] ax.api.client: Trial 67 marked EARLY_STOPPED.
[INFO 01-22 17:49:35] ax.api.client: Trial 68 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.321911002721988]
- Thresholds: [3.1188159563515536]
- Number of trials: [69]
- Trial indices: [[0, 15, 63, 36, 3, 27, 51, 28, 1, 61, 8, 17, 40, 64, 39, 49, 26, 9, 14, 37, 21, 47, 42, 19, 43, 41, 52, 54, 62, 18, 50, 53, 24, 31, 59, 32, 55, 12, 44, 13, 22, 57, 65, 16, 45, 4, 46, 7, 11, 25, 29, 35, 6, 23, 56, 58, 2, 33, 34, 48, 60, 20, 10, 30, 67, 38, 66, 68, 5]]
[INFO 01-22 17:49:35] ax.api.client: Trial 68 marked EARLY_STOPPED.
[INFO 01-22 17:49:37] ax.api.client: Generated new trial 69 with parameters {'x1': 0.0, 'x2': 1.0, 'x3': 1.0, 'x4': 0.367685, 'x5': 0.0, 'x6': 0.846475} using GenerationNode MBM.
[INFO 01-22 17:49:37] ax.api.client: Generated new trial 70 with parameters {'x1': 0.0, 'x2': 1.0, 'x3': 0.0, 'x4': 0.363718, 'x5': 0.0, 'x6': 1.0} using GenerationNode MBM.
[INFO 01-22 17:49:37] ax.api.client: Generated new trial 71 with parameters {'x1': 0.0, 'x2': 1.0, 'x3': 0.0, 'x4': 1.0, 'x5': 0.0, 'x6': 1.0} using GenerationNode MBM.
[INFO 01-22 17:49:37] ax.api.client: Trial 69 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.291408777490168]
- Thresholds: [3.1241513530566647]
- Number of trials: [70]
- Trial indices: [[0, 68, 10, 69, 28, 51, 49, 9, 65, 3, 63, 62, 53, 47, 52, 4, 54, 55, 18, 44, 57, 32, 43, 46, 24, 11, 22, 45, 31, 1, 17, 30, 41, 40, 59, 56, 60, 16, 26, 12, 38, 36, 66, 2, 37, 58, 39, 6, 25, 13, 15, 5, 35, 23, 7, 42, 33, 19, 27, 21, 61, 34, 20, 8, 64, 50, 48, 14, 67, 29]]
[INFO 01-22 17:49:37] ax.api.client: Trial 69 marked EARLY_STOPPED.
[INFO 01-22 17:49:38] ax.api.client: Trial 70 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.3092306010235233]
- Thresholds: [3.1294867497617758]
- Number of trials: [71]
- Trial indices: [[0, 20, 68, 6, 67, 43, 52, 55, 3, 24, 51, 10, 54, 11, 36, 7, 58, 56, 53, 41, 65, 66, 1, 4, 57, 37, 38, 31, 30, 49, 50, 25, 23, 15, 28, 14, 59, 18, 39, 63, 64, 13, 48, 34, 2, 46, 60, 61, 26, 35, 8, 62, 12, 40, 33, 19, 5, 27, 22, 47, 44, 45, 9, 32, 69, 42, 29, 17, 16, 70, 21]]
[INFO 01-22 17:49:38] ax.api.client: Trial 70 marked EARLY_STOPPED.
[INFO 01-22 17:49:39] ax.api.client: Trial 71 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.3218416554509522]
- Thresholds: [3.132743554074353]
- Number of trials: [72]
- Trial indices: [[0, 69, 48, 68, 47, 1, 25, 29, 30, 13, 49, 66, 45, 51, 11, 67, 5, 50, 23, 46, 6, 33, 14, 4, 44, 42, 18, 21, 28, 64, 52, 20, 26, 53, 56, 54, 10, 32, 34, 65, 63, 55, 16, 27, 24, 22, 58, 36, 35, 3, 57, 15, 9, 59, 2, 37, 39, 17, 40, 62, 61, 7, 41, 43, 71, 38, 60, 8, 12, 19, 70, 31]]
[INFO 01-22 17:49:39] ax.api.client: Trial 71 marked EARLY_STOPPED.
[INFO 01-22 17:49:41] ax.api.client: Generated new trial 72 with parameters {'x1': 1.0, 'x2': 0.159469, 'x3': 0.0, 'x4': 0.383819, 'x5': 1.0, 'x6': 1.0} using GenerationNode MBM.
[INFO 01-22 17:49:41] ax.api.client: Generated new trial 73 with parameters {'x1': 0.0, 'x2': 1.0, 'x3': 1.0, 'x4': 0.416785, 'x5': 1.0, 'x6': 1.0} using GenerationNode MBM.
[INFO 01-22 17:49:41] ax.api.client: Generated new trial 74 with parameters {'x1': 0.587549, 'x2': 1.0, 'x3': 0.0, 'x4': 0.392186, 'x5': 0.0, 'x6': 1.0} using GenerationNode MBM.
[INFO 01-22 17:49:42] ax.api.client: Trial 72 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.321885647663103]
- Thresholds: [3.136000358386931]
- Number of trials: [73]
- Trial indices: [[0, 48, 4, 23, 39, 70, 18, 38, 49, 29, 68, 67, 41, 40, 30, 22, 59, 10, 24, 31, 13, 1, 12, 60, 28, 65, 44, 47, 7, 56, 58, 37, 61, 3, 42, 54, 64, 66, 45, 26, 32, 43, 2, 36, 8, 62, 53, 35, 51, 52, 5, 34, 11, 55, 33, 19, 25, 21, 63, 17, 16, 27, 57, 9, 72, 71, 20, 69, 50, 6, 46, 14, 15]]
[INFO 01-22 17:49:42] ax.api.client: Trial 72 marked EARLY_STOPPED.
[INFO 01-22 17:49:42] ax.api.client: Trial 73 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.321358969212918]
- Thresholds: [3.13822643882579]
- Number of trials: [74]
- Trial indices: [[53, 27, 24, 60, 61, 72, 0, 58, 70, 54, 4, 50, 51, 1, 68, 38, 69, 23, 49, 39, 10, 37, 40, 21, 12, 11, 13, 22, 28, 7, 59, 36, 47, 19, 66, 29, 20, 46, 35, 34, 41, 42, 67, 8, 15, 31, 44, 33, 9, 16, 57, 30, 5, 43, 64, 17, 55, 56, 18, 65, 48, 2, 3, 62, 73, 26, 32, 14, 25, 71, 52, 6, 63, 45]]
[INFO 01-22 17:49:42] ax.api.client: Trial 73 marked EARLY_STOPPED.
[INFO 01-22 17:49:43] ax.api.client: Trial 74 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.3121428968712925]
- Thresholds: [3.140452519264649]
- Number of trials: [75]
- Trial indices: [[0, 71, 54, 7, 15, 55, 53, 1, 52, 37, 35, 56, 6, 57, 39, 38, 36, 12, 13, 32, 50, 58, 5, 70, 25, 48, 69, 18, 16, 33, 8, 51, 68, 2, 22, 49, 67, 29, 60, 59, 41, 11, 47, 30, 44, 63, 42, 46, 4, 27, 66, 28, 45, 21, 40, 23, 62, 43, 9, 61, 34, 26, 17, 20, 19, 64, 14, 31, 65, 3, 24, 10, 73, 74, 72]]
[INFO 01-22 17:49:43] ax.api.client: Trial 74 marked EARLY_STOPPED.
[INFO 01-22 17:49:45] ax.api.client: Generated new trial 75 with parameters {'x1': 0.463321, 'x2': 1.0, 'x3': 0.0, 'x4': 1.0, 'x5': 0.0, 'x6': 0.316452} using GenerationNode MBM.
[INFO 01-22 17:49:45] ax.api.client: Generated new trial 76 with parameters {'x1': 0.0, 'x2': 1.0, 'x3': 1.0, 'x4': 0.430478, 'x5': 1.0, 'x6': 0.849734} using GenerationNode MBM.
[INFO 01-22 17:49:45] ax.api.client: Generated new trial 77 with parameters {'x1': 1.0, 'x2': 1.0, 'x3': 0.0, 'x4': 1.0, 'x5': 0.0, 'x6': 0.375755} using GenerationNode MBM.
[INFO 01-22 17:49:46] ax.api.client: Trial 75 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.1783432683874526]
- Thresholds: [3.1522113740891307]
- Number of trials: [76]
- Trial indices: [[57, 42, 4, 66, 45, 58, 31, 47, 59, 27, 8, 16, 64, 46, 74, 62, 69, 3, 54, 37, 56, 40, 41, 65, 26, 25, 5, 63, 72, 1, 67, 68, 19, 50, 18, 20, 33, 9, 60, 22, 70, 71, 21, 35, 10, 52, 49, 11, 2, 12, 39, 38, 6, 61, 23, 34, 48, 32, 55, 36, 24, 13, 17, 51, 0, 30, 73, 44, 75, 29, 7, 28, 15, 14, 53, 43]]
[INFO 01-22 17:49:46] ax.api.client: Trial 75 marked EARLY_STOPPED.
[INFO 01-22 17:49:47] ax.api.client: Trial 76 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.32074267441226]
- Thresholds: [3.1639702289136125]
- Number of trials: [77]
- Trial indices: [[0, 39, 73, 40, 4, 31, 71, 30, 6, 43, 74, 72, 1, 69, 54, 56, 18, 44, 45, 70, 55, 10, 42, 68, 52, 35, 32, 57, 38, 17, 9, 33, 36, 46, 11, 66, 53, 22, 16, 29, 65, 23, 67, 47, 41, 48, 3, 51, 15, 24, 14, 25, 34, 63, 27, 8, 58, 12, 2, 7, 37, 28, 59, 50, 62, 26, 60, 64, 5, 49, 13, 61, 21, 75, 20, 76, 19]]
[INFO 01-22 17:49:47] ax.api.client: Trial 76 marked EARLY_STOPPED.
[INFO 01-22 17:49:48] ax.api.client: Trial 77 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.321685765105178]
- Thresholds: [3.1711567486505325]
- Number of trials: [78]
- Trial indices: [[0, 41, 74, 72, 4, 30, 11, 29, 12, 58, 28, 56, 20, 21, 42, 57, 13, 59, 71, 53, 43, 39, 36, 38, 37, 44, 70, 73, 60, 55, 31, 54, 62, 10, 63, 32, 67, 34, 69, 35, 33, 64, 49, 25, 45, 48, 68, 3, 26, 14, 18, 8, 23, 50, 65, 61, 19, 22, 15, 5, 27, 1, 52, 51, 9, 2, 17, 76, 7, 77, 75, 40, 66, 6, 24, 47, 46, 16]]
[INFO 01-22 17:49:48] ax.api.client: Trial 77 marked EARLY_STOPPED.
[INFO 01-22 17:49:50] ax.api.client: Generated new trial 78 with parameters {'x1': 0.0, 'x2': 0.508919, 'x3': 1.0, 'x4': 0.38326, 'x5': 0.0, 'x6': 1.0} using GenerationNode MBM.
[INFO 01-22 17:49:50] ax.api.client: Generated new trial 79 with parameters {'x1': 0.0, 'x2': 1.0, 'x3': 1.0, 'x4': 0.353556, 'x5': 1.0, 'x6': 1.0} using GenerationNode MBM.
[INFO 01-22 17:49:50] ax.api.client: Generated new trial 80 with parameters {'x1': 0.0, 'x2': 0.0, 'x3': 0.0, 'x4': 1.0, 'x5': 0.0, 'x6': 0.0} using GenerationNode MBM.
[INFO 01-22 17:49:51] ax.api.client: Trial 78 should be stopped early: Trial objective values at progressions in [11.00, 11.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [11.0]
- Underperforms: [True]
- Trial objective values: [2.503973348246813]
- Thresholds: [2.08059058352133]
- Number of trials: [33]
- Trial indices: [[0, 78, 5, 4, 11, 35, 31, 51, 9, 36, 12, 29, 27, 13, 28, 14, 3, 6, 1, 26, 15, 10, 8, 17, 23, 25, 2, 24, 18, 56, 21, 7, 22]]
[INFO 01-22 17:49:51] ax.api.client: Trial 78 marked EARLY_STOPPED.
[INFO 01-22 17:49:51] ax.api.client: Trial 79 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.3212639555803585]
- Thresholds: [3.1711567486505325]
- Number of trials: [80]
- Trial indices: [[23, 50, 51, 6, 21, 76, 13, 54, 25, 20, 55, 5, 74, 19, 53, 49, 56, 75, 14, 28, 1, 44, 59, 48, 57, 15, 46, 58, 12, 31, 30, 47, 7, 33, 73, 11, 8, 72, 42, 45, 71, 4, 40, 38, 16, 3, 36, 10, 39, 18, 17, 27, 64, 61, 34, 63, 60, 32, 62, 41, 2, 70, 69, 43, 24, 9, 35, 22, 37, 65, 66, 68, 67, 26, 0, 79, 52, 77, 29, 78]]
[INFO 01-22 17:49:51] ax.api.client: Trial 79 marked EARLY_STOPPED.
[INFO 01-22 17:49:52] ax.api.client: Trial 80 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.3217934929489674]
- Thresholds: [3.1783432683874526]
- Number of trials: [81]
- Trial indices: [[78, 38, 36, 75, 15, 76, 56, 26, 25, 11, 74, 33, 35, 16, 41, 1, 6, 40, 42, 24, 17, 72, 70, 14, 73, 18, 45, 44, 58, 43, 57, 54, 59, 60, 53, 55, 34, 46, 20, 32, 71, 69, 67, 64, 48, 52, 22, 50, 27, 10, 61, 3, 30, 19, 13, 21, 47, 66, 49, 2, 68, 8, 31, 5, 51, 9, 65, 62, 23, 28, 12, 63, 4, 77, 29, 37, 79, 0, 7, 39, 80]]
[INFO 01-22 17:49:52] ax.api.client: Trial 80 marked EARLY_STOPPED.
[INFO 01-22 17:49:55] ax.api.client: Generated new trial 81 with parameters {'x1': 1.0, 'x2': 1.0, 'x3': 0.0, 'x4': 1.0, 'x5': 1.0, 'x6': 1.0} using GenerationNode MBM.
[INFO 01-22 17:49:55] ax.api.client: Generated new trial 82 with parameters {'x1': 0.0, 'x2': 1.0, 'x3': 0.0, 'x4': 0.422573, 'x5': 0.0, 'x6': 0.904694} using GenerationNode MBM.
[INFO 01-22 17:49:55] ax.api.client: Generated new trial 83 with parameters {'x1': 0.0, 'x2': 0.612498, 'x3': 1.0, 'x4': 0.379281, 'x5': 1.0, 'x6': 0.897453} using GenerationNode MBM.
[INFO 01-22 17:49:56] ax.api.client: Trial 81 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.321928066762857]
- Thresholds: [3.1967621647124482]
- Number of trials: [82]
- Trial indices: [[0, 81, 78, 77, 1, 39, 36, 62, 61, 27, 22, 59, 4, 10, 18, 57, 26, 64, 3, 17, 24, 75, 7, 54, 65, 66, 63, 58, 55, 52, 67, 56, 35, 9, 69, 5, 38, 11, 21, 41, 42, 16, 23, 12, 30, 68, 31, 34, 53, 51, 49, 29, 19, 45, 43, 2, 48, 6, 14, 50, 71, 72, 25, 70, 8, 46, 73, 28, 74, 20, 47, 44, 37, 76, 40, 15, 33, 32, 13, 60, 79, 80]]
[INFO 01-22 17:49:56] ax.api.client: Trial 81 marked EARLY_STOPPED.
[INFO 01-22 17:49:56] ax.api.client: Trial 82 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.304526866189882]
- Thresholds: [3.215181061037444]
- Number of trials: [83]
- Trial indices: [[0, 81, 47, 49, 50, 45, 7, 48, 77, 44, 54, 42, 23, 79, 46, 6, 39, 40, 37, 20, 21, 58, 57, 56, 5, 78, 41, 55, 53, 8, 43, 52, 61, 18, 33, 25, 65, 12, 64, 11, 30, 66, 13, 29, 75, 72, 74, 31, 35, 19, 34, 24, 63, 4, 62, 32, 1, 9, 60, 10, 36, 76, 38, 59, 67, 16, 73, 27, 26, 80, 71, 2, 28, 69, 15, 70, 14, 17, 82, 51, 22, 3, 68]]
[INFO 01-22 17:49:56] ax.api.client: Trial 82 marked EARLY_STOPPED.
[INFO 01-22 17:49:57] ax.api.client: Trial 83 should be stopped early: Trial objective values at progressions in [10.00, 10.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [10.0]
- Underperforms: [True]
- Trial objective values: [3.3165765176774005]
- Thresholds: [3.228387539955535]
- Number of trials: [84]
- Trial indices: [[0, 53, 82, 11, 12, 37, 13, 35, 64, 8, 51, 50, 1, 79, 78, 80, 66, 4, 62, 23, 27, 7, 56, 21, 65, 68, 38, 67, 54, 55, 22, 48, 69, 63, 49, 5, 46, 20, 42, 9, 77, 75, 60, 6, 29, 58, 43, 15, 40, 18, 19, 41, 14, 34, 32, 28, 45, 47, 74, 73, 76, 2, 39, 33, 31, 61, 72, 57, 16, 71, 70, 3, 44, 17, 26, 52, 24, 81, 30, 59, 36, 10, 83, 25]]
[INFO 01-22 17:49:57] ax.api.client: Trial 83 marked EARLY_STOPPED.
[INFO 01-22 17:49:59] ax.api.client: Generated new trial 84 with parameters {'x1': 0.0, 'x2': 0.0, 'x3': 1.0, 'x4': 0.375738, 'x5': 0.0, 'x6': 1.0} using GenerationNode MBM.
[INFO 01-22 17:49:59] ax.api.client: Generated new trial 85 with parameters {'x1': 0.0, 'x2': 0.0, 'x3': 1.0, 'x4': 0.310527, 'x5': 0.0, 'x6': 0.769653} using GenerationNode MBM.
[INFO 01-22 17:49:59] ax.api.client: Generated new trial 86 with parameters {'x1': 0.0, 'x2': 0.0, 'x3': 0.0, 'x4': 0.388607, 'x5': 0.0, 'x6': 1.0} using GenerationNode MBM.
[INFO 01-22 17:50:00] ax.api.client: Trial 84 should be stopped early: Trial objective values at progressions in [11.00, 11.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [11.0]
- Underperforms: [True]
- Trial objective values: [2.9702128334359563]
- Thresholds: [2.244198759608217]
- Number of trials: [34]
- Trial indices: [[0, 1, 51, 4, 31, 22, 28, 23, 11, 9, 35, 6, 13, 26, 29, 25, 78, 12, 2, 7, 21, 8, 18, 17, 15, 27, 3, 10, 24, 56, 84, 5, 14, 36]]
[INFO 01-22 17:50:00] ax.api.client: Trial 84 marked EARLY_STOPPED.
[INFO 01-22 17:50:00] ax.api.client: Trial 85 should be stopped early: Trial objective values at progressions in [11.00, 11.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [11.0]
- Underperforms: [True]
- Trial objective values: [2.9032450878709053]
- Thresholds: [2.4078069356951044]
- Number of trials: [35]
- Trial indices: [[0, 85, 27, 24, 26, 10, 51, 11, 4, 29, 22, 1, 12, 5, 6, 7, 13, 9, 56, 3, 21, 14, 78, 25, 15, 8, 36, 17, 2, 18, 28, 35, 84, 31, 23]]
[INFO 01-22 17:50:00] ax.api.client: Trial 85 marked EARLY_STOPPED.
[INFO 01-22 17:50:01] ax.api.client: Trial 86 should be stopped early: Trial objective values at progressions in [11.00, 11.00] are all worse than 50.0-th percentile across comparable trials 
- Progressions: [11.0]
- Underperforms: [True]
- Trial objective values: [3.0388012177413244]
- Thresholds: [2.4458824662986762]
- Number of trials: [36]
- Trial indices: [[0, 8, 11, 84, 15, 26, 18, 22, 23, 36, 5, 4, 35, 1, 14, 7, 6, 10, 3, 12, 28, 56, 24, 31, 9, 51, 25, 17, 86, 85, 29, 2, 21, 13, 78, 27]]
[INFO 01-22 17:50:01] ax.api.client: Trial 86 marked EARLY_STOPPED.

Step 6: Analyze Results

After running trials, you can analyze the results. Most commonly this means extracting the parameterization from the best performing trial you conducted.

best_parameters, prediction, index, name = client.get_best_parameterization()
print("Best Parameters:", best_parameters)
print("Prediction (mean, variance):", prediction)

Output:

Best Parameters: {'x1': 0.0, 'x2': 0.0, 'x3': 0.4083361425260161, 'x4': 0.3019243214195876, 'x5': 0.3269149246171461, 'x6': 0.706133478238345}
Prediction (mean, variance): {'hartmann6': (np.float64(-2.408031512528801), np.float64(0.003911620186027599))}

Step 7: Compute Analyses

Ax can also produce a number of analyses to help interpret the results of the experiment via client.compute_analyses. Users can manually select which analyses to run, or can allow Ax to select which would be most relevant. In this case Ax selects the following:

• Arm Effects Plots show the metric value for each arm on the experiment. Ax produces one plot using values from its internal surrogate model (this can be helpful for seeing the true effect of an arm when evaluations are noisy) and another using the raw metric values as observed.
• Summary lists all trials generated along with their parameterizations, observations, and miscellaneous metadata
• Sensitivity Analysis Plot shows which parameters have the largest affect on the objective using Sobol Indicies
• Slice Plot shows how the model predicts a single parameter effects the objective along with a confidence interval
• Contour Plot shows how the model predicts a pair of parameters effects the objective as a 2D surface
• Cross Validation helps to visualize how well the surrogate model is able to predict out of sample points

# display=True instructs Ax to sort then render the resulting analyses
cards = client.compute_analyses(display=True)

Output:

[ERROR 01-22 17:50:06] ax.analysis.analysis: Failed to compute CrossValidationPlot
[ERROR 01-22 17:50:06] ax.analysis.analysis: Traceback (most recent call last):
File "/home/runner/work/Ax/Ax/ax/analysis/analysis.py", line 107, in compute_result
  card = self.compute(
         ^^^^^^^^^^^^^
File "/home/runner/work/Ax/Ax/ax/analysis/plotly/cross_validation.py", line 134, in compute
  cv_results = cross_validate(
               ^^^^^^^^^^^^^^^
File "/home/runner/work/Ax/Ax/ax/adapter/cross_validation.py", line 155, in cross_validate
  return _fold_cross_validate(
         ^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/Ax/Ax/ax/adapter/cross_validation.py", line 440, in _fold_cross_validate
  cv_test_observations = t.transform_observations(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/Ax/Ax/ax/adapter/transforms/base.py", line 147, in transform_observations
  obs_data = self._transform_observation_data(observation_data=obs_data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/Ax/Ax/ax/adapter/transforms/winsorize.py", line 147, in _transform_observation_data
  obsd.means[idx] = max(
  ~~~~~~~~~~^^^^^
ValueError: assignment destination is read-only
[ERROR 01-22 17:50:06] ax.analysis.analysis: Failed to compute PredictableMetricsAnalysis
[ERROR 01-22 17:50:06] ax.analysis.analysis: Traceback (most recent call last):
File "/home/runner/work/Ax/Ax/ax/analysis/analysis.py", line 107, in compute_result
  card = self.compute(
         ^^^^^^^^^^^^^
File "/home/runner/work/Ax/Ax/ax/analysis/healthcheck/predictable_metrics.py", line 151, in compute
  warning_message = warn_if_unpredictable_metrics(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/Ax/Ax/ax/service/utils/report_utils.py", line 1467, in warn_if_unpredictable_metrics
  model_fit_dict = compute_model_fit_metrics_from_adapter(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/Ax/Ax/ax/adapter/cross_validation.py", line 780, in compute_model_fit_metrics_from_adapter
  y_obs, y_pred, se_pred = predict_func(adapter=adapter, untransform=untransform)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/Ax/Ax/ax/adapter/cross_validation.py", line 903, in _predict_on_cross_validation_data
  cv = cross_validate(adapter=adapter, untransform=untransform)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/Ax/Ax/ax/adapter/cross_validation.py", line 155, in cross_validate
  return _fold_cross_validate(
         ^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/Ax/Ax/ax/adapter/cross_validation.py", line 440, in _fold_cross_validate
  cv_test_observations = t.transform_observations(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/Ax/Ax/ax/adapter/transforms/base.py", line 147, in transform_observations
  obs_data = self._transform_observation_data(observation_data=obs_data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/Ax/Ax/ax/adapter/transforms/winsorize.py", line 147, in _transform_observation_data
  obsd.means[idx] = max(
  ~~~~~~~~~~^^^^^
ValueError: assignment destination is read-only

Modeled Arm Effects on hartmann6

Modeled effects on hartmann6. This plot visualizes predictions of the true metric changes for each arm based on Ax's model. This is the expected delta you would expect if you (re-)ran that arm. This plot helps in anticipating the outcomes and performance of arms based on the model's predictions. Note, flat predictions across arms indicate that the model predicts that there is no effect, meaning if you were to re-run the experiment, the delta you would see would be small and fall within the confidence interval indicated in the plot.

loading...

Observed Arm Effects on hartmann6

Observed effects on hartmann6. This plot visualizes the effects from previously-run arms on a specific metric, providing insights into their performance. This plot allows one to compare and contrast the effectiveness of different arms, highlighting which configurations have yielded the most favorable outcomes.

loading...

Utility Progression

Shows the best hartmann6 value achieved so far across completed trials (objective is to minimize). The x-axis shows trace index, which counts completed or early-stopped trials sequentially (1, 2, 3, ...). This differs from trial index, which may have gaps if some trials failed or were abandoned. For example, if trials 0, 2, and 5 completed while trials 1, 3, and 4 failed, the trace indices would be 1, 2, 3 corresponding to trial indices 0, 2, 5. The y-axis shows cumulative best utility. Only improvements are plotted, so flat segments indicate trials that didn't surpass the previous best. Infeasible trials (violating outcome constraints) don't contribute to the improvements.

loading...

hartmann6 by progression

The progression plot tracks the evolution of each metric over the course of the experiment. This visualization is typically used to monitor the improvement of metrics over Trial iterations, but can also be useful in informing decisions about early stopping for Trials.

loading...

hartmann6 by wallclock time

The progression plot tracks the evolution of each metric over the course of the experiment. This visualization is typically used to monitor the improvement of metrics over Trial iterations, but can also be useful in informing decisions about early stopping for Trials.

loading...

Best Trial for Experiment

Displays the trial with the best objective value based on raw observations. This reflects actual measured performance during execution. This trial achieved the optimal objective value and represents the recommended configuration for your optimization goal. Only considering COMPLETED trials.

	trial_index	arm_name	trial_status	generation_node	hartmann6	x1	x2	x3	x4	x5	x6
0	35	35_0	COMPLETED	MBM	-2.59551	0.196118	0.397471	0.423135	0.293296	0.283777	0.699847

Summary for Experiment

High-level summary of the Trial-s in this Experiment

	trial_index	arm_name	trial_status	generation_node	hartmann6	x1	x2	x3	x4	x5	x6
0	0	0_0	COMPLETED	CenterOfSearchSpace	-0.505315	0.500000	0.500000	0.500000	0.500000	0.500000	0.500000
1	1	1_0	COMPLETED	Sobol	-0.019006	0.718403	0.132347	0.207968	0.886665	0.284014	0.974865
2	2	2_0	COMPLETED	Sobol	-0.238616	0.061872	0.889394	0.729777	0.336118	0.853096	0.025046
3	3	3_0	COMPLETED	Sobol	-0.185928	0.334068	0.355383	0.339440	0.545887	0.683735	0.442011
4	4	4_0	COMPLETED	Sobol	-0.238433	0.928634	0.596501	0.847813	0.247200	0.241579	0.558082
...	...	...	...	...	...	...	...	...	...	...	...
82	82	82_0	EARLY_STOPPED	MBM	3.304527	0.000000	1.000000	0.000000	0.422573	0.000000	0.904694
83	83	83_0	EARLY_STOPPED	MBM	3.316577	0.000000	0.612498	1.000000	0.379281	1.000000	0.897453
84	84	84_0	EARLY_STOPPED	MBM	2.970213	0.000000	0.000000	1.000000	0.375738	0.000000	1.000000
85	85	85_0	EARLY_STOPPED	MBM	2.903245	0.000000	0.000000	1.000000	0.310527	0.000000	0.769653
86	86	86_0	EARLY_STOPPED	MBM	3.038801	0.000000	0.000000	0.000000	0.388607	0.000000	1.000000

Sensitivity Analysis for hartmann6

Understand how each parameter affects hartmann6 according to a second-order sensitivity analysis.

loading...

hartmann6 vs. x6

The slice plot provides a one-dimensional view of predicted outcomes for hartmann6 as a function of a single parameter, while keeping all other parameters fixed at their status_quo value (or mean value if status_quo is unavailable). This visualization helps in understanding the sensitivity and impact of changes in the selected parameter on the predicted metric outcomes.

loading...

hartmann6 (Mean) vs. x4, x6

The contour plot visualizes the predicted outcomes for hartmann6 across a two-dimensional parameter space, with other parameters held fixed at their status_quo value (or mean value if status_quo is unavailable). This plot helps in identifying regions of optimal performance and understanding how changes in the selected parameters influence the predicted outcomes. Contour lines represent levels of constant predicted values, providing insights into the gradient and potential optima within the parameter space.

loading...

hartmann6 (Mean) vs. x4, x5

The contour plot visualizes the predicted outcomes for hartmann6 across a two-dimensional parameter space, with other parameters held fixed at their status_quo value (or mean value if status_quo is unavailable). This plot helps in identifying regions of optimal performance and understanding how changes in the selected parameters influence the predicted outcomes. Contour lines represent levels of constant predicted values, providing insights into the gradient and potential optima within the parameter space.

loading...

CrossValidationPlot Error

ValueError encountered while computing CrossValidationPlot.

Generation Strategy Graph

GenerationStrategy: Center+Sobol+MBM:fast

Visualize the structure of a GenerationStrategy as a directed graph. Each node represents a GenerationNode in the strategy, and edges represent transitions between nodes based on TransitionCriterion. Edge labels show the criterion class names that trigger the transition.

	node_name	generators	transitions	is_current
0	CenterOfSearchSpace	nan	-> Sobol: AutoTransitionAfterGen	False
1	Sobol	Sobol	-> MBM: MinTrials(5), MinTrials(2)	False
2	MBM	BoTorch	nan	True

PredictableMetricsAnalysis Error

ValueError encountered while computing PredictableMetricsAnalysis.

Conclusion

This tutorial demonstates Ax's early stopping capabilities, which utilize timeseries-like data to monitor the results of expensive evaluations and terminate those that are unlikely to produce promising results, freeing up resources to explore more configurations. This can be used in a number of applications, and is especially useful in machine learning contexts.