Version: Next

Automating Orchestration

Previously, we've demonstrated using Ax for ask-tell optimization, a paradigm in which we "ask" Ax for candidate configurations and "tell" Ax our observations. This can be effective in many scenerios, and it can be automated through use of flow control statements like for and while loops. However there are some situations where it would be beneficial to allow Ax to orchestrate the entire optimization: deploying trials to external systems, polling their status, and reading reading their results. This can be common in a number of real world engineering tasks, including:

Large scale machine learning experiments running workloads on high-performance computing clusters
A/B tests conducted using an external experimentation platform
Materials science optimizations utilizing a self-driving laboratory

Ax's Client can orchestrate automated adaptive experiments like this using its method run_trials. Users create custom classes which implement Ax's IMetric and IRunner protocols to handle data fetching and trial deployment respectively. Then, users simply configure their Client as they would normally and call run_trials; Ax will deploy trials, fetch data, generate candidates, and repeat as necessary. Ax can manage complex orchestration tasks including launching multiple trials in parallel while still respecting a user-defined concurrency limit, and gracefully handling trial failure by allowing the experiment to continue even if some trials do not complete successfully or data fetching fails.

In this tutorial we will optimize the Hartmann6 function as before, but we will configure custom Runners and Metrics to mimic an external execution system. The Runner will calculate Hartmann6 with the appropriate parameters, write the result to a file, and tell Ax the trial is ready after 5 seconds. The Metric will find the appropriate file and report the results back to Ax.

Learning Objectives

Learn when it can be appropriate and/or advantageous to run Ax in a closed-loop
Configure custom Runners and Metrics, allowing Ax to deploy trials and fetch data automatically
Understand tradeoffs between parallelism and optimization performance

Prerequisites

Understanding of adaptive experimentation and Bayesian optimization
Familiarity with configuring and conducting experiments in Ax

Step 1: Import Necessary Modules

First, ensure you have all the necessary imports:

import os
import time
from typing import Any, Mapping

import numpy as np
from ax.api.client import Client
from ax.api.configs import RangeParameterConfig
from ax.api.protocols.metric import IMetric
from ax.api.protocols.runner import IRunner, TrialStatus
from ax.api.types import TParameterization

Step 2: Defining our custom Runner and Metric

As stated before, we will be creating custom Runner and Metric classes to mimic an external system. Let's start by defining our Hartmann6 function as before.

# Hartmann6 function
def hartmann6(x1, x2, x3, x4, x5, x6):
    alpha = np.array([1.0, 1.2, 3.0, 3.2])
    A = np.array([
        [10, 3, 17, 3.5, 1.7, 8],
        [0.05, 10, 17, 0.1, 8, 14],
        [3, 3.5, 1.7, 10, 17, 8],
        [17, 8, 0.05, 10, 0.1, 14]
    ])
    P = 10**-4 * np.array([
        [1312, 1696, 5569, 124, 8283, 5886],
        [2329, 4135, 8307, 3736, 1004, 9991],
        [2348, 1451, 3522, 2883, 3047, 6650],
        [4047, 8828, 8732, 5743, 1091, 381]
    ])

    outer = 0.0
    for i in range(4):
        inner = 0.0
        for j, x in enumerate([x1, x2, x3, x4, x5, x6]):
            inner += A[i, j] * (x - P[i, j])**2
        outer += alpha[i] * np.exp(-inner)
    return -outer

hartmann6(0.1, 0.45, 0.8, 0.25, 0.552, 1.0)

Output:
np.float64(-0.4878737485613134)

Next, we will define the MockRunner. The MockRunner requires two methods: run_trial and poll_trial.

run_trial deploys a trial to the external system with the given parameters. In this case, we will simply save a file containing the result of a call to the Hartmann6 function.

poll_trial queries the external system to see if the trial has completed, failed, or if it's still running. In this mock example, we will check to see how many seconds have elapsed since the run_trial was called and only report a trial as completed once 5 seconds have elapsed.

Runner's may also optionally implement a stop_trial method to terminate a trial's execution before it has completed. This is necessary for using early stopping in closed-loop experimentation, but we will skip this for now.

class MockRunner(IRunner):
    def run_trial(
        self, trial_index: int, parameterization: TParameterization
    ) -> dict[str, Any]:
        file_name = f"{int(time.time())}.txt"

        x1 = parameterization["x1"]
        x2 = parameterization["x2"]
        x3 = parameterization["x3"]
        x4 = parameterization["x4"]
        x5 = parameterization["x5"]
        x6 = parameterization["x6"]

        result = hartmann6(x1, x2, x3, x4, x5, x6)

        with open(file_name, "w") as f:
            f.write(f"{result}")

        return {"file_name": file_name}

    def poll_trial(
        self, trial_index: int, trial_metadata: Mapping[str, Any]
    ) -> TrialStatus:
        file_name = trial_metadata["file_name"]
        time_elapsed = time.time() - int(file_name[:4])

        if time_elapsed < 5:
            return TrialStatus.RUNNING

        return TrialStatus.COMPLETED

It's worthwhile to instantiate your Runner and test it is behaving as expected. Let's deploy a mock trial by manually calling run_trial and ensuring it creates a file.

runner = MockRunner()

trial_metadata = runner.run_trial(
    trial_index=-1,
    parameterization={
        "x1": 0.1,
        "x2": 0.45,
        "x3": 0.8,
        "x4": 0.25,
        "x5": 0.552,
        "x6": 1.0,
    },
)

os.path.exists(trial_metadata["file_name"])

Output:
True

Now, we will implement the Metric. Metrics only need to implement a fetch method, which returns a progression value (i.e. a step in a timeseries) and an observation value. Note that the observation can either be a simple float or a (mean, SEM) pair if the external system can report observed noise.

In this case, we have neither a relevant progression value nor observed noise so we will simply read the file and report (0, value).

class MockMetric(IMetric):
    def fetch(
        self,
        trial_index: int,
        trial_metadata: Mapping[str, Any],
    ) -> tuple[int, float | tuple[float, float]]:
        file_name = trial_metadata["file_name"]

        with open(file_name, 'r') as file:
            value = float(file.readline())
            return (0, value)

Again, let's validate the Metric created above by instantiating it and reporting the value from the file generated during testing of the Runner.

# Note: all Metrics must have a name. This will become relevant when attaching metrics to the Client
hartmann6_metric = MockMetric(name="hartmann6")

hartmann6_metric.fetch(trial_index=-1, trial_metadata=trial_metadata)

Output:
(0, -0.4878737485613134)

Step 3: Initialize the Client and Configure the Experiment

Finally, we can initialize the Client and configure the experiment as before. This will be familiar to readers of the Getting Started with Ax tutorial -- the only difference is we will attach the previously defined Runner and Metric by calling configure_runner and configure_metrics respectively.

Note that when initializing hartmann6_metric we set name=hartmann6, matching the objective we now set in configure_optimization. The configure_metrics method uses this name to ensure that data fetched by this Metric is used correctly during the experiment. Be careful to correctly set the name of the Metric to reflect its use as an objective or outcome constraint.

client = Client()
# Define six float parameters for the Hartmann6 function
parameters = [
    RangeParameterConfig(name=f"x{i + 1}", parameter_type="float", bounds=(0, 1))
    for i in range(6)
]

client.configure_experiment(
    parameters=parameters,
    # The following arguments are only necessary when saving to the DB
    name="hartmann6_experiment",
    description="Optimization of the Hartmann6 function",
    owner="developer",
)
client.configure_optimization(objective="-hartmann6")

client.configure_runner(runner=runner)
client.configure_metrics(metrics=[hartmann6_metric])

Step 5: Run trials

Once the Client has been configured, we can begin running trials.

Internally, Ax uses a class named Scheduler to orchestrate the trial deployment, polling, data fetching, and candidate generation.

Scheduler state machine

The run_trials method provides users with control over various orchestration settings as well as the total maximum number of trials to evaluate:

parallelism defines the maximum number of trials that may be run at once. If your external system supports multiple evaluations in parallel, increasing this number can significantly decrease experimentation time. However, it is important to note that as parallelism increases, optimiztion performance often decreases. This is because adaptive experimentation methods rely on previously observed data for candidate generation -- the more tirals that have been observed prior to generation of a new candidate, the more accurate Ax's model will be for generation of that candidate.
tolerated_trial_failure_rate sets the proportion of trials are allowed to fail before Ax raises an Exception. Depending on how expensive a single trial is to evaluate or how unreliable trials are expected to be, the experimenter may want to be notified as soon as a single trial fails or they may not care until more than half the trials are failing. Set this value as is appropriate for your context.
initial_seconds_between_polls sets the frequency at which the status of a trial is checked and the results are attempted to be fetched. Set this to be low for trials that are expected to complete quickly or high for trials the are expected to take a long time.

client.run_trials(
    max_trials=30,
    parallelism=3,
    tolerated_trial_failure_rate=0.1,
    initial_seconds_between_polls=1,
)

Output:
[INFO 01-22 17:46:00] ax.api.client: GenerationStrategy(name='Center+Sobol+MBM:fast', nodes=[CenterGenerationNode(next_node_name='Sobol'), GenerationNode(name='Sobol', generator_specs=[GeneratorSpec(generator_enum=Sobol, generator_key_override=None)], transition_criteria=[MinTrials(transition_to='MBM'), MinTrials(transition_to='MBM')]), GenerationNode(name='MBM', generator_specs=[GeneratorSpec(generator_enum=BoTorch, generator_key_override=None)], transition_criteria=[])]) chosen based on user input and problem structure.
[INFO 01-22 17:46:00] Orchestrator: Orchestrator requires experiment to have immutable search space and optimization config. Setting property immutable_search_space_and_opt_config to True on experiment.
[INFO 01-22 17:46:00] Orchestrator: Running trials [0]...
[INFO 01-22 17:46:01] Orchestrator: Running trials [1]...
[INFO 01-22 17:46:02] Orchestrator: Running trials [2]...
[INFO 01-22 17:46:03] Orchestrator: Retrieved COMPLETED trials: 0 - 2.
[INFO 01-22 17:46:03] Orchestrator: Running trials [3]...
[INFO 01-22 17:46:04] Orchestrator: Running trials [4]...
[INFO 01-22 17:46:06] Orchestrator: Running trials [5]...
[INFO 01-22 17:46:07] Orchestrator: Retrieved COMPLETED trials: 3 - 5.
[INFO 01-22 17:46:07] Orchestrator: Running trials [6]...
[INFO 01-22 17:46:09] Orchestrator: Running trials [7]...
[INFO 01-22 17:46:10] Orchestrator: Running trials [8]...
[INFO 01-22 17:46:11] Orchestrator: Retrieved COMPLETED trials: 6 - 8.
[INFO 01-22 17:46:11] Orchestrator: Running trials [9]...
[INFO 01-22 17:46:13] Orchestrator: Running trials [10]...
[INFO 01-22 17:46:14] Orchestrator: Running trials [11]...
[INFO 01-22 17:46:15] Orchestrator: Retrieved COMPLETED trials: 9 - 11.
[INFO 01-22 17:46:16] Orchestrator: Running trials [12]...
[INFO 01-22 17:46:17] Orchestrator: Running trials [13]...
[INFO 01-22 17:46:19] Orchestrator: Running trials [14]...
[INFO 01-22 17:46:20] Orchestrator: Retrieved COMPLETED trials: 12 - 14.
[INFO 01-22 17:46:20] Orchestrator: Running trials [15]...
[INFO 01-22 17:46:22] Orchestrator: Running trials [16]...
[INFO 01-22 17:46:23] Orchestrator: Running trials [17]...
[INFO 01-22 17:46:24] Orchestrator: Retrieved COMPLETED trials: 15 - 17.
[INFO 01-22 17:46:25] Orchestrator: Running trials [18]...
[INFO 01-22 17:46:26] Orchestrator: Running trials [19]...
[INFO 01-22 17:46:27] Orchestrator: Running trials [20]...
[INFO 01-22 17:46:28] Orchestrator: Retrieved COMPLETED trials: 18 - 20.
[INFO 01-22 17:46:29] Orchestrator: Running trials [21]...
[INFO 01-22 17:46:30] Orchestrator: Running trials [22]...
[INFO 01-22 17:46:32] Orchestrator: Running trials [23]...
[INFO 01-22 17:46:33] Orchestrator: Retrieved COMPLETED trials: 21 - 23.
[INFO 01-22 17:46:33] Orchestrator: Running trials [24]...
[INFO 01-22 17:46:35] Orchestrator: Running trials [25]...
[INFO 01-22 17:46:36] Orchestrator: Running trials [26]...
[INFO 01-22 17:46:37] Orchestrator: Retrieved COMPLETED trials: 24 - 26.
[INFO 01-22 17:46:38] Orchestrator: Running trials [27]...
[INFO 01-22 17:46:39] Orchestrator: Running trials [28]...
[INFO 01-22 17:46:40] Orchestrator: Running trials [29]...
[INFO 01-22 17:46:41] Orchestrator: Retrieved COMPLETED trials: 27 - 29.

Step 6: Analyze Results

As before, Ax can compute the best parameterization observed and produce a number of analyses to help interpret the results of the experiment.

It is also worth noting that the experiment can be resumed at any time using Ax's storage functionality. When configured to use a SQL databse, the Client saves a snapshot of itself at various points throughout the call to run_trials, making it incredibly easy to continue optimization after an unexpected failure. You can learn more about storage in Ax here.

best_parameters, prediction, index, name = client.get_best_parameterization()
print("Best Parameters:", best_parameters)
print("Prediction (mean, variance):", prediction)

Output:
Best Parameters: {'x1': 0.40642611409459417, 'x2': 0.9014106023130555, 'x3': 1.0, 'x4': 0.5823863995843687, 'x5': 0.0, 'x6': 0.0}
Prediction (mean, variance): {'hartmann6': (np.float64(-3.059961884952793), np.float64(0.0033639705085552938))}

# display=True instructs Ax to sort then render the resulting analyses
cards = client.compute_analyses(display=True)

Output:
[ERROR 01-22 17:46:43] ax.analysis.analysis: Failed to compute CrossValidationPlot
[ERROR 01-22 17:46:43] ax.analysis.analysis: Traceback (most recent call last):
File "/home/runner/work/Ax/Ax/ax/analysis/analysis.py", line 107, in compute_result
  card = self.compute(
         ^^^^^^^^^^^^^
File "/home/runner/work/Ax/Ax/ax/analysis/plotly/cross_validation.py", line 134, in compute
  cv_results = cross_validate(
               ^^^^^^^^^^^^^^^
File "/home/runner/work/Ax/Ax/ax/adapter/cross_validation.py", line 155, in cross_validate
  return _fold_cross_validate(
         ^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/Ax/Ax/ax/adapter/cross_validation.py", line 440, in _fold_cross_validate
  cv_test_observations = t.transform_observations(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/Ax/Ax/ax/adapter/transforms/base.py", line 147, in transform_observations
  obs_data = self._transform_observation_data(observation_data=obs_data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/Ax/Ax/ax/adapter/transforms/winsorize.py", line 147, in _transform_observation_data
  obsd.means[idx] = max(
  ~~~~~~~~~~^^^^^
ValueError: assignment destination is read-only
[ERROR 01-22 17:46:43] ax.analysis.analysis: Failed to compute PredictableMetricsAnalysis
[ERROR 01-22 17:46:43] ax.analysis.analysis: Traceback (most recent call last):
File "/home/runner/work/Ax/Ax/ax/analysis/analysis.py", line 107, in compute_result
  card = self.compute(
         ^^^^^^^^^^^^^
File "/home/runner/work/Ax/Ax/ax/analysis/healthcheck/predictable_metrics.py", line 151, in compute
  warning_message = warn_if_unpredictable_metrics(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/Ax/Ax/ax/service/utils/report_utils.py", line 1467, in warn_if_unpredictable_metrics
  model_fit_dict = compute_model_fit_metrics_from_adapter(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/Ax/Ax/ax/adapter/cross_validation.py", line 780, in compute_model_fit_metrics_from_adapter
  y_obs, y_pred, se_pred = predict_func(adapter=adapter, untransform=untransform)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/Ax/Ax/ax/adapter/cross_validation.py", line 903, in _predict_on_cross_validation_data
  cv = cross_validate(adapter=adapter, untransform=untransform)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/Ax/Ax/ax/adapter/cross_validation.py", line 155, in cross_validate
  return _fold_cross_validate(
         ^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/Ax/Ax/ax/adapter/cross_validation.py", line 440, in _fold_cross_validate
  cv_test_observations = t.transform_observations(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/Ax/Ax/ax/adapter/transforms/base.py", line 147, in transform_observations
  obs_data = self._transform_observation_data(observation_data=obs_data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/Ax/Ax/ax/adapter/transforms/winsorize.py", line 147, in _transform_observation_data
  obsd.means[idx] = max(
  ~~~~~~~~~~^^^^^
ValueError: assignment destination is read-only

Modeled Arm Effects on hartmann6

Modeled effects on hartmann6. This plot visualizes predictions of the true metric changes for each arm based on Ax's model. This is the expected delta you would expect if you (re-)ran that arm. This plot helps in anticipating the outcomes and performance of arms based on the model's predictions. Note, flat predictions across arms indicate that the model predicts that there is no effect, meaning if you were to re-run the experiment, the delta you would see would be small and fall within the confidence interval indicated in the plot.

Observed Arm Effects on hartmann6

Observed effects on hartmann6. This plot visualizes the effects from previously-run arms on a specific metric, providing insights into their performance. This plot allows one to compare and contrast the effectiveness of different arms, highlighting which configurations have yielded the most favorable outcomes.

Utility Progression

Shows the best hartmann6 value achieved so far across completed trials (objective is to minimize). The x-axis shows trace index, which counts completed or early-stopped trials sequentially (1, 2, 3, ...). This differs from trial index, which may have gaps if some trials failed or were abandoned. For example, if trials 0, 2, and 5 completed while trials 1, 3, and 4 failed, the trace indices would be 1, 2, 3 corresponding to trial indices 0, 2, 5. The y-axis shows cumulative best utility. Only improvements are plotted, so flat segments indicate trials that didn't surpass the previous best. Infeasible trials (violating outcome constraints) don't contribute to the improvements.

hartmann6 by progression

The progression plot tracks the evolution of each metric over the course of the experiment. This visualization is typically used to monitor the improvement of metrics over Trial iterations, but can also be useful in informing decisions about early stopping for Trials.

hartmann6 by wallclock time

Best Trial for Experiment

Displays the trial with the best objective value based on raw observations. This reflects actual measured performance during execution. This trial achieved the optimal objective value and represents the recommended configuration for your optimization goal. Only considering COMPLETED trials.

	trial_index	arm_name	trial_status	generation_node	hartmann6	x1	x2	x3	x4	x5	x6
0	23	23_0	COMPLETED	MBM	-3.119	0.406426	0.901411	1	0.582386	0	0

Summary for hartmann6_experiment

High-level summary of the Trial-s in this Experiment

	trial_index	arm_name	trial_status	generation_node	hartmann6	x1	x2	x3	x4	x5	x6
0	0	0_0	COMPLETED	CenterOfSearchSpace	-0.505315	0.5	0.5	0.5	0.5	0.5	0.5
1	1	1_0	COMPLETED	Sobol	-0.007216	0.916166	0.134356	0.55681	0.868467	0.41926	0.310295
2	2	2_0	COMPLETED	Sobol	-0.057388	0.000178	0.578484	0.004127	0.022738	0.594148	0.983505
3	3	3_0	COMPLETED	Sobol	-0.034661	0.487133	0.454583	0.912753	0.507046	0.790582	0.571452
4	4	4_0	COMPLETED	Sobol	-0.871425	0.586774	0.758502	0.39916	0.352725	0.223344	0.151898
5	5	5_0	COMPLETED	MBM	-0.426613	0.412944	0.470323	0.269985	0.649872	0.416071	0.656547
6	6	6_0	COMPLETED	MBM	-0.14333	0.604923	0.665316	0.270066	0.087778	0.245606	0.14411
7	7	7_0	COMPLETED	MBM	-2.25051	0.466393	0.817335	0.516333	0.531544	0.112707	0.168812
8	8	8_0	COMPLETED	MBM	-0.445463	0.676632	0.877414	0.285457	0.444281	0.413764	0.247696
9	9	9_0	COMPLETED	MBM	-2.65178	0.43668	0.871051	0.527853	0.63328	0.010081	0.134213
10	10	10_0	COMPLETED	MBM	-2.04704	0.417125	0.903861	0.545576	0.467452	0.173347	0.193675
11	11	11_0	COMPLETED	MBM	-2.67529	0.435015	0.742365	0.729559	0.594269	0	0.044744
12	12	12_0	COMPLETED	MBM	-2.30355	0.3571	0.949435	0.572057	0.72613	0	0.001593
13	13	13_0	COMPLETED	MBM	-0.11966	0.340068	0.26637	0.027216	0.697516	0	0.009969
14	14	14_0	COMPLETED	MBM	-2.10914	0.41097	1	1	0.739426	0	0.085723
15	15	15_0	COMPLETED	MBM	-0.08899	0.318472	1	0.726188	0.57517	0	0.546891
16	16	16_0	COMPLETED	MBM	-0.701025	0.690277	1	0.746546	0.577812	0	0
17	17	17_0	COMPLETED	MBM	-0.173732	0	1	0.708595	0.560971	0	0
18	18	18_0	COMPLETED	MBM	-1.70759	0.422496	0.795133	0.32194	0.807399	0	0.026956
19	19	19_0	COMPLETED	MBM	-1.50923	0.40131	0.768067	1	0.821716	0	0.085993
20	20	20_0	COMPLETED	MBM	-2.74662	0.421733	0.822256	0	0.651207	0	0
21	21	21_0	COMPLETED	MBM	-3.0602	0.394255	0.884738	0.218508	0.578854	0	0
22	22	22_0	COMPLETED	MBM	-3.0543	0.393369	0.878124	0.359105	0.595539	0.389181	0
23	23	23_0	COMPLETED	MBM	-3.119	0.406426	0.901411	1	0.582386	0	0
24	24	24_0	COMPLETED	MBM	-2.82661	0.391115	0.970407	1	0.557319	0.705635	0
25	25	25_0	COMPLETED	MBM	-2.99527	0.360013	0.897694	1	0.545513	0	0
26	26	26_0	COMPLETED	MBM	-2.74501	0.403777	1	0.279914	0.552794	0	0
27	27	27_0	COMPLETED	MBM	-3.07187	0.380433	0.850049	1	0.573984	0.255275	0
28	28	28_0	COMPLETED	MBM	-2.43718	0.340762	0.803716	0	0.535607	1	0
29	29	29_0	COMPLETED	MBM	-0.353995	0.197639	0.552815	0	0.336651	0	0

Sensitivity Analysis for hartmann6

Understand how each parameter affects hartmann6 according to a second-order sensitivity analysis.

hartmann6 vs. x6

The slice plot provides a one-dimensional view of predicted outcomes for hartmann6 as a function of a single parameter, while keeping all other parameters fixed at their status_quo value (or mean value if status_quo is unavailable). This visualization helps in understanding the sensitivity and impact of changes in the selected parameter on the predicted metric outcomes.

hartmann6 vs. x4

hartmann6 (Mean) vs. x2, x6

The contour plot visualizes the predicted outcomes for hartmann6 across a two-dimensional parameter space, with other parameters held fixed at their status_quo value (or mean value if status_quo is unavailable). This plot helps in identifying regions of optimal performance and understanding how changes in the selected parameters influence the predicted outcomes. Contour lines represent levels of constant predicted values, providing insights into the gradient and potential optima within the parameter space.

CrossValidationPlot Error

ValueError encountered while computing CrossValidationPlot.

Generation Strategy Graph

GenerationStrategy: Center+Sobol+MBM:fast

Visualize the structure of a GenerationStrategy as a directed graph. Each node represents a GenerationNode in the strategy, and edges represent transitions between nodes based on TransitionCriterion. Edge labels show the criterion class names that trigger the transition.

	node_name	generators	transitions	is_current
0	CenterOfSearchSpace	nan	-> Sobol: AutoTransitionAfterGen	False
1	Sobol	Sobol	-> MBM: MinTrials(5), MinTrials(2)	False
2	MBM	BoTorch	nan	True

PredictableMetricsAnalysis Error

ValueError encountered while computing PredictableMetricsAnalysis.

Conclusion

This tutorial demonstrates how to use Ax's Client for closed-loop optimization using the Hartmann6 function as an example. This style of optimization is useful in scenarios where trials are evaluated on some external system or when experimenters wish to take advantage of parallel evaluation, trial failure handling, or simply to manage long-running optimization tasks without human intervention. You can define your own Runner and Metric classes to communicate with whatever external systems you wish to interface with, and control optimization using the OrchestrationConfig.

Learning Objectives​

Prerequisites​

Step 1: Import Necessary Modules​

Step 2: Defining our custom Runner and Metric

Step 3: Initialize the Client and Configure the Experiment​

Step 5: Run trials​

Step 6: Analyze Results​

Conclusion​