The Ax Service API is designed to allow the user to control scheduling of trials and data computation while having an easy to use interface with Ax.
The user iteratively:
from ax.service.ax_client import AxClient
from ax.utils.measurement.synthetic_functions import hartmann6
from ax.utils.notebook.plotting import render, init_notebook_plotting
init_notebook_plotting()
[INFO 09-15 17:10:28] ax.utils.notebook.plotting: Injecting Plotly library into cell. Do not overwrite or delete cell.
Create a client object to interface with Ax APIs. By default this runs locally without storage.
ax_client = AxClient()
[INFO 09-15 17:10:28] ax.service.ax_client: Starting optimization with verbose logging. To disable logging, set the `verbose_logging` argument to `False`. Note that float values in the logs are rounded to 6 decimal points.
An experiment consists of a search space (parameters and parameter constraints) and optimization configuration (objective name, minimization setting, and outcome constraints). Note that:
name, parameters, and objective_name arguments are required.parameters have the following required keys: "name" - parameter name, "type" - parameter type ("range", "choice" or "fixed"), "bounds" for range parameters, "values" for choice parameters, and "value" for fixed parameters.parameters can optionally include "value_type" ("int", "float", "bool" or "str"), "log_scale" flag for range parameters, and "is_ordered" flag for choice parameters.parameter_constraints should be a list of strings of form "p1 >= p2" or "p1 + p2 <= some_bound".outcome_constraints should be a list of strings of form "constrained_metric <= some_bound".ax_client.create_experiment(
name="hartmann_test_experiment",
parameters=[
{
"name": "x1",
"type": "range",
"bounds": [0.0, 1.0],
"value_type": "float", # Optional, defaults to inference from type of "bounds".
"log_scale": False, # Optional, defaults to False.
},
{
"name": "x2",
"type": "range",
"bounds": [0.0, 1.0],
},
{
"name": "x3",
"type": "range",
"bounds": [0.0, 1.0],
},
{
"name": "x4",
"type": "range",
"bounds": [0.0, 1.0],
},
{
"name": "x5",
"type": "range",
"bounds": [0.0, 1.0],
},
{
"name": "x6",
"type": "range",
"bounds": [0.0, 1.0],
},
],
objective_name="hartmann6",
minimize=True, # Optional, defaults to False.
parameter_constraints=["x1 + x2 <= 2.0"], # Optional.
outcome_constraints=["l2norm <= 1.25"], # Optional.
)
[INFO 09-15 17:10:28] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x2. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 09-15 17:10:28] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x3. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 09-15 17:10:28] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x4. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 09-15 17:10:28] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x5. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 09-15 17:10:28] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x6. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 09-15 17:10:28] ax.service.utils.instantiation: Created search space: SearchSpace(parameters=[RangeParameter(name='x1', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x2', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x3', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x4', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x5', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x6', parameter_type=FLOAT, range=[0.0, 1.0])], parameter_constraints=[ParameterConstraint(1.0*x1 + 1.0*x2 <= 2.0)]).
[INFO 09-15 17:10:28] ax.modelbridge.dispatch_utils: Using Bayesian optimization since there are more ordered parameters than there are categories for the unordered categorical parameters.
[INFO 09-15 17:10:28] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+GPEI', steps=[Sobol for 12 trials, GPEI for subsequent trials]). Iterations after 12 will take longer to generate due to model-fitting.
When using Ax a service, evaluation of parameterizations suggested by Ax is done either locally or, more commonly, using an external scheduler. Below is a dummy evaluation function that outputs data for two metrics "hartmann6" and "l2norm". Note that all returned metrics correspond to either the objective_name set on experiment creation or the metric names mentioned in outcome_constraints.
import numpy as np
def evaluate(parameters):
x = np.array([parameters.get(f"x{i+1}") for i in range(6)])
# In our case, standard error is 0, since we are computing a synthetic function.
return {"hartmann6": (hartmann6(x), 0.0), "l2norm": (np.sqrt((x ** 2).sum()), 0.0)}
Result of the evaluation should generally be a mapping of the format: {metric_name -> (mean, SEM)}. If there is only one metric in the experiment – the objective – then evaluation function can return a single tuple of mean and SEM, in which case Ax will assume that evaluation corresponds to the objective. It can also return only the mean as a float, in which case Ax will treat SEM as unknown and use a model that can infer it.
For more details on evaluation function, refer to the "Trial Evaluation" section in the Ax docs at ax.dev
With the experiment set up, we can start the optimization loop.
At each step, the user queries the client for a new trial then submits the evaluation of that trial back to the client.
Note that Ax auto-selects an appropriate optimization algorithm based on the search space. For more advance use cases that require a specific optimization algorithm, pass a generation_strategy argument into the AxClient constructor. Note that when Bayesian Optimization is used, generating new trials may take a few minutes.
for i in range(25):
parameters, trial_index = ax_client.get_next_trial()
# Local evaluation here can be replaced with deployment to external system.
ax_client.complete_trial(trial_index=trial_index, raw_data=evaluate(parameters))
[INFO 09-15 17:10:28] ax.service.ax_client: Generated new trial 0 with parameters {'x1': 0.808197, 'x2': 0.515484, 'x3': 0.311732, 'x4': 0.927427, 'x5': 0.021217, 'x6': 0.141828}.
[INFO 09-15 17:10:28] ax.service.ax_client: Completed trial 0 with data: {'hartmann6': (-0.016945, 0.0), 'l2norm': (1.377233, 0.0)}.
[INFO 09-15 17:10:28] ax.service.ax_client: Generated new trial 1 with parameters {'x1': 0.751789, 'x2': 0.009418, 'x3': 0.935766, 'x4': 0.481968, 'x5': 0.276135, 'x6': 0.014904}.
[INFO 09-15 17:10:28] ax.service.ax_client: Completed trial 1 with data: {'hartmann6': (-0.017174, 0.0), 'l2norm': (1.322762, 0.0)}.
[INFO 09-15 17:10:28] ax.service.ax_client: Generated new trial 2 with parameters {'x1': 0.430394, 'x2': 0.453745, 'x3': 0.42868, 'x4': 0.041489, 'x5': 0.34033, 'x6': 0.030578}.
[INFO 09-15 17:10:28] ax.service.ax_client: Completed trial 2 with data: {'hartmann6': (-0.095472, 0.0), 'l2norm': (0.832689, 0.0)}.
[INFO 09-15 17:10:28] ax.service.ax_client: Generated new trial 3 with parameters {'x1': 0.09921, 'x2': 0.790143, 'x3': 0.532389, 'x4': 0.15217, 'x5': 0.418675, 'x6': 0.883382}.
[INFO 09-15 17:10:28] ax.service.ax_client: Completed trial 3 with data: {'hartmann6': (-0.416461, 0.0), 'l2norm': (1.377104, 0.0)}.
[INFO 09-15 17:10:28] ax.service.ax_client: Generated new trial 4 with parameters {'x1': 0.835693, 'x2': 0.075814, 'x3': 0.453538, 'x4': 0.14145, 'x5': 0.533597, 'x6': 0.462874}.
[INFO 09-15 17:10:28] ax.service.ax_client: Completed trial 4 with data: {'hartmann6': (-0.238305, 0.0), 'l2norm': (1.19533, 0.0)}.
[INFO 09-15 17:10:28] ax.service.ax_client: Generated new trial 5 with parameters {'x1': 0.619054, 'x2': 0.542657, 'x3': 0.266134, 'x4': 0.969628, 'x5': 0.721164, 'x6': 0.855231}.
[INFO 09-15 17:10:28] ax.service.ax_client: Completed trial 5 with data: {'hartmann6': (-0.000897, 0.0), 'l2norm': (1.714704, 0.0)}.
[INFO 09-15 17:10:28] ax.service.ax_client: Generated new trial 6 with parameters {'x1': 0.892539, 'x2': 0.383274, 'x3': 0.470328, 'x4': 0.133546, 'x5': 0.00762, 'x6': 0.743713}.
[INFO 09-15 17:10:28] ax.service.ax_client: Completed trial 6 with data: {'hartmann6': (-0.157822, 0.0), 'l2norm': (1.317473, 0.0)}.
[INFO 09-15 17:10:28] ax.service.ax_client: Generated new trial 7 with parameters {'x1': 0.923559, 'x2': 0.960248, 'x3': 0.605177, 'x4': 0.927958, 'x5': 0.780139, 'x6': 0.705904}.
[INFO 09-15 17:10:28] ax.service.ax_client: Completed trial 7 with data: {'hartmann6': (-0.000232, 0.0), 'l2norm': (2.02714, 0.0)}.
[INFO 09-15 17:10:28] ax.service.ax_client: Generated new trial 8 with parameters {'x1': 0.708208, 'x2': 0.330662, 'x3': 0.080302, 'x4': 0.558602, 'x5': 0.163857, 'x6': 0.759264}.
[INFO 09-15 17:10:28] ax.service.ax_client: Completed trial 8 with data: {'hartmann6': (-0.383489, 0.0), 'l2norm': (1.238027, 0.0)}.
[INFO 09-15 17:10:28] ax.service.ax_client: Generated new trial 9 with parameters {'x1': 0.293659, 'x2': 0.97249, 'x3': 0.875223, 'x4': 0.99334, 'x5': 0.336409, 'x6': 0.140827}.
[INFO 09-15 17:10:28] ax.service.ax_client: Completed trial 9 with data: {'hartmann6': (-0.360936, 0.0), 'l2norm': (1.708132, 0.0)}.
[INFO 09-15 17:10:28] ax.service.ax_client: Generated new trial 10 with parameters {'x1': 0.871813, 'x2': 0.776927, 'x3': 0.807538, 'x4': 0.643326, 'x5': 0.572012, 'x6': 0.67223}.
[INFO 09-15 17:10:28] ax.service.ax_client: Completed trial 10 with data: {'hartmann6': (-0.024997, 0.0), 'l2norm': (1.791298, 0.0)}.
[INFO 09-15 17:10:28] ax.service.ax_client: Generated new trial 11 with parameters {'x1': 0.265784, 'x2': 0.00613, 'x3': 0.388793, 'x4': 0.364144, 'x5': 0.640832, 'x6': 0.167751}.
[INFO 09-15 17:10:28] ax.service.ax_client: Completed trial 11 with data: {'hartmann6': (-0.126329, 0.0), 'l2norm': (0.890644, 0.0)}.
/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/botorch/optim/optimize.py:287: RuntimeWarning:
Optimization failed in `gen_candidates_scipy` with the following warning(s):
[OptimizationWarning('Optimization failed within `scipy.optimize.minimize` with status 9.')]
Trying again with a new set of initial conditions.
/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/botorch/optim/optimize.py:298: RuntimeWarning:
Optimization failed on the second try, after generating a new set of initial conditions.
[INFO 09-15 17:10:48] ax.service.ax_client: Generated new trial 12 with parameters {'x1': 0.59258, 'x2': 0.273142, 'x3': 0.058336, 'x4': 0.423357, 'x5': 0.184469, 'x6': 0.741409}.
[INFO 09-15 17:10:48] ax.service.ax_client: Completed trial 12 with data: {'hartmann6': (-1.036659, 0.0), 'l2norm': (1.091837, 0.0)}.
/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/botorch/optim/optimize.py:287: RuntimeWarning:
Optimization failed in `gen_candidates_scipy` with the following warning(s):
[OptimizationWarning('Optimization failed within `scipy.optimize.minimize` with status 9.')]
Trying again with a new set of initial conditions.
/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/botorch/optim/optimize.py:298: RuntimeWarning:
Optimization failed on the second try, after generating a new set of initial conditions.
[INFO 09-15 17:11:07] ax.service.ax_client: Generated new trial 13 with parameters {'x1': 0.551129, 'x2': 0.261791, 'x3': 0.039915, 'x4': 0.384436, 'x5': 0.18932, 'x6': 0.750001}.
[INFO 09-15 17:11:07] ax.service.ax_client: Completed trial 13 with data: {'hartmann6': (-1.232265, 0.0), 'l2norm': (1.058303, 0.0)}.
/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/botorch/optim/optimize.py:287: RuntimeWarning:
Optimization failed in `gen_candidates_scipy` with the following warning(s):
[OptimizationWarning('Optimization failed within `scipy.optimize.minimize` with status 9.')]
Trying again with a new set of initial conditions.
/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/botorch/optim/optimize.py:298: RuntimeWarning:
Optimization failed on the second try, after generating a new set of initial conditions.
[INFO 09-15 17:11:26] ax.service.ax_client: Generated new trial 14 with parameters {'x1': 0.504631, 'x2': 0.244459, 'x3': 0.017705, 'x4': 0.337493, 'x5': 0.191796, 'x6': 0.759005}.
[INFO 09-15 17:11:26] ax.service.ax_client: Completed trial 14 with data: {'hartmann6': (-1.410859, 0.0), 'l2norm': (1.02054, 0.0)}.
/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/botorch/optim/optimize.py:287: RuntimeWarning:
Optimization failed in `gen_candidates_scipy` with the following warning(s):
[OptimizationWarning('Optimization failed within `scipy.optimize.minimize` with status 9.')]
Trying again with a new set of initial conditions.
/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/botorch/optim/optimize.py:298: RuntimeWarning:
Optimization failed on the second try, after generating a new set of initial conditions.
[INFO 09-15 17:11:45] ax.service.ax_client: Generated new trial 15 with parameters {'x1': 0.446076, 'x2': 0.213803, 'x3': 0.0, 'x4': 0.273792, 'x5': 0.196028, 'x6': 0.765181}.
[INFO 09-15 17:11:45] ax.service.ax_client: Completed trial 15 with data: {'hartmann6': (-1.57544, 0.0), 'l2norm': (0.971384, 0.0)}.
[INFO 09-15 17:11:54] ax.service.ax_client: Generated new trial 16 with parameters {'x1': 0.375206, 'x2': 0.165681, 'x3': 0.001586, 'x4': 0.219009, 'x5': 0.21342, 'x6': 0.751781}.
[INFO 09-15 17:11:54] ax.service.ax_client: Completed trial 16 with data: {'hartmann6': (-1.785514, 0.0), 'l2norm': (0.909352, 0.0)}.
/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/botorch/optim/optimize.py:287: RuntimeWarning:
Optimization failed in `gen_candidates_scipy` with the following warning(s):
[OptimizationWarning('Optimization failed within `scipy.optimize.minimize` with status 9.')]
Trying again with a new set of initial conditions.
/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/botorch/optim/optimize.py:298: RuntimeWarning:
Optimization failed on the second try, after generating a new set of initial conditions.
[INFO 09-15 17:12:13] ax.service.ax_client: Generated new trial 17 with parameters {'x1': 0.300877, 'x2': 0.115408, 'x3': 0.024442, 'x4': 0.211551, 'x5': 0.24803, 'x6': 0.706709}.
[INFO 09-15 17:12:13] ax.service.ax_client: Completed trial 17 with data: {'hartmann6': (-2.167579, 0.0), 'l2norm': (0.842706, 0.0)}.
[INFO 09-15 17:12:22] ax.service.ax_client: Generated new trial 18 with parameters {'x1': 0.233536, 'x2': 0.083638, 'x3': 0.038716, 'x4': 0.234747, 'x5': 0.279406, 'x6': 0.658766}.
[INFO 09-15 17:12:22] ax.service.ax_client: Completed trial 18 with data: {'hartmann6': (-2.411644, 0.0), 'l2norm': (0.793839, 0.0)}.
[INFO 09-15 17:12:30] ax.service.ax_client: Generated new trial 19 with parameters {'x1': 0.146209, 'x2': 0.005342, 'x3': 0.048771, 'x4': 0.278852, 'x5': 0.278569, 'x6': 0.653842}.
[INFO 09-15 17:12:30] ax.service.ax_client: Completed trial 19 with data: {'hartmann6': (-2.313867, 0.0), 'l2norm': (0.778879, 0.0)}.
[INFO 09-15 17:12:39] ax.service.ax_client: Generated new trial 20 with parameters {'x1': 0.189787, 'x2': 0.119162, 'x3': 0.00199, 'x4': 0.211476, 'x5': 0.31105, 'x6': 0.612905}.
[INFO 09-15 17:12:39] ax.service.ax_client: Completed trial 20 with data: {'hartmann6': (-2.228972, 0.0), 'l2norm': (0.753226, 0.0)}.
[INFO 09-15 17:12:49] ax.service.ax_client: Generated new trial 21 with parameters {'x1': 0.24814, 'x2': 0.059964, 'x3': 0.106228, 'x4': 0.269188, 'x5': 0.291451, 'x6': 0.642139}.
[INFO 09-15 17:12:49] ax.service.ax_client: Completed trial 21 with data: {'hartmann6': (-2.621875, 0.0), 'l2norm': (0.803867, 0.0)}.
[INFO 09-15 17:12:58] ax.service.ax_client: Generated new trial 22 with parameters {'x1': 0.223991, 'x2': 0.081532, 'x3': 0.166045, 'x4': 0.219476, 'x5': 0.293558, 'x6': 0.633415}.
[INFO 09-15 17:12:58] ax.service.ax_client: Completed trial 22 with data: {'hartmann6': (-2.666839, 0.0), 'l2norm': (0.78737, 0.0)}.
[INFO 09-15 17:13:07] ax.service.ax_client: Generated new trial 23 with parameters {'x1': 0.247451, 'x2': 0.043382, 'x3': 0.166454, 'x4': 0.224677, 'x5': 0.356192, 'x6': 0.656468}.
[INFO 09-15 17:13:07] ax.service.ax_client: Completed trial 23 with data: {'hartmann6': (-2.537283, 0.0), 'l2norm': (0.836136, 0.0)}.
[INFO 09-15 17:13:15] ax.service.ax_client: Generated new trial 24 with parameters {'x1': 0.257378, 'x2': 0.034039, 'x3': 0.156994, 'x4': 0.212263, 'x5': 0.248616, 'x6': 0.600127}.
[INFO 09-15 17:13:15] ax.service.ax_client: Completed trial 24 with data: {'hartmann6': (-2.352214, 0.0), 'l2norm': (0.747708, 0.0)}.
By default, Ax restricts number of trials that can run in parallel for some optimization stages, in order to improve the optimization performance and reduce the number of trials that the optimization will require. To check the maximum parallelism for each optimization stage:
ax_client.get_max_parallelism()
[(12, 12), (-1, 3)]
The output of this function is a list of tuples of form (number of trials, max parallelism), so the example above means "the max parallelism is 12 for the first 12 trials and 3 for all subsequent trials." This is because the first 12 trials are produced quasi-randomly and can all be evaluated at once, and subsequent trials are produced via Bayesian optimization, which converges on optimal point in fewer trials when parallelism is limited. MaxParallelismReachedException indicates that the parallelism limit has been reached –– refer to the 'Service API Exceptions Meaning and Handling' section at the end of the tutorial for handling.
ax_client.generation_strategy.trials_as_df
[INFO 09-15 17:13:15] ax.modelbridge.generation_strategy: Note that parameter values in dataframe are rounded to 2 decimal points; the values in the dataframe are thus not the exact ones suggested by Ax in trials.
| Generation Step | Generation Model | Trial Index | Trial Status | Arm Parameterizations | |
|---|---|---|---|---|---|
| 0 | 0 | Sobol | 0 | COMPLETED | {'0_0': {'x1': 0.81, 'x2': 0.52, 'x3': 0.31, '... |
| 1 | 0 | Sobol | 1 | COMPLETED | {'1_0': {'x1': 0.75, 'x2': 0.01, 'x3': 0.94, '... |
| 2 | 0 | Sobol | 2 | COMPLETED | {'2_0': {'x1': 0.43, 'x2': 0.45, 'x3': 0.43, '... |
| 3 | 0 | Sobol | 3 | COMPLETED | {'3_0': {'x1': 0.1, 'x2': 0.79, 'x3': 0.53, 'x... |
| 4 | 0 | Sobol | 4 | COMPLETED | {'4_0': {'x1': 0.84, 'x2': 0.08, 'x3': 0.45, '... |
| 5 | 0 | Sobol | 5 | COMPLETED | {'5_0': {'x1': 0.62, 'x2': 0.54, 'x3': 0.27, '... |
| 6 | 0 | Sobol | 6 | COMPLETED | {'6_0': {'x1': 0.89, 'x2': 0.38, 'x3': 0.47, '... |
| 7 | 0 | Sobol | 7 | COMPLETED | {'7_0': {'x1': 0.92, 'x2': 0.96, 'x3': 0.61, '... |
| 8 | 0 | Sobol | 8 | COMPLETED | {'8_0': {'x1': 0.71, 'x2': 0.33, 'x3': 0.08, '... |
| 9 | 0 | Sobol | 9 | COMPLETED | {'9_0': {'x1': 0.29, 'x2': 0.97, 'x3': 0.88, '... |
| 10 | 0 | Sobol | 10 | COMPLETED | {'10_0': {'x1': 0.87, 'x2': 0.78, 'x3': 0.81, ... |
| 11 | 0 | Sobol | 11 | COMPLETED | {'11_0': {'x1': 0.27, 'x2': 0.01, 'x3': 0.39, ... |
| 12 | 1 | GPEI | 12 | COMPLETED | {'12_0': {'x1': 0.59, 'x2': 0.27, 'x3': 0.06, ... |
| 13 | 1 | GPEI | 13 | COMPLETED | {'13_0': {'x1': 0.55, 'x2': 0.26, 'x3': 0.04, ... |
| 14 | 1 | GPEI | 14 | COMPLETED | {'14_0': {'x1': 0.5, 'x2': 0.24, 'x3': 0.02, '... |
| 15 | 1 | GPEI | 15 | COMPLETED | {'15_0': {'x1': 0.45, 'x2': 0.21, 'x3': 0.0, '... |
| 16 | 1 | GPEI | 16 | COMPLETED | {'16_0': {'x1': 0.38, 'x2': 0.17, 'x3': 0.0, '... |
| 17 | 1 | GPEI | 17 | COMPLETED | {'17_0': {'x1': 0.3, 'x2': 0.12, 'x3': 0.02, '... |
| 18 | 1 | GPEI | 18 | COMPLETED | {'18_0': {'x1': 0.23, 'x2': 0.08, 'x3': 0.04, ... |
| 19 | 1 | GPEI | 19 | COMPLETED | {'19_0': {'x1': 0.15, 'x2': 0.01, 'x3': 0.05, ... |
| 20 | 1 | GPEI | 20 | COMPLETED | {'20_0': {'x1': 0.19, 'x2': 0.12, 'x3': 0.0, '... |
| 21 | 1 | GPEI | 21 | COMPLETED | {'21_0': {'x1': 0.25, 'x2': 0.06, 'x3': 0.11, ... |
| 22 | 1 | GPEI | 22 | COMPLETED | {'22_0': {'x1': 0.22, 'x2': 0.08, 'x3': 0.17, ... |
| 23 | 1 | GPEI | 23 | COMPLETED | {'23_0': {'x1': 0.25, 'x2': 0.04, 'x3': 0.17, ... |
| 24 | 1 | GPEI | 24 | COMPLETED | {'24_0': {'x1': 0.26, 'x2': 0.03, 'x3': 0.16, ... |
Once it's complete, we can access the best parameters found, as well as the corresponding metric values.
best_parameters, values = ax_client.get_best_parameters()
best_parameters
{'x1': 0.22399122198464227,
'x2': 0.08153161805289375,
'x3': 0.16604511225609692,
'x4': 0.21947589985065627,
'x5': 0.2935581235385248,
'x6': 0.6334151028417162}
means, covariances = values
means
{'l2norm': 0.7873707188905149, 'hartmann6': -2.6668265555499846}
For comparison, Hartmann6 minimum:
hartmann6.fmin
-3.32237
Here we arbitrarily select "x1" and "x2" as the two parameters to plot for both metrics, "hartmann6" and "l2norm".
render(ax_client.get_contour_plot())
[INFO 09-15 17:13:16] ax.service.ax_client: Retrieving contour plot with parameter 'x1' on X-axis and 'x2' on Y-axis, for metric 'hartmann6'. Remaining parameters are affixed to the middle of their range.
We can also retrieve a contour plot for the other metric, "l2norm" –– say, we are interested in seeing the response surface for parameters "x3" and "x4" for this one.
render(ax_client.get_contour_plot(param_x="x3", param_y="x4", metric_name="l2norm"))
[INFO 09-15 17:13:17] ax.service.ax_client: Retrieving contour plot with parameter 'x3' on X-axis and 'x4' on Y-axis, for metric 'l2norm'. Remaining parameters are affixed to the middle of their range.
Here we plot the optimization trace, showing the progression of finding the point with the optimal objective:
render(ax_client.get_optimization_trace(objective_optimum=hartmann6.fmin)) # Objective_optimum is optional.
We can serialize the state of optimization to JSON and save it to a .json file or save it to the SQL backend. For the former:
ax_client.save_to_json_file() # For custom filepath, pass `filepath` argument.
[INFO 09-15 17:13:18] ax.service.ax_client: Saved JSON-serialized state of optimization to `ax_client_snapshot.json`.
restored_ax_client = AxClient.load_from_json_file() # For custom filepath, pass `filepath` argument.
[INFO 09-15 17:13:18] ax.service.ax_client: Starting optimization with verbose logging. To disable logging, set the `verbose_logging` argument to `False`. Note that float values in the logs are rounded to 6 decimal points.
To store state of optimization to an SQL backend, first follow setup instructions on Ax website.
Having set up the SQL backend, pass DBSettings to AxClient on instantiation (note that SQLAlchemy dependency will have to be installed – for installation, refer to optional dependencies on Ax website):
from ax.storage.sqa_store.structs import DBSettings
# URL is of the form "dialect+driver://username:password@host:port/database".
db_settings = DBSettings(url="sqlite:///foo.db")
# Instead of URL, can provide a `creator function`; can specify custom encoders/decoders if necessary.
new_ax = AxClient(db_settings=db_settings)
[INFO 09-15 17:13:18] ax.service.ax_client: Starting optimization with verbose logging. To disable logging, set the `verbose_logging` argument to `False`. Note that float values in the logs are rounded to 6 decimal points.
When valid DBSettings are passed into AxClient, a unique experiment name is a required argument (name) to ax_client.create_experiment. The state of the optimization is auto-saved any time it changes (i.e. a new trial is added or completed, etc).
To reload an optimization state later, instantiate AxClient with the same DBSettings and use ax_client.load_experiment_from_database(experiment_name="my_experiment").
Evaluation failure: should any optimization iterations fail during evaluation, log_trial_failure will ensure that the same trial is not proposed again.
_, trial_index = ax_client.get_next_trial()
ax_client.log_trial_failure(trial_index=trial_index)
[INFO 09-15 17:13:26] ax.service.ax_client: Generated new trial 25 with parameters {'x1': 0.218766, 'x2': 0.108186, 'x3': 0.168311, 'x4': 0.287487, 'x5': 0.307756, 'x6': 0.679613}.
[INFO 09-15 17:13:26] ax.service.ax_client: Registered failure of trial 25.
Adding custom trials: should there be need to evaluate a specific parameterization, attach_trial will add it to the experiment.
ax_client.attach_trial(parameters={"x1": 0.9, "x2": 0.9, "x3": 0.9, "x4": 0.9, "x5": 0.9, "x6": 0.9})
[INFO 09-15 17:13:26] ax.service.ax_client: Attached custom parameterization {'x1': 0.9, 'x2': 0.9, 'x3': 0.9, 'x4': 0.9, 'x5': 0.9, 'x6': 0.9} as trial 26.
({'x1': 0.9, 'x2': 0.9, 'x3': 0.9, 'x4': 0.9, 'x5': 0.9, 'x6': 0.9}, 26)
Need to run many trials in parallel: for optimal results and optimization efficiency, we strongly recommend sequential optimization (generating a few trials, then waiting for them to be completed with evaluation data). However, if your use case needs to dispatch many trials in parallel before they are updated with data and you are running into the "All trials for current model have been generated, but not enough data has been observed to fit next model" error, instantiate AxClient as AxClient(enforce_sequential_optimization=False).
DataRequiredError: Ax generation strategy needs to be updated with more data to proceed to the next optimization model. When the optimization moves from initialization stage to the Bayesian optimization stage, the underlying BayesOpt model needs sufficient data to train. For optimal results and optimization efficiency (finding the optimal point in the least number of trials), we recommend sequential optimization (generating a few trials, then waiting for them to be completed with evaluation data). Therefore, the correct way to handle this exception is to wait until more trial evaluations complete and log their data via ax_client.complete_trial(...).
However, if there is strong need to generate more trials before more data is available, instantiate AxClient as AxClient(enforce_sequential_optimization=False). With this setting, as many trials will be generated from the initialization stage as requested, and the optimization will move to the BayesOpt stage whenever enough trials are completed.
MaxParallelismReachedException: generation strategy restricts the number of trials that can be ran simultaneously (to encourage sequential optimization), and the parallelism limit has been reached. The correct way to handle this exception is the same as DataRequiredError – to wait until more trial evluations complete and log their data via ax_client.complete_trial(...).
In some cases higher parallelism is important, so enforce_sequential_optimization=False kwarg to AxClient allows to suppress limiting of parallelism. It's also possible to override the default parallelism setting for all stages of the optimization by passing choose_generation_strategy_kwargs to ax_client.create_experiment:
ax_client = AxClient()
ax_client.create_experiment(
parameters=[
{"name": "x", "type": "range", "bounds": [-5.0, 10.0]},
{"name": "y", "type": "range", "bounds": [0.0, 15.0]},
],
# Sets max parallelism to 10 for all steps of the generation strategy.
choose_generation_strategy_kwargs={"max_parallelism_override": 10},
)
[INFO 09-15 17:13:26] ax.service.ax_client: Starting optimization with verbose logging. To disable logging, set the `verbose_logging` argument to `False`. Note that float values in the logs are rounded to 6 decimal points.
[INFO 09-15 17:13:26] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 09-15 17:13:26] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter y. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 09-15 17:13:26] ax.service.utils.instantiation: Created search space: SearchSpace(parameters=[RangeParameter(name='x', parameter_type=FLOAT, range=[-5.0, 10.0]), RangeParameter(name='y', parameter_type=FLOAT, range=[0.0, 15.0])], parameter_constraints=[]).
[INFO 09-15 17:13:26] ax.modelbridge.dispatch_utils: Using Bayesian optimization since there are more ordered parameters than there are categories for the unordered categorical parameters.
[INFO 09-15 17:13:26] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+GPEI', steps=[Sobol for 5 trials, GPEI for subsequent trials]). Iterations after 5 will take longer to generate due to model-fitting.
ax_client.get_max_parallelism() # Max parallelism is now 10 for all stages of the optimization.
[(5, 10), (-1, 10)]
Total runtime of script: 3 minutes, 4.09 seconds.