{ "cells": [ { "cell_type": "markdown", "id": "dd7113a9", "metadata": { "papermill": { "duration": 0.005669, "end_time": "2024-03-01T16:33:56.179804", "exception": false, "start_time": "2024-03-01T16:33:56.174135", "status": "completed" }, "tags": [] }, "source": [ "# Service API Example on Hartmann6\n", "\n", "The Ax Service API is designed to allow the user to control scheduling of trials and data computation while having an easy to use interface with Ax.\n", "\n", "The user iteratively:\n", "- Queries Ax for candidates\n", "- Schedules / deploys them however they choose\n", "- Computes data and logs to Ax\n", "- Repeat" ] }, { "cell_type": "code", "execution_count": 1, "id": "b6d6fdb5", "metadata": { "execution": { "iopub.execute_input": "2024-03-01T16:33:56.190918Z", "iopub.status.busy": "2024-03-01T16:33:56.190297Z", "iopub.status.idle": "2024-03-01T16:33:59.629614Z", "shell.execute_reply": "2024-03-01T16:33:59.628679Z" }, "papermill": { "duration": 3.461919, "end_time": "2024-03-01T16:33:59.646540", "exception": false, "start_time": "2024-03-01T16:33:56.184621", "status": "completed" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:33:59] ax.utils.notebook.plotting: Injecting Plotly library into cell. Do not overwrite or delete cell.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:33:59] ax.utils.notebook.plotting: Please see\n", " (https://ax.dev/tutorials/visualizations.html#Fix-for-plots-that-are-not-rendering)\n", " if visualizations are not rendering.\n" ] }, { "data": { "text/html": [ " \n", " " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from ax.service.ax_client import AxClient, ObjectiveProperties\n", "from ax.utils.measurement.synthetic_functions import hartmann6\n", "from ax.utils.notebook.plotting import init_notebook_plotting, render\n", "\n", "init_notebook_plotting()" ] }, { "cell_type": "markdown", "id": "af935514", "metadata": { "papermill": { "duration": 0.043637, "end_time": "2024-03-01T16:33:59.730907", "exception": false, "start_time": "2024-03-01T16:33:59.687270", "status": "completed" }, "tags": [] }, "source": [ "## 1. Initialize client\n", "\n", "Create a client object to interface with Ax APIs. By default this runs locally without storage." ] }, { "cell_type": "code", "execution_count": 2, "id": "1f299262", "metadata": { "execution": { "iopub.execute_input": "2024-03-01T16:33:59.820045Z", "iopub.status.busy": "2024-03-01T16:33:59.819563Z", "iopub.status.idle": "2024-03-01T16:33:59.823851Z", "shell.execute_reply": "2024-03-01T16:33:59.823179Z" }, "papermill": { "duration": 0.050741, "end_time": "2024-03-01T16:33:59.825229", "exception": false, "start_time": "2024-03-01T16:33:59.774488", "status": "completed" }, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:33:59] ax.service.ax_client: Starting optimization with verbose logging. To disable logging, set the `verbose_logging` argument to `False`. Note that float values in the logs are rounded to 6 decimal points.\n" ] } ], "source": [ "ax_client = AxClient()" ] }, { "cell_type": "markdown", "id": "e9885c8a", "metadata": { "papermill": { "duration": 0.044178, "end_time": "2024-03-01T16:33:59.913002", "exception": false, "start_time": "2024-03-01T16:33:59.868824", "status": "completed" }, "tags": [] }, "source": [ "## 2. Set up experiment\n", "An experiment consists of a **search space** (parameters and parameter constraints) and **optimization configuration** (objectives and outcome constraints). Note that:\n", "- Only `parameters`, and `objectives` arguments are required.\n", "- Dictionaries in `parameters` have the following required keys: \"name\" - parameter name, \"type\" - parameter type (\"range\", \"choice\" or \"fixed\"), \"bounds\" for range parameters, \"values\" for choice parameters, and \"value\" for fixed parameters.\n", "- Dictionaries in `parameters` can optionally include \"value_type\" (\"int\", \"float\", \"bool\" or \"str\"), \"log_scale\" flag for range parameters, and \"is_ordered\" flag for choice parameters.\n", "- `parameter_constraints` should be a list of strings of form \"p1 >= p2\" or \"p1 + p2 <= some_bound\".\n", "- `outcome_constraints` should be a list of strings of form \"constrained_metric <= some_bound\"." ] }, { "cell_type": "code", "execution_count": 3, "id": "bdb4a3ba", "metadata": { "execution": { "iopub.execute_input": "2024-03-01T16:34:00.005365Z", "iopub.status.busy": "2024-03-01T16:34:00.004738Z", "iopub.status.idle": "2024-03-01T16:34:00.021597Z", "shell.execute_reply": "2024-03-01T16:34:00.020868Z" }, "papermill": { "duration": 0.06498, "end_time": "2024-03-01T16:34:00.023059", "exception": false, "start_time": "2024-03-01T16:33:59.958079", "status": "completed" }, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x2. If that is not the expected value type, you can explicitly specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x3. If that is not the expected value type, you can explicitly specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x4. If that is not the expected value type, you can explicitly specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x5. If that is not the expected value type, you can explicitly specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x6. If that is not the expected value type, you can explicitly specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.utils.instantiation: Created search space: SearchSpace(parameters=[RangeParameter(name='x1', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x2', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x3', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x4', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x5', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x6', parameter_type=FLOAT, range=[0.0, 1.0])], parameter_constraints=[ParameterConstraint(1.0*x1 + 1.0*x2 <= 2.0)]).\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.modelbridge.dispatch_utils: Using Models.BOTORCH_MODULAR since there is at least one ordered parameter and there are no unordered categorical parameters.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.modelbridge.dispatch_utils: Calculating the number of remaining initialization trials based on num_initialization_trials=None max_initialization_trials=None num_tunable_parameters=6 num_trials=None use_batch_trials=False\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.modelbridge.dispatch_utils: calculated num_initialization_trials=12\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.modelbridge.dispatch_utils: num_completed_initialization_trials=0 num_remaining_initialization_trials=12\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.modelbridge.dispatch_utils: `verbose`, `disable_progbar`, and `jit_compile` are not yet supported when using `choose_generation_strategy` with ModularBoTorchModel, dropping these arguments.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+BoTorch', steps=[Sobol for 12 trials, BoTorch for subsequent trials]). Iterations after 12 will take longer to generate due to model-fitting.\n" ] } ], "source": [ "ax_client.create_experiment(\n", " name=\"hartmann_test_experiment\",\n", " parameters=[\n", " {\n", " \"name\": \"x1\",\n", " \"type\": \"range\",\n", " \"bounds\": [0.0, 1.0],\n", " \"value_type\": \"float\", # Optional, defaults to inference from type of \"bounds\".\n", " \"log_scale\": False, # Optional, defaults to False.\n", " },\n", " {\n", " \"name\": \"x2\",\n", " \"type\": \"range\",\n", " \"bounds\": [0.0, 1.0],\n", " },\n", " {\n", " \"name\": \"x3\",\n", " \"type\": \"range\",\n", " \"bounds\": [0.0, 1.0],\n", " },\n", " {\n", " \"name\": \"x4\",\n", " \"type\": \"range\",\n", " \"bounds\": [0.0, 1.0],\n", " },\n", " {\n", " \"name\": \"x5\",\n", " \"type\": \"range\",\n", " \"bounds\": [0.0, 1.0],\n", " },\n", " {\n", " \"name\": \"x6\",\n", " \"type\": \"range\",\n", " \"bounds\": [0.0, 1.0],\n", " },\n", " ],\n", " objectives={\"hartmann6\": ObjectiveProperties(minimize=True)},\n", " parameter_constraints=[\"x1 + x2 <= 2.0\"], # Optional.\n", " outcome_constraints=[\"l2norm <= 1.25\"], # Optional.\n", ")" ] }, { "cell_type": "markdown", "id": "76b7f5e7", "metadata": { "papermill": { "duration": 0.045464, "end_time": "2024-03-01T16:34:00.114533", "exception": false, "start_time": "2024-03-01T16:34:00.069069", "status": "completed" }, "tags": [] }, "source": [ "## 3. Define how to evaluate trials\n", "When using Ax a service, evaluation of parameterizations suggested by Ax is done either locally or, more commonly, using an external scheduler. Below is a dummy evaluation function that outputs data for two metrics \"hartmann6\" and \"l2norm\". Note that all returned metrics correspond to either the `objectives` set on experiment creation or the metric names mentioned in `outcome_constraints`." ] }, { "cell_type": "code", "execution_count": 4, "id": "0f7a4550", "metadata": { "execution": { "iopub.execute_input": "2024-03-01T16:34:00.205757Z", "iopub.status.busy": "2024-03-01T16:34:00.205250Z", "iopub.status.idle": "2024-03-01T16:34:00.209566Z", "shell.execute_reply": "2024-03-01T16:34:00.208893Z" }, "papermill": { "duration": 0.051169, "end_time": "2024-03-01T16:34:00.210855", "exception": false, "start_time": "2024-03-01T16:34:00.159686", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "import numpy as np\n", "\n", "\n", "def evaluate(parameterization):\n", " x = np.array([parameterization.get(f\"x{i+1}\") for i in range(6)])\n", " # In our case, standard error is 0, since we are computing a synthetic function.\n", " return {\"hartmann6\": (hartmann6(x), 0.0), \"l2norm\": (np.sqrt((x**2).sum()), 0.0)}" ] }, { "cell_type": "markdown", "id": "1a0d64d2", "metadata": { "papermill": { "duration": 0.044635, "end_time": "2024-03-01T16:34:00.300426", "exception": false, "start_time": "2024-03-01T16:34:00.255791", "status": "completed" }, "tags": [] }, "source": [ "Result of the evaluation should generally be a mapping of the format: `{metric_name -> (mean, SEM)}`. If there is only one metric in the experiment – the objective – then evaluation function can return a single tuple of mean and SEM, in which case Ax will assume that evaluation corresponds to the objective. _It can also return only the mean as a float, in which case Ax will treat SEM as unknown and use a model that can infer it._ \n", "\n", "For more details on evaluation function, refer to the \"Trial Evaluation\" section in the Ax docs at [ax.dev](https://ax.dev/)" ] }, { "cell_type": "markdown", "id": "c24a1b0e", "metadata": { "papermill": { "duration": 0.045176, "end_time": "2024-03-01T16:34:00.430165", "exception": false, "start_time": "2024-03-01T16:34:00.384989", "status": "completed" }, "tags": [] }, "source": [ "## 4. Run optimization loop\n", "With the experiment set up, we can start the optimization loop.\n", "\n", "At each step, the user queries the client for a new trial then submits the evaluation of that trial back to the client.\n", "\n", "Note that Ax auto-selects an appropriate optimization algorithm based on the search space. For more advance use cases that require a specific optimization algorithm, pass a `generation_strategy` argument into the `AxClient` constructor. Note that when Bayesian Optimization is used, generating new trials may take a few minutes." ] }, { "cell_type": "code", "execution_count": 5, "id": "da516c9a", "metadata": { "execution": { "iopub.execute_input": "2024-03-01T16:34:00.522657Z", "iopub.status.busy": "2024-03-01T16:34:00.521895Z", "iopub.status.idle": "2024-03-01T16:35:35.808356Z", "shell.execute_reply": "2024-03-01T16:35:35.807623Z" }, "papermill": { "duration": 95.334745, "end_time": "2024-03-01T16:35:35.810172", "exception": false, "start_time": "2024-03-01T16:34:00.475427", "status": "completed" }, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.ax_client: Generated new trial 0 with parameters {'x1': 0.979432, 'x2': 0.784923, 'x3': 0.297002, 'x4': 0.430659, 'x5': 0.786892, 'x6': 0.07397} using model Sobol.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.ax_client: Completed trial 0 with data: {'hartmann6': (-0.008235, 0.0), 'l2norm': (1.572813, 0.0)}.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.ax_client: Generated new trial 1 with parameters {'x1': 0.249398, 'x2': 0.568361, 'x3': 0.531979, 'x4': 0.271478, 'x5': 0.804936, 'x6': 0.230381} using model Sobol.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.ax_client: Completed trial 1 with data: {'hartmann6': (-0.373106, 0.0), 'l2norm': (1.201222, 0.0)}.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.ax_client: Generated new trial 2 with parameters {'x1': 0.004508, 'x2': 0.274208, 'x3': 0.350825, 'x4': 0.503775, 'x5': 0.874895, 'x6': 0.905855} using model Sobol.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.ax_client: Completed trial 2 with data: {'hartmann6': (-0.080531, 0.0), 'l2norm': (1.427618, 0.0)}.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.ax_client: Generated new trial 3 with parameters {'x1': 0.029083, 'x2': 0.393902, 'x3': 0.595405, 'x4': 0.916642, 'x5': 0.915106, 'x6': 0.568628} using model Sobol.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.ax_client: Completed trial 3 with data: {'hartmann6': (-0.042971, 0.0), 'l2norm': (1.584771, 0.0)}.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.ax_client: Generated new trial 4 with parameters {'x1': 0.572154, 'x2': 0.218557, 'x3': 0.024659, 'x4': 0.077701, 'x5': 0.823543, 'x6': 0.654053} using model Sobol.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.ax_client: Completed trial 4 with data: {'hartmann6': (-0.012623, 0.0), 'l2norm': (1.219746, 0.0)}.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.ax_client: Generated new trial 5 with parameters {'x1': 0.912443, 'x2': 0.410203, 'x3': 0.956054, 'x4': 0.558295, 'x5': 0.143371, 'x6': 0.673672} using model Sobol.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.ax_client: Completed trial 5 with data: {'hartmann6': (-0.298741, 0.0), 'l2norm': (1.643454, 0.0)}.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.ax_client: Generated new trial 6 with parameters {'x1': 0.470742, 'x2': 0.809175, 'x3': 0.888021, 'x4': 0.923421, 'x5': 0.008376, 'x6': 0.851274} using model Sobol.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.ax_client: Completed trial 6 with data: {'hartmann6': (-0.159117, 0.0), 'l2norm': (1.800663, 0.0)}.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.ax_client: Generated new trial 7 with parameters {'x1': 0.953139, 'x2': 0.757169, 'x3': 0.884338, 'x4': 0.19917, 'x5': 0.63656, 'x6': 0.947568} using model Sobol.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.ax_client: Completed trial 7 with data: {'hartmann6': (-0.04091, 0.0), 'l2norm': (1.899104, 0.0)}.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.ax_client: Generated new trial 8 with parameters {'x1': 0.065875, 'x2': 0.830336, 'x3': 0.209183, 'x4': 0.519807, 'x5': 0.39785, 'x6': 0.210257} using model Sobol.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.ax_client: Completed trial 8 with data: {'hartmann6': (-0.329269, 0.0), 'l2norm': (1.100112, 0.0)}.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.ax_client: Generated new trial 9 with parameters {'x1': 0.538918, 'x2': 0.988224, 'x3': 0.941543, 'x4': 0.282868, 'x5': 0.189363, 'x6': 0.421953} using model Sobol.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.ax_client: Completed trial 9 with data: {'hartmann6': (-0.170027, 0.0), 'l2norm': (1.56443, 0.0)}.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.ax_client: Generated new trial 10 with parameters {'x1': 0.591468, 'x2': 0.371087, 'x3': 0.92552, 'x4': 0.358084, 'x5': 0.495212, 'x6': 0.769208} using model Sobol.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.ax_client: Completed trial 10 with data: {'hartmann6': (-0.603883, 0.0), 'l2norm': (1.519628, 0.0)}.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.ax_client: Generated new trial 11 with parameters {'x1': 0.690951, 'x2': 0.030641, 'x3': 0.216716, 'x4': 0.716453, 'x5': 0.291473, 'x6': 0.575743} using model Sobol.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:00] ax.service.ax_client: Completed trial 11 with data: {'hartmann6': (-0.223249, 0.0), 'l2norm': (1.206258, 0.0)}.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:08] ax.service.ax_client: Generated new trial 12 with parameters {'x1': 0.177877, 'x2': 0.674407, 'x3': 0.421198, 'x4': 0.338802, 'x5': 0.594461, 'x6': 0.219123} using model BoTorch.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:08] ax.service.ax_client: Completed trial 12 with data: {'hartmann6': (-0.45371, 0.0), 'l2norm': (1.086305, 0.0)}.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:16] ax.service.ax_client: Generated new trial 13 with parameters {'x1': 0.231516, 'x2': 0.616074, 'x3': 0.476394, 'x4': 0.335011, 'x5': 0.465578, 'x6': 0.263612} using model BoTorch.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:16] ax.service.ax_client: Completed trial 13 with data: {'hartmann6': (-0.634692, 0.0), 'l2norm': (1.028875, 0.0)}.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:26] ax.service.ax_client: Generated new trial 14 with parameters {'x1': 0.291882, 'x2': 0.5522, 'x3': 0.568437, 'x4': 0.331709, 'x5': 0.410356, 'x6': 0.333812} using model BoTorch.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:26] ax.service.ax_client: Completed trial 14 with data: {'hartmann6': (-0.845565, 0.0), 'l2norm': (1.050283, 0.0)}.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:35] ax.service.ax_client: Generated new trial 15 with parameters {'x1': 0.33613, 'x2': 0.500325, 'x3': 0.646939, 'x4': 0.333627, 'x5': 0.377844, 'x6': 0.398972} using model BoTorch.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:35] ax.service.ax_client: Completed trial 15 with data: {'hartmann6': (-1.056369, 0.0), 'l2norm': (1.093202, 0.0)}.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:42] ax.service.ax_client: Generated new trial 16 with parameters {'x1': 0.366051, 'x2': 0.451651, 'x3': 0.71543, 'x4': 0.335184, 'x5': 0.347802, 'x6': 0.453928} using model BoTorch.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:42] ax.service.ax_client: Completed trial 16 with data: {'hartmann6': (-1.251565, 0.0), 'l2norm': (1.135424, 0.0)}.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:48] ax.service.ax_client: Generated new trial 17 with parameters {'x1': 0.369722, 'x2': 0.396793, 'x3': 0.776089, 'x4': 0.331656, 'x5': 0.300074, 'x6': 0.485381} using model BoTorch.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:48] ax.service.ax_client: Completed trial 17 with data: {'hartmann6': (-1.392822, 0.0), 'l2norm': (1.154161, 0.0)}.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:53] ax.service.ax_client: Generated new trial 18 with parameters {'x1': 0.336904, 'x2': 0.344148, 'x3': 0.819443, 'x4': 0.31952, 'x5': 0.248514, 'x6': 0.469396} using model BoTorch.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:34:53] ax.service.ax_client: Completed trial 18 with data: {'hartmann6': (-1.303993, 0.0), 'l2norm': (1.134731, 0.0)}.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:35:03] ax.service.ax_client: Generated new trial 19 with parameters {'x1': 0.396041, 'x2': 0.393813, 'x3': 0.761329, 'x4': 0.341066, 'x5': 0.261872, 'x6': 0.548757} using model BoTorch.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:35:03] ax.service.ax_client: Completed trial 19 with data: {'hartmann6': (-1.56399, 0.0), 'l2norm': (1.173711, 0.0)}.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:35:08] ax.service.ax_client: Generated new trial 20 with parameters {'x1': 0.401077, 'x2': 0.389952, 'x3': 0.719855, 'x4': 0.352102, 'x5': 0.207668, 'x6': 0.597962} using model BoTorch.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:35:08] ax.service.ax_client: Completed trial 20 with data: {'hartmann6': (-1.592274, 0.0), 'l2norm': (1.164378, 0.0)}.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:35:17] ax.service.ax_client: Generated new trial 21 with parameters {'x1': 0.44262, 'x2': 0.378485, 'x3': 0.729044, 'x4': 0.317232, 'x5': 0.199577, 'x6': 0.569486} using model BoTorch.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:35:17] ax.service.ax_client: Completed trial 21 with data: {'hartmann6': (-1.45421, 0.0), 'l2norm': (1.155617, 0.0)}.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:35:23] ax.service.ax_client: Generated new trial 22 with parameters {'x1': 0.372955, 'x2': 0.391623, 'x3': 0.768302, 'x4': 0.37871, 'x5': 0.244363, 'x6': 0.601324} using model BoTorch.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:35:23] ax.service.ax_client: Completed trial 22 with data: {'hartmann6': (-1.616371, 0.0), 'l2norm': (1.203112, 0.0)}.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:35:29] ax.service.ax_client: Generated new trial 23 with parameters {'x1': 0.351682, 'x2': 0.407125, 'x3': 0.737527, 'x4': 0.338623, 'x5': 0.257507, 'x6': 0.623074} using model BoTorch.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:35:29] ax.service.ax_client: Completed trial 23 with data: {'hartmann6': (-1.865098, 0.0), 'l2norm': (1.184303, 0.0)}.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:35:35] ax.service.ax_client: Generated new trial 24 with parameters {'x1': 0.305094, 'x2': 0.423341, 'x3': 0.722495, 'x4': 0.301513, 'x5': 0.279932, 'x6': 0.666682} using model BoTorch.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:35:35] ax.service.ax_client: Completed trial 24 with data: {'hartmann6': (-2.08928, 0.0), 'l2norm': (1.186607, 0.0)}.\n" ] } ], "source": [ "for i in range(25):\n", " parameterization, trial_index = ax_client.get_next_trial()\n", " # Local evaluation here can be replaced with deployment to external system.\n", " ax_client.complete_trial(trial_index=trial_index, raw_data=evaluate(parameterization))" ] }, { "cell_type": "markdown", "id": "260a661c", "metadata": { "papermill": { "duration": 0.054381, "end_time": "2024-03-01T16:35:35.939914", "exception": false, "start_time": "2024-03-01T16:35:35.885533", "status": "completed" }, "tags": [] }, "source": [ "### How many trials can run in parallel?\n", "By default, Ax restricts number of trials that can run in parallel for some optimization stages, in order to improve the optimization performance and reduce the number of trials that the optimization will require. To check the maximum parallelism for each optimization stage:" ] }, { "cell_type": "code", "execution_count": 6, "id": "b52a0002", "metadata": { "execution": { "iopub.execute_input": "2024-03-01T16:35:36.040976Z", "iopub.status.busy": "2024-03-01T16:35:36.040326Z", "iopub.status.idle": "2024-03-01T16:35:36.047618Z", "shell.execute_reply": "2024-03-01T16:35:36.046906Z" }, "papermill": { "duration": 0.059475, "end_time": "2024-03-01T16:35:36.049018", "exception": false, "start_time": "2024-03-01T16:35:35.989543", "status": "completed" }, "tags": [] }, "outputs": [ { "data": { "text/plain": [ "[(12, 12), (-1, 3)]" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ax_client.get_max_parallelism()" ] }, { "cell_type": "markdown", "id": "aaa2aff2", "metadata": { "papermill": { "duration": 0.047396, "end_time": "2024-03-01T16:35:36.145159", "exception": false, "start_time": "2024-03-01T16:35:36.097763", "status": "completed" }, "tags": [] }, "source": [ "The output of this function is a list of tuples of form (number of trials, max parallelism), so the example above means \"the max parallelism is 12 for the first 12 trials and 3 for all subsequent trials.\" This is because the first 12 trials are produced quasi-randomly and can all be evaluated at once, and subsequent trials are produced via Bayesian optimization, which converges on optimal point in fewer trials when parallelism is limited. `MaxParallelismReachedException` indicates that the parallelism limit has been reached –– refer to the 'Service API Exceptions Meaning and Handling' section at the end of the tutorial for handling." ] }, { "cell_type": "markdown", "id": "886754fc", "metadata": { "papermill": { "duration": 0.046769, "end_time": "2024-03-01T16:35:36.239160", "exception": false, "start_time": "2024-03-01T16:35:36.192391", "status": "completed" }, "tags": [] }, "source": [ "### How to view all existing trials during optimization?" ] }, { "cell_type": "code", "execution_count": 7, "id": "84d624df", "metadata": { "execution": { "iopub.execute_input": "2024-03-01T16:35:36.335192Z", "iopub.status.busy": "2024-03-01T16:35:36.334537Z", "iopub.status.idle": "2024-03-01T16:35:36.355678Z", "shell.execute_reply": "2024-03-01T16:35:36.355104Z" }, "papermill": { "duration": 0.070764, "end_time": "2024-03-01T16:35:36.357091", "exception": false, "start_time": "2024-03-01T16:35:36.286327", "status": "completed" }, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[INFO 03-01 16:35:36] ax.modelbridge.generation_strategy: Note that parameter values in dataframe are rounded to 2 decimal points; the values in the dataframe are thus not the exact ones suggested by Ax in trials.\n" ] }, { "data": { "text/html": [ "
\n", " | Generation Step | \n", "Generation Model | \n", "Trial Index | \n", "Trial Status | \n", "Arm Parameterizations | \n", "
---|---|---|---|---|---|
0 | \n", "GenerationStep_0 | \n", "Sobol | \n", "0 | \n", "COMPLETED | \n", "{'0_0': {'x1': 0.98, 'x2': 0.78, 'x3': 0.3, 'x... | \n", "
1 | \n", "GenerationStep_0 | \n", "Sobol | \n", "1 | \n", "COMPLETED | \n", "{'1_0': {'x1': 0.25, 'x2': 0.57, 'x3': 0.53, '... | \n", "
2 | \n", "GenerationStep_0 | \n", "Sobol | \n", "2 | \n", "COMPLETED | \n", "{'2_0': {'x1': 0.0, 'x2': 0.27, 'x3': 0.35, 'x... | \n", "
3 | \n", "GenerationStep_0 | \n", "Sobol | \n", "3 | \n", "COMPLETED | \n", "{'3_0': {'x1': 0.03, 'x2': 0.39, 'x3': 0.6, 'x... | \n", "
4 | \n", "GenerationStep_0 | \n", "Sobol | \n", "4 | \n", "COMPLETED | \n", "{'4_0': {'x1': 0.57, 'x2': 0.22, 'x3': 0.02, '... | \n", "
5 | \n", "GenerationStep_0 | \n", "Sobol | \n", "5 | \n", "COMPLETED | \n", "{'5_0': {'x1': 0.91, 'x2': 0.41, 'x3': 0.96, '... | \n", "
6 | \n", "GenerationStep_0 | \n", "Sobol | \n", "6 | \n", "COMPLETED | \n", "{'6_0': {'x1': 0.47, 'x2': 0.81, 'x3': 0.89, '... | \n", "
7 | \n", "GenerationStep_0 | \n", "Sobol | \n", "7 | \n", "COMPLETED | \n", "{'7_0': {'x1': 0.95, 'x2': 0.76, 'x3': 0.88, '... | \n", "
8 | \n", "GenerationStep_0 | \n", "Sobol | \n", "8 | \n", "COMPLETED | \n", "{'8_0': {'x1': 0.07, 'x2': 0.83, 'x3': 0.21, '... | \n", "
9 | \n", "GenerationStep_0 | \n", "Sobol | \n", "9 | \n", "COMPLETED | \n", "{'9_0': {'x1': 0.54, 'x2': 0.99, 'x3': 0.94, '... | \n", "
10 | \n", "GenerationStep_0 | \n", "Sobol | \n", "10 | \n", "COMPLETED | \n", "{'10_0': {'x1': 0.59, 'x2': 0.37, 'x3': 0.93, ... | \n", "
11 | \n", "GenerationStep_0 | \n", "Sobol | \n", "11 | \n", "COMPLETED | \n", "{'11_0': {'x1': 0.69, 'x2': 0.03, 'x3': 0.22, ... | \n", "
12 | \n", "GenerationStep_1 | \n", "BoTorch | \n", "12 | \n", "COMPLETED | \n", "{'12_0': {'x1': 0.18, 'x2': 0.67, 'x3': 0.42, ... | \n", "
13 | \n", "GenerationStep_1 | \n", "BoTorch | \n", "13 | \n", "COMPLETED | \n", "{'13_0': {'x1': 0.23, 'x2': 0.62, 'x3': 0.48, ... | \n", "
14 | \n", "GenerationStep_1 | \n", "BoTorch | \n", "14 | \n", "COMPLETED | \n", "{'14_0': {'x1': 0.29, 'x2': 0.55, 'x3': 0.57, ... | \n", "
15 | \n", "GenerationStep_1 | \n", "BoTorch | \n", "15 | \n", "COMPLETED | \n", "{'15_0': {'x1': 0.34, 'x2': 0.5, 'x3': 0.65, '... | \n", "
16 | \n", "GenerationStep_1 | \n", "BoTorch | \n", "16 | \n", "COMPLETED | \n", "{'16_0': {'x1': 0.37, 'x2': 0.45, 'x3': 0.72, ... | \n", "
17 | \n", "GenerationStep_1 | \n", "BoTorch | \n", "17 | \n", "COMPLETED | \n", "{'17_0': {'x1': 0.37, 'x2': 0.4, 'x3': 0.78, '... | \n", "
18 | \n", "GenerationStep_1 | \n", "BoTorch | \n", "18 | \n", "COMPLETED | \n", "{'18_0': {'x1': 0.34, 'x2': 0.34, 'x3': 0.82, ... | \n", "
19 | \n", "GenerationStep_1 | \n", "BoTorch | \n", "19 | \n", "COMPLETED | \n", "{'19_0': {'x1': 0.4, 'x2': 0.39, 'x3': 0.76, '... | \n", "
20 | \n", "GenerationStep_1 | \n", "BoTorch | \n", "20 | \n", "COMPLETED | \n", "{'20_0': {'x1': 0.4, 'x2': 0.39, 'x3': 0.72, '... | \n", "
21 | \n", "GenerationStep_1 | \n", "BoTorch | \n", "21 | \n", "COMPLETED | \n", "{'21_0': {'x1': 0.44, 'x2': 0.38, 'x3': 0.73, ... | \n", "
22 | \n", "GenerationStep_1 | \n", "BoTorch | \n", "22 | \n", "COMPLETED | \n", "{'22_0': {'x1': 0.37, 'x2': 0.39, 'x3': 0.77, ... | \n", "
23 | \n", "GenerationStep_1 | \n", "BoTorch | \n", "23 | \n", "COMPLETED | \n", "{'23_0': {'x1': 0.35, 'x2': 0.41, 'x3': 0.74, ... | \n", "
24 | \n", "GenerationStep_1 | \n", "BoTorch | \n", "24 | \n", "COMPLETED | \n", "{'24_0': {'x1': 0.31, 'x2': 0.42, 'x3': 0.72, ... | \n", "