{ "cells": [ { "cell_type": "markdown", "metadata": { "customInput": null, "originalKey": "06e172a0-2da3-4c90-93c2-be01bf4f6d45", "showInput": false }, "source": [ "This tutorial illustrates use of a Global Stopping Strategy (GSS) in combination with the Service API. For background on the Service API, see the Service API Tutorial: https://ax.dev/tutorials/gpei_hartmann_service.html GSS is also supported in the Scheduler API, where it can be provided as part of `SchedulerOptions`. For more on `Scheduler`, see the Scheduler tutorial: https://ax.dev/tutorials/scheduler.html\n", "\n", "Global Stopping stops an optimization loop when some data-based criteria are met which suggest that future trials will not be very helpful. For example, we might stop when there has been very little improvement in the last five trials. This is as opposed to trial-level early stopping, which monitors the results of expensive evaluations and terminates those that are unlikely to produce promising results, freeing resources to explore more promising configurations. For more on trial-level early stopping, see the tutorial: https://ax.dev/tutorials/early_stopping/early_stopping.html" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false, "customOutput": null, "execution": { "iopub.execute_input": "2023-08-11T20:00:40.020318Z", "iopub.status.busy": "2023-08-11T20:00:40.020059Z", "iopub.status.idle": "2023-08-11T20:00:58.292366Z", "shell.execute_reply": "2023-08-11T20:00:58.291609Z" }, "executionStartTime": 1683829335587, "executionStopTime": 1683829339370, "originalKey": "00a04d2c-d990-41c1-9eef-bbb05fba000d", "requestMsgId": "1c560539-1c7d-4c7a-ae55-e87c3b601859" }, "outputs": [ { "data": { "text/html": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 08-11 16:00:57] ax.utils.notebook.plotting: Injecting Plotly library into cell. Do not overwrite or delete cell.\n" ] }, { "data": { "text/html": [ " \n", " " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import numpy as np\n", "\n", "from ax.service.ax_client import AxClient, ObjectiveProperties\n", "from ax.utils.measurement.synthetic_functions import Branin, branin\n", "from ax.utils.notebook.plotting import render, init_notebook_plotting\n", "\n", "init_notebook_plotting()" ] }, { "cell_type": "markdown", "metadata": { "customInput": null, "originalKey": "8688d729-b402-4a4c-b796-94fdcf5e022c", "showInput": false }, "source": [ "# 1. What happens without global stopping? Optimization can run for too long.\n", "This example uses the Branin test problem. We run 25 trials, which turns out to be far more than needed, because we get close to the optimum quite quickly." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false, "customInput": null, "customOutput": null, "execution": { "iopub.execute_input": "2023-08-11T20:00:58.328741Z", "iopub.status.busy": "2023-08-11T20:00:58.328243Z", "iopub.status.idle": "2023-08-11T20:00:58.332265Z", "shell.execute_reply": "2023-08-11T20:00:58.331615Z" }, "executionStartTime": 1683829339516, "executionStopTime": 1683829339531, "originalKey": "320a952b-9e78-43e1-a55b-76a355e90f83", "requestMsgId": "14e3a517-c7d0-4300-92d9-57ceb5afca34", "showInput": true }, "outputs": [], "source": [ "def evaluate(parameters):\n", " x = np.array([parameters.get(f\"x{i+1}\") for i in range(2)])\n", " return {\"branin\": (branin(x), 0.0)}" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false, "customInput": null, "customOutput": null, "execution": { "iopub.execute_input": "2023-08-11T20:00:58.335131Z", "iopub.status.busy": "2023-08-11T20:00:58.334889Z", "iopub.status.idle": "2023-08-11T20:00:58.338800Z", "shell.execute_reply": "2023-08-11T20:00:58.338074Z" }, "executionStartTime": 1683829339659, "executionStopTime": 1683829339668, "originalKey": "5740fbc2-97d6-465b-b01c-61e6c34c0220", "requestMsgId": "ff819cc9-ff17-4763-a857-83662b01e955", "showInput": true }, "outputs": [], "source": [ "params = [\n", " {\n", " \"name\": f\"x{i + 1}\",\n", " \"type\": \"range\",\n", " \"bounds\": [*Branin._domain[i]],\n", " \"value_type\": \"float\",\n", " \"log_scale\": False,\n", " }\n", "\n", " for i in range(2)\n", "]" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": false, "customInput": null, "customOutput": null, "execution": { "iopub.execute_input": "2023-08-11T20:00:58.341952Z", "iopub.status.busy": "2023-08-11T20:00:58.341687Z", "iopub.status.idle": "2023-08-11T20:00:58.350535Z", "shell.execute_reply": "2023-08-11T20:00:58.349832Z" }, "executionStartTime": 1683829339782, "executionStopTime": 1683829339834, "originalKey": "65667172-14df-437b-bdd0-5a59580e4054", "requestMsgId": "e0bc2847-17a5-43d7-bf49-ed97c90f1d50", "showInput": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[WARNING 08-11 16:00:58] ax.service.ax_client: Random seed set to 0. Note that this setting only affects the Sobol quasi-random generator and BoTorch-powered Bayesian optimization models. For the latter models, setting random seed to the same number for two optimizations will make the generated trials similar, but not exactly the same, and over time the trials will diverge more.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 08-11 16:00:58] ax.service.utils.instantiation: Created search space: SearchSpace(parameters=[RangeParameter(name='x1', parameter_type=FLOAT, range=[-5.0, 10.0]), RangeParameter(name='x2', parameter_type=FLOAT, range=[0.0, 15.0])], parameter_constraints=[]).\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 08-11 16:00:58] ax.core.experiment: The is_test flag has been set to True. This flag is meant purely for development and integration testing purposes. If you are running a live experiment, please set this flag to False\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 08-11 16:00:58] ax.modelbridge.dispatch_utils: Using Models.GPEI since there are more ordered parameters than there are categories for the unordered categorical parameters.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 08-11 16:00:58] ax.modelbridge.dispatch_utils: Calculating the number of remaining initialization trials based on num_initialization_trials=None max_initialization_trials=None num_tunable_parameters=2 num_trials=None use_batch_trials=False\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 08-11 16:00:58] ax.modelbridge.dispatch_utils: calculated num_initialization_trials=5\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 08-11 16:00:58] ax.modelbridge.dispatch_utils: num_completed_initialization_trials=0 num_remaining_initialization_trials=5\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 08-11 16:00:58] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+GPEI', steps=[Sobol for 5 trials, GPEI for subsequent trials]). Iterations after 5 will take longer to generate due to model-fitting.\n" ] } ], "source": [ "ax_client = AxClient(random_seed=0, verbose_logging=False)\n", "\n", "ax_client.create_experiment(\n", " name=\"branin_test_experiment\",\n", " parameters=params,\n", " objectives={\"branin\": ObjectiveProperties(minimize=True)},\n", " is_test=True,\n", ")" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false, "customInput": null, "customOutput": null, "execution": { "iopub.execute_input": "2023-08-11T20:00:58.354960Z", "iopub.status.busy": "2023-08-11T20:00:58.354668Z", "iopub.status.idle": "2023-08-11T20:01:08.631092Z", "shell.execute_reply": "2023-08-11T20:01:08.630347Z" }, "executionStartTime": 1683829339928, "executionStopTime": 1683829356006, "originalKey": "1f208de3-5189-4847-a779-940795977845", "requestMsgId": "95f327f2-327f-4284-93ae-3053c9b6ec45", "showInput": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 1min 15s, sys: 2.44 s, total: 1min 17s\n", "Wall time: 10.3 s\n" ] } ], "source": [ "%%time\n", "for i in range(25):\n", " parameters, trial_index = ax_client.get_next_trial()\n", " # Local evaluation here can be replaced with deployment to external system.\n", " ax_client.complete_trial(\n", " trial_index=trial_index, raw_data=evaluate(parameters)\n", " )" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": false, "customInput": null, "customOutput": null, "execution": { "iopub.execute_input": "2023-08-11T20:01:08.634217Z", "iopub.status.busy": "2023-08-11T20:01:08.633940Z", "iopub.status.idle": "2023-08-11T20:01:12.148366Z", "shell.execute_reply": "2023-08-11T20:01:12.147600Z" }, "executionStartTime": 1683829356136, "executionStopTime": 1683829356616, "originalKey": "a369aafa-8ee4-4c02-bea6-673271da81ab", "requestMsgId": "b601e1e9-fd2d-4faf-a369-04e5c4a9f8cb", "showInput": true }, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "linkText": "Export to plot.ly", "plotlyServerURL": "https://plot.ly", "showLink": false }, "data": [ { "hoverinfo": "none", "legendgroup": "", "line": { "width": 0 }, "mode": "lines", "showlegend": false, "type": "scatter", "x": [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25 ], "y": [ 37.28956500488602, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 2.890008559727825, 0.5941375435392295, 0.5941375435392295, 0.5941375435392295, 0.41590737103680553, 0.41590737103680553, 0.41590737103680553, 0.41590737103680553, 0.41590737103680553, 0.41590737103680553, 0.41590737103680553, 0.41590737103680553, 0.40552541774520456, 0.40552541774520456, 0.4050805087279503, 0.4050805087279503, 0.4050805087279503, 0.4050805087279503 ] }, { "fill": "tonexty", "fillcolor": "rgba(128,177,211,0.3)", "legendgroup": "objective value", "line": { "color": "rgba(128,177,211,1)" }, "mode": "lines", "name": "objective value", "text": [ "
Parameterization:
x1: 2.126607894897461
x2: 8.887859880924225", "
Parameterization:
x1: 3.681450095027685
x2: 0.5568291060626507", "
Parameterization:
x1: 9.260048242285848
x2: 12.935160705819726", "
Parameterization:
x1: -3.1931319646537304
x2: 3.921633088029921", "
Parameterization:
x1: -1.7758215544745326
x2: 14.612732883542776", "
Parameterization:
x1: 6.234655721146142
x2: 0.0", "
Parameterization:
x1: 3.8319331249178656
x2: 3.0609605448388035", "
Parameterization:
x1: 2.505878627244072
x2: 2.037908980024512", "
Parameterization:
x1: 3.2758148572241463
x2: 1.8411622630788205", "
Parameterization:
x1: -5.0
x2: 15.0", "
Parameterization:
x1: 2.198632547916451
x2: 0.0", "
Parameterization:
x1: 3.12888694779258
x2: 2.4162496042939194", "
Parameterization:
x1: 10.0
x2: 0.0", "
Parameterization:
x1: 10.0
x2: 3.453846597648022", "
Parameterization:
x1: 8.960413745514993
x2: 2.9003398235061697", "
Parameterization:
x1: 9.91312866882485
x2: 2.313273386287401", "
Parameterization:
x1: 7.805301801882997
x2: 4.615653559363886", "
Parameterization:
x1: 9.47741227386467
x2: 2.819590138989362", "
Parameterization:
x1: 9.263254876901119
x2: 2.1124639005542107", "
Parameterization:
x1: 3.126433256297437
x2: 2.206014000929609", "
Parameterization:
x1: -5.0
x2: 10.613451027654788", "
Parameterization:
x1: 9.388830159649919
x2: 2.4763033999482427", "
Parameterization:
x1: 3.2455042652932526
x2: 2.2397880007459046", "
Parameterization:
x1: 4.419291296153043
x2: 15.0", "
Parameterization:
x1: 1.7797616340591453
x2: 4.3455155203533655" ], "type": "scatter", "x": [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25 ], "y": [ 37.28956500488602, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 2.890008559727825, 0.5941375435392295, 0.5941375435392295, 0.5941375435392295, 0.41590737103680553, 0.41590737103680553, 0.41590737103680553, 0.41590737103680553, 0.41590737103680553, 0.41590737103680553, 0.41590737103680553, 0.41590737103680553, 0.40552541774520456, 0.40552541774520456, 0.4050805087279503, 0.4050805087279503, 0.4050805087279503, 0.4050805087279503 ] }, { "fill": "tonexty", "fillcolor": "rgba(128,177,211,0.3)", "hoverinfo": "none", "legendgroup": "", "line": { "width": 0 }, "mode": "lines", "showlegend": false, "type": "scatter", "x": [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25 ], "y": [ 37.28956500488602, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 2.890008559727825, 0.5941375435392295, 0.5941375435392295, 0.5941375435392295, 0.41590737103680553, 0.41590737103680553, 0.41590737103680553, 0.41590737103680553, 0.41590737103680553, 0.41590737103680553, 0.41590737103680553, 0.41590737103680553, 0.40552541774520456, 0.40552541774520456, 0.4050805087279503, 0.4050805087279503, 0.4050805087279503, 0.4050805087279503 ] }, { "line": { "color": "rgba(141,211,199,1)", "dash": "dash" }, "mode": "lines", "name": "model change", "type": "scatter", "x": [ 5, 5 ], "y": [ 0.4050805087279503, 37.28956500488602 ] } ], "layout": { "showlegend": true, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "fillpattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "sequentialminus": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "title": { "text": "Model performance vs. # of iterations" }, "xaxis": { "title": { "text": "Iteration" } }, "yaxis": { "title": { "text": "Branin" } } } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "render(ax_client.get_optimization_trace())" ] }, { "cell_type": "markdown", "metadata": { "customInput": null, "originalKey": "ca391462-4695-44f1-bc53-070a947c5648", "showInput": false }, "source": [ "# 2. Optimization with global stopping, with the Service API" ] }, { "cell_type": "markdown", "metadata": { "customInput": null, "originalKey": "5a2690ef-0990-4cbd-9bc9-529b1455a4c3", "showInput": false }, "source": [ "Rather than running a fixed number of trials, we can use a GlobalStoppingStrategy (GSS), which checks whether some stopping criteria have been met when `get_next_trial` is called. Here, we use an `ImprovementGlobalStoppingStrategy`, which checks whether the the last `window_size` trials have improved by more than some threshold amount.\n", "\n", "For single-objective optimization, which we are doing here, `ImprovementGlobalStoppingStrategy` checks if an improvement is \"significant\" by comparing it to the inter-quartile range (IQR) of the objective values attained so far. \n", "\n", "`ImprovementGlobalStoppingStrategy` also supports multi-objective optimization (MOO), in which case it checks whether the percentage improvement in hypervolume over the last `window_size` trials exceeds `improvement_bar`." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": false, "customInput": null, "customOutput": null, "execution": { "iopub.execute_input": "2023-08-11T20:01:12.152367Z", "iopub.status.busy": "2023-08-11T20:01:12.152059Z", "iopub.status.idle": "2023-08-11T20:01:12.155808Z", "shell.execute_reply": "2023-08-11T20:01:12.155099Z" }, "executionStartTime": 1683829356716, "executionStopTime": 1683829356725, "originalKey": "a6634232-448a-4b84-98cd-399c755537df", "requestMsgId": "7e428336-eeeb-4e5b-91c4-fcf5a671773d", "showInput": true }, "outputs": [], "source": [ "from ax.global_stopping.strategies.improvement import ImprovementGlobalStoppingStrategy\n", "from ax.exceptions.core import OptimizationShouldStop" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "collapsed": false, "customInput": null, "customOutput": null, "execution": { "iopub.execute_input": "2023-08-11T20:01:12.159192Z", "iopub.status.busy": "2023-08-11T20:01:12.158905Z", "iopub.status.idle": "2023-08-11T20:01:12.162413Z", "shell.execute_reply": "2023-08-11T20:01:12.161762Z" }, "executionStartTime": 1683829356822, "executionStopTime": 1683829356829, "originalKey": "c313de63-03ee-4a65-aa5c-5e7b6f436480", "requestMsgId": "953b064b-8db6-430f-909d-872469bc1e16", "showInput": true }, "outputs": [], "source": [ "# Start considering stopping only after the 5 initialization trials + 5 GPEI trials.\n", "# Stop if the improvement in the best point in the past 5 trials is less than\n", "# 1% of the IQR thus far.\n", "stopping_strategy = ImprovementGlobalStoppingStrategy(\n", " min_trials=5 + 5, window_size=5, improvement_bar=0.01\n", ")" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "collapsed": false, "customInput": null, "customOutput": null, "execution": { "iopub.execute_input": "2023-08-11T20:01:12.166522Z", "iopub.status.busy": "2023-08-11T20:01:12.166235Z", "iopub.status.idle": "2023-08-11T20:01:12.177031Z", "shell.execute_reply": "2023-08-11T20:01:12.176432Z" }, "executionStartTime": 1683829356961, "executionStopTime": 1683829356997, "originalKey": "a2c6c699-f0d2-4001-9bee-3964594e435c", "requestMsgId": "2ba6f82b-1443-4274-83d1-03c56f0190d0", "showInput": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[WARNING 08-11 16:01:12] ax.service.ax_client: Random seed set to 0. Note that this setting only affects the Sobol quasi-random generator and BoTorch-powered Bayesian optimization models. For the latter models, setting random seed to the same number for two optimizations will make the generated trials similar, but not exactly the same, and over time the trials will diverge more.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 08-11 16:01:12] ax.service.utils.instantiation: Created search space: SearchSpace(parameters=[RangeParameter(name='x1', parameter_type=FLOAT, range=[-5.0, 10.0]), RangeParameter(name='x2', parameter_type=FLOAT, range=[0.0, 15.0])], parameter_constraints=[]).\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 08-11 16:01:12] ax.core.experiment: The is_test flag has been set to True. This flag is meant purely for development and integration testing purposes. If you are running a live experiment, please set this flag to False\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 08-11 16:01:12] ax.modelbridge.dispatch_utils: Using Models.GPEI since there are more ordered parameters than there are categories for the unordered categorical parameters.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 08-11 16:01:12] ax.modelbridge.dispatch_utils: Calculating the number of remaining initialization trials based on num_initialization_trials=None max_initialization_trials=None num_tunable_parameters=2 num_trials=None use_batch_trials=False\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 08-11 16:01:12] ax.modelbridge.dispatch_utils: calculated num_initialization_trials=5\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 08-11 16:01:12] ax.modelbridge.dispatch_utils: num_completed_initialization_trials=0 num_remaining_initialization_trials=5\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 08-11 16:01:12] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+GPEI', steps=[Sobol for 5 trials, GPEI for subsequent trials]). Iterations after 5 will take longer to generate due to model-fitting.\n" ] } ], "source": [ "ax_client_gss = AxClient(\n", " global_stopping_strategy=stopping_strategy, random_seed=0, verbose_logging=False\n", ")\n", "\n", "ax_client_gss.create_experiment(\n", " name=\"branin_test_experiment\",\n", " parameters=params,\n", " objectives={\"branin\": ObjectiveProperties(minimize=True)},\n", " is_test=True,\n", ")" ] }, { "cell_type": "markdown", "metadata": { "customInput": null, "originalKey": "7ff170a1-e885-429f-9695-8b64b5b8e209", "showInput": false }, "source": [ "If there has not been much improvement, `ImprovementGlobalStoppingStrategy` will raise an exception. If the exception is raised, we catch it and terminate optimization." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": false, "customInput": null, "customOutput": null, "execution": { "iopub.execute_input": "2023-08-11T20:01:12.180882Z", "iopub.status.busy": "2023-08-11T20:01:12.180651Z", "iopub.status.idle": "2023-08-11T20:01:16.210803Z", "shell.execute_reply": "2023-08-11T20:01:16.209974Z" }, "executionStartTime": 1683829357114, "executionStopTime": 1683829363866, "originalKey": "3db097cb-1e6e-4320-806a-981dcef6bade", "requestMsgId": "fd039109-2a23-4287-8935-b74274405e56", "showInput": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The improvement in best objective in the past 5 trials (=0.005) is less than 0.01 times the interquartile range (IQR) of objectives attained so far (IQR=33.434).\n" ] } ], "source": [ "for i in range(25):\n", " try:\n", " parameters, trial_index = ax_client_gss.get_next_trial()\n", " except OptimizationShouldStop as exc:\n", " print(exc.message)\n", " break\n", " ax_client_gss.complete_trial(trial_index=trial_index, raw_data=evaluate(parameters))" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "collapsed": false, "customInput": null, "customOutput": null, "execution": { "iopub.execute_input": "2023-08-11T20:01:16.214684Z", "iopub.status.busy": "2023-08-11T20:01:16.214317Z", "iopub.status.idle": "2023-08-11T20:01:16.266301Z", "shell.execute_reply": "2023-08-11T20:01:16.265475Z" }, "executionStartTime": 1683829363988, "executionStopTime": 1683829364103, "originalKey": "ffb53ed2-8775-492d-a357-348957637454", "requestMsgId": "f0f765dd-85db-4519-90d0-064a1bf64b6d", "showInput": true }, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "linkText": "Export to plot.ly", "plotlyServerURL": "https://plot.ly", "showLink": false }, "data": [ { "hoverinfo": "none", "legendgroup": "", "line": { "width": 0 }, "mode": "lines", "showlegend": false, "type": "scatter", "x": [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13 ], "y": [ 37.28956500488602, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 2.890008559727825, 0.5941375435392295, 0.5941375435392295, 0.5941375435392295, 0.41590737103680553, 0.41590737103680553 ] }, { "fill": "tonexty", "fillcolor": "rgba(128,177,211,0.3)", "legendgroup": "objective value", "line": { "color": "rgba(128,177,211,1)" }, "mode": "lines", "name": "objective value", "text": [ "
Parameterization:
x1: 2.126607894897461
x2: 8.887859880924225", "
Parameterization:
x1: 3.681450095027685
x2: 0.5568291060626507", "
Parameterization:
x1: 9.260048242285848
x2: 12.935160705819726", "
Parameterization:
x1: -3.1931319646537304
x2: 3.921633088029921", "
Parameterization:
x1: -1.7758215544745326
x2: 14.612732883542776", "
Parameterization:
x1: 6.234655721146142
x2: 0.0", "
Parameterization:
x1: 3.8319331249178656
x2: 3.0609605448388035", "
Parameterization:
x1: 2.505878627244072
x2: 2.037908980024512", "
Parameterization:
x1: 3.2758148572241463
x2: 1.8411622630788205", "
Parameterization:
x1: -5.0
x2: 15.0", "
Parameterization:
x1: 2.198632547916451
x2: 0.0", "
Parameterization:
x1: 3.12888694779258
x2: 2.4162496042939194", "
Parameterization:
x1: 10.0
x2: 0.0" ], "type": "scatter", "x": [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13 ], "y": [ 37.28956500488602, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 2.890008559727825, 0.5941375435392295, 0.5941375435392295, 0.5941375435392295, 0.41590737103680553, 0.41590737103680553 ] }, { "fill": "tonexty", "fillcolor": "rgba(128,177,211,0.3)", "hoverinfo": "none", "legendgroup": "", "line": { "width": 0 }, "mode": "lines", "showlegend": false, "type": "scatter", "x": [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13 ], "y": [ 37.28956500488602, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 2.890008559727825, 0.5941375435392295, 0.5941375435392295, 0.5941375435392295, 0.41590737103680553, 0.41590737103680553 ] }, { "line": { "color": "rgba(141,211,199,1)", "dash": "dash" }, "mode": "lines", "name": "model change", "type": "scatter", "x": [ 5, 5 ], "y": [ 0.41590737103680553, 37.28956500488602 ] } ], "layout": { "showlegend": true, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "fillpattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "sequentialminus": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "title": { "text": "Model performance vs. # of iterations" }, "xaxis": { "title": { "text": "Iteration" } }, "yaxis": { "title": { "text": "Branin" } } } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "render(ax_client_gss.get_optimization_trace())" ] }, { "cell_type": "markdown", "metadata": { "customInput": null, "originalKey": "b01707f3-0bbf-4003-9222-29ba5e3c77b2", "showInput": false }, "source": [ "# 3. Write your own custom Global Stopping Strategy" ] }, { "cell_type": "markdown", "metadata": { "customInput": null, "originalKey": "23b8372b-0067-4934-b599-210b994e06f1", "showInput": false }, "source": [ "You can write a custom Global Stopping Strategy by subclassing `BaseGlobalStoppingStrategy` and use it where `ImprovementGlobalStoppingStrategy` was used above." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "collapsed": false, "customInput": null, "customOutput": null, "execution": { "iopub.execute_input": "2023-08-11T20:01:16.270494Z", "iopub.status.busy": "2023-08-11T20:01:16.270223Z", "iopub.status.idle": "2023-08-11T20:01:16.274431Z", "shell.execute_reply": "2023-08-11T20:01:16.273729Z" }, "executionStartTime": 1683829364214, "executionStopTime": 1683829364222, "originalKey": "2e5512a9-82ed-43a0-8616-6cee7f648b0f", "requestMsgId": "d5c268a1-fefe-49d5-8ff4-a2cb40fe278b", "showInput": true }, "outputs": [], "source": [ "from ax.global_stopping.strategies.base import BaseGlobalStoppingStrategy\n", "from typing import Tuple\n", "from ax.core.experiment import Experiment\n", "from ax.core.base_trial import TrialStatus\n", "from ax.global_stopping.strategies.improvement import constraint_satisfaction" ] }, { "cell_type": "markdown", "metadata": { "customInput": null, "originalKey": "584df5ac-c0f6-4c48-8cec-f9765a04e635", "showInput": false }, "source": [ "Here, we define `SimpleThresholdGlobalStoppingStrategy`, which stops when we observe a point better than a provided threshold. This can be useful when there is a known optimum. For example, the Branin function has an optimum of zero. When the optimum is not known, this can still be useful from a satisficing perspective: For example, maybe we need a model to take up less than a certain amount of RAM so it doesn't crash our usual hardware, but there is no benefit to further improvements." ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "collapsed": false, "customInput": null, "customOutput": null, "execution": { "iopub.execute_input": "2023-08-11T20:01:16.277731Z", "iopub.status.busy": "2023-08-11T20:01:16.277511Z", "iopub.status.idle": "2023-08-11T20:01:16.284172Z", "shell.execute_reply": "2023-08-11T20:01:16.283486Z" }, "executionStartTime": 1683829490325, "executionStopTime": 1683829490340, "originalKey": "bbd24d6e-a873-49d6-abe3-4d832acb8a60", "requestMsgId": "74b77cb7-54eb-4321-afae-942b62b90f5d", "showInput": true }, "outputs": [], "source": [ "class SimpleThresholdGlobalStoppingStrategy(BaseGlobalStoppingStrategy):\n", " \"\"\"\n", " A GSS that stops when we observe a point better than `threshold`.\n", " \"\"\"\n", " def __init__(\n", " self,\n", " min_trials: int,\n", " inactive_when_pending_trials: bool = True,\n", " threshold: float = 0.1\n", " ):\n", " self.threshold = threshold\n", " super().__init__(\n", " min_trials=min_trials,\n", " inactive_when_pending_trials=inactive_when_pending_trials\n", " )\n", " \n", " def _should_stop_optimization(\n", " self, experiment: Experiment\n", " ) -> Tuple[bool, str]:\n", " \"\"\"\n", " Check if the best seen is better than `self.threshold`.\n", " \"\"\"\n", " feasible_objectives = [\n", " trial.objective_mean\n", " for trial in experiment.trials_by_status[TrialStatus.COMPLETED]\n", " if constraint_satisfaction(trial)\n", " ]\n", "\n", " # Computing the interquartile for scaling the difference\n", " if len(feasible_objectives) <= 1:\n", " message = \"There are not enough feasible arms tried yet.\"\n", " return False, message\n", " \n", " minimize = experiment.optimization_config.objective.minimize\n", " if minimize:\n", " best = np.min(feasible_objectives)\n", " stop = best < self.threshold\n", " else:\n", " best = np.max(feasible_objectives)\n", " stop = best > self.threshold\n", "\n", " comparison = \"less\" if minimize else \"greater\"\n", " if stop:\n", " message = (\n", " f\"The best objective seen is {best:.3f}, which is {comparison} \"\n", " f\"than the threshold of {self.threshold:.3f}.\"\n", " )\n", " else:\n", " message = \"\"\n", "\n", " return stop, message" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "collapsed": false, "customInput": null, "customOutput": null, "execution": { "iopub.execute_input": "2023-08-11T20:01:16.287266Z", "iopub.status.busy": "2023-08-11T20:01:16.286988Z", "iopub.status.idle": "2023-08-11T20:01:16.290927Z", "shell.execute_reply": "2023-08-11T20:01:16.290100Z" }, "executionStartTime": 1683829491609, "executionStopTime": 1683829491626, "originalKey": "f3dc5682-0539-4c85-a66a-0d3128f0cc1c", "requestMsgId": "9ee9e413-be32-49fc-a7bc-8e1898d1dbf5", "showInput": true }, "outputs": [], "source": [ "stopping_strategy = SimpleThresholdGlobalStoppingStrategy(min_trials=5, threshold=1.)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "collapsed": false, "customInput": null, "customOutput": null, "execution": { "iopub.execute_input": "2023-08-11T20:01:16.294515Z", "iopub.status.busy": "2023-08-11T20:01:16.294214Z", "iopub.status.idle": "2023-08-11T20:01:16.304768Z", "shell.execute_reply": "2023-08-11T20:01:16.303938Z" }, "executionStartTime": 1683829491833, "executionStopTime": 1683829491894, "originalKey": "3d6c1ab2-c3ee-49c8-9969-45f2455bbd60", "requestMsgId": "08232010-46f8-4b28-b581-454ddacdc57b", "showInput": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[WARNING 08-11 16:01:16] ax.service.ax_client: Random seed set to 0. Note that this setting only affects the Sobol quasi-random generator and BoTorch-powered Bayesian optimization models. For the latter models, setting random seed to the same number for two optimizations will make the generated trials similar, but not exactly the same, and over time the trials will diverge more.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 08-11 16:01:16] ax.service.utils.instantiation: Created search space: SearchSpace(parameters=[RangeParameter(name='x1', parameter_type=FLOAT, range=[-5.0, 10.0]), RangeParameter(name='x2', parameter_type=FLOAT, range=[0.0, 15.0])], parameter_constraints=[]).\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 08-11 16:01:16] ax.core.experiment: The is_test flag has been set to True. This flag is meant purely for development and integration testing purposes. If you are running a live experiment, please set this flag to False\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 08-11 16:01:16] ax.modelbridge.dispatch_utils: Using Models.GPEI since there are more ordered parameters than there are categories for the unordered categorical parameters.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 08-11 16:01:16] ax.modelbridge.dispatch_utils: Calculating the number of remaining initialization trials based on num_initialization_trials=None max_initialization_trials=None num_tunable_parameters=2 num_trials=None use_batch_trials=False\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 08-11 16:01:16] ax.modelbridge.dispatch_utils: calculated num_initialization_trials=5\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 08-11 16:01:16] ax.modelbridge.dispatch_utils: num_completed_initialization_trials=0 num_remaining_initialization_trials=5\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 08-11 16:01:16] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+GPEI', steps=[Sobol for 5 trials, GPEI for subsequent trials]). Iterations after 5 will take longer to generate due to model-fitting.\n" ] } ], "source": [ "ax_client_custom_gss = AxClient(\n", " global_stopping_strategy=stopping_strategy,\n", " random_seed=0,\n", " verbose_logging=False,\n", ")\n", "\n", "ax_client_custom_gss.create_experiment(\n", " name=\"branin_test_experiment\",\n", " parameters=params,\n", " objectives={\"branin\": ObjectiveProperties(minimize=True)},\n", " is_test=True,\n", ")" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "collapsed": false, "customInput": null, "customOutput": null, "execution": { "iopub.execute_input": "2023-08-11T20:01:16.309795Z", "iopub.status.busy": "2023-08-11T20:01:16.309527Z", "iopub.status.idle": "2023-08-11T20:01:18.426403Z", "shell.execute_reply": "2023-08-11T20:01:18.425640Z" }, "executionStartTime": 1683829492064, "executionStopTime": 1683829495338, "originalKey": "a306cb15-364f-4e91-b569-9067843a7578", "requestMsgId": "81121dac-3a2a-4dde-b866-44e448e73ad5", "showInput": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The best objective seen is 0.594, which is less than the threshold of 1.000.\n" ] } ], "source": [ "for i in range(25):\n", " try:\n", " parameters, trial_index = ax_client_custom_gss.get_next_trial()\n", " except OptimizationShouldStop as exc:\n", " print(exc.message)\n", " break\n", " ax_client_custom_gss.complete_trial(\n", " trial_index=trial_index, raw_data=evaluate(parameters)\n", " )" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "collapsed": false, "customInput": null, "customOutput": null, "execution": { "iopub.execute_input": "2023-08-11T20:01:18.429689Z", "iopub.status.busy": "2023-08-11T20:01:18.429425Z", "iopub.status.idle": "2023-08-11T20:01:18.480871Z", "shell.execute_reply": "2023-08-11T20:01:18.480148Z" }, "executionStartTime": 1683829495351, "executionStopTime": 1683829495740, "originalKey": "3cb59624-d9bb-4b7a-9f57-7cb968dce889", "requestMsgId": "4dd4ed93-07ab-4dd1-92a9-f003f405ccbc", "showInput": true }, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "linkText": "Export to plot.ly", "plotlyServerURL": "https://plot.ly", "showLink": false }, "data": [ { "hoverinfo": "none", "legendgroup": "", "line": { "width": 0 }, "mode": "lines", "showlegend": false, "type": "scatter", "x": [ 1, 2, 3, 4, 5, 6, 7, 8, 9 ], "y": [ 37.28956500488602, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 2.890008559727825, 0.5941375435392295 ] }, { "fill": "tonexty", "fillcolor": "rgba(128,177,211,0.3)", "legendgroup": "objective value", "line": { "color": "rgba(128,177,211,1)" }, "mode": "lines", "name": "objective value", "text": [ "
Parameterization:
x1: 2.126607894897461
x2: 8.887859880924225", "
Parameterization:
x1: 3.681450095027685
x2: 0.5568291060626507", "
Parameterization:
x1: 9.260048242285848
x2: 12.935160705819726", "
Parameterization:
x1: -3.1931319646537304
x2: 3.921633088029921", "
Parameterization:
x1: -1.7758215544745326
x2: 14.612732883542776", "
Parameterization:
x1: 6.234655721146142
x2: 0.0", "
Parameterization:
x1: 3.8319331249178656
x2: 3.0609605448388035", "
Parameterization:
x1: 2.505878627244072
x2: 2.037908980024512", "
Parameterization:
x1: 3.2758148572241463
x2: 1.8411622630788205" ], "type": "scatter", "x": [ 1, 2, 3, 4, 5, 6, 7, 8, 9 ], "y": [ 37.28956500488602, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 2.890008559727825, 0.5941375435392295 ] }, { "fill": "tonexty", "fillcolor": "rgba(128,177,211,0.3)", "hoverinfo": "none", "legendgroup": "", "line": { "width": 0 }, "mode": "lines", "showlegend": false, "type": "scatter", "x": [ 1, 2, 3, 4, 5, 6, 7, 8, 9 ], "y": [ 37.28956500488602, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 3.545194409651578, 2.890008559727825, 0.5941375435392295 ] }, { "line": { "color": "rgba(141,211,199,1)", "dash": "dash" }, "mode": "lines", "name": "model change", "type": "scatter", "x": [ 5, 5 ], "y": [ 0.5941375435392295, 37.28956500488602 ] } ], "layout": { "showlegend": true, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "fillpattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "sequentialminus": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "title": { "text": "Model performance vs. # of iterations" }, "xaxis": { "title": { "text": "Iteration" } }, "yaxis": { "title": { "text": "Branin" } } } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "render(ax_client_custom_gss.get_optimization_trace())" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "customInput": null, "originalKey": "5f4eaa42-a8cb-42b2-b8b4-b2fa53398270", "showInput": true }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "cinder_runtime": true, "display_name": "python3", "ipyflow_runtime": false, "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.17" } }, "nbformat": 4, "nbformat_minor": 2 }