{ "cells": [ { "cell_type": "markdown", "id": "16b7a578", "metadata": { "code_folding": [], "hidden_ranges": [], "originalKey": "08064d6a-453e-44d7-85dc-896d40b6303a", "papermill": { "duration": 0.004583, "end_time": "2024-07-23T19:35:36.744733", "exception": false, "start_time": "2024-07-23T19:35:36.740150", "status": "completed" }, "showInput": true, "tags": [] }, "source": [ "# Developer API Example on Hartmann6\n", "\n", "The Developer API is suitable when the user wants maximal customization of the optimization loop. This tutorial demonstrates optimization of a Hartmann6 function using the `Experiment` construct. In this example, trials will be evaluated synchronously." ] }, { "cell_type": "code", "execution_count": 1, "id": "f8f2e7e8", "metadata": { "code_folding": [], "execution": { "iopub.execute_input": "2024-07-23T19:35:36.754740Z", "iopub.status.busy": "2024-07-23T19:35:36.754184Z", "iopub.status.idle": "2024-07-23T19:35:39.959113Z", "shell.execute_reply": "2024-07-23T19:35:39.958391Z" }, "executionStartTime": 1646323252842, "executionStopTime": 1646323256492, "hidden_ranges": [], "originalKey": "7b98b243-30da-468b-82c7-7e22dbce6b57", "papermill": { "duration": 3.225234, "end_time": "2024-07-23T19:35:39.974171", "exception": false, "start_time": "2024-07-23T19:35:36.748937", "status": "completed" }, "requestMsgId": "7b98b243-30da-468b-82c7-7e22dbce6b57", "tags": [] }, "outputs": [ { "data": { "text/html": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 07-23 19:35:39] ax.utils.notebook.plotting: Injecting Plotly library into cell. Do not overwrite or delete cell.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 07-23 19:35:39] ax.utils.notebook.plotting: Please see\n", " (https://ax.dev/tutorials/visualizations.html#Fix-for-plots-that-are-not-rendering)\n", " if visualizations are not rendering.\n" ] }, { "data": { "text/html": [ " \n", " " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from ax import (\n", " ChoiceParameter,\n", " ComparisonOp,\n", " Experiment,\n", " FixedParameter,\n", " Metric,\n", " Objective,\n", " OptimizationConfig,\n", " OrderConstraint,\n", " OutcomeConstraint,\n", " ParameterType,\n", " RangeParameter,\n", " SearchSpace,\n", " SumConstraint,\n", ")\n", "from ax.modelbridge.registry import Models\n", "from ax.utils.notebook.plotting import init_notebook_plotting, render\n", "\n", "init_notebook_plotting()" ] }, { "cell_type": "markdown", "id": "6206c459", "metadata": { "code_folding": [], "hidden_ranges": [], "originalKey": "f522bb04-8372-4647-8c90-cffb8a664be3", "papermill": { "duration": 0.037012, "end_time": "2024-07-23T19:35:40.047700", "exception": false, "start_time": "2024-07-23T19:35:40.010688", "status": "completed" }, "showInput": true, "tags": [] }, "source": [ "## 1. Create Search Space\n", "\n", "First, we define a search space, which defines the type and allowed range for the parameters." ] }, { "cell_type": "code", "execution_count": 2, "id": "ff80e8ec", "metadata": { "execution": { "iopub.execute_input": "2024-07-23T19:35:40.123058Z", "iopub.status.busy": "2024-07-23T19:35:40.122541Z", "iopub.status.idle": "2024-07-23T19:35:40.127950Z", "shell.execute_reply": "2024-07-23T19:35:40.127382Z" }, "executionStartTime": 1646323256533, "executionStopTime": 1646323256546, "originalKey": "9b782d53-f9e2-4b13-a8ba-b7941aba802e", "papermill": { "duration": 0.044905, "end_time": "2024-07-23T19:35:40.129320", "exception": false, "start_time": "2024-07-23T19:35:40.084415", "status": "completed" }, "requestMsgId": "9b782d53-f9e2-4b13-a8ba-b7941aba802e", "tags": [] }, "outputs": [], "source": [ "from ax.metrics.l2norm import L2NormMetric\n", "from ax.metrics.hartmann6 import Hartmann6Metric\n", "\n", "\n", "hartmann_search_space = SearchSpace(\n", " parameters=[\n", " RangeParameter(\n", " name=f\"x{i}\", parameter_type=ParameterType.FLOAT, lower=0.0, upper=1.0\n", " )\n", " for i in range(6)\n", " ]\n", ")" ] }, { "cell_type": "markdown", "id": "24765971", "metadata": { "customInput": null, "originalKey": "9e0c312c-e290-4e7b-bf9c-45bd5c360c25", "papermill": { "duration": 0.037223, "end_time": "2024-07-23T19:35:40.204070", "exception": false, "start_time": "2024-07-23T19:35:40.166847", "status": "completed" }, "showInput": false, "tags": [] }, "source": [ "Note that there are two other parameter classes, FixedParameter and ChoiceParameter. Although we won't use these in this example, you can create them as follows.\n", "\n" ] }, { "cell_type": "code", "execution_count": 3, "id": "11e108ca", "metadata": { "code_folding": [], "customInput": null, "execution": { "iopub.execute_input": "2024-07-23T19:35:40.280736Z", "iopub.status.busy": "2024-07-23T19:35:40.280106Z", "iopub.status.idle": "2024-07-23T19:35:40.284503Z", "shell.execute_reply": "2024-07-23T19:35:40.283816Z" }, "executionStartTime": 1646323256562, "executionStopTime": 1646323256584, "hidden_ranges": [], "originalKey": "e29cbb8f-9045-4d9c-8a57-aeff1cd91da6", "papermill": { "duration": 0.044219, "end_time": "2024-07-23T19:35:40.285877", "exception": false, "start_time": "2024-07-23T19:35:40.241658", "status": "completed" }, "requestMsgId": "e29cbb8f-9045-4d9c-8a57-aeff1cd91da6", "showInput": true, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/tmp/ipykernel_2285/1805422181.py:1: AxParameterWarning:\n", "\n", "`is_ordered` is not specified for `ChoiceParameter` \"choice\". Defaulting to `True` since there are exactly two choices.. To override this behavior (or avoid this warning), specify `is_ordered` during `ChoiceParameter` construction. Note that choice parameters with exactly 2 choices are always considered ordered and that the user-supplied `is_ordered` has no effect in this particular case.\n", "\n", "/tmp/ipykernel_2285/1805422181.py:1: AxParameterWarning:\n", "\n", "`sort_values` is not specified for `ChoiceParameter` \"choice\". Defaulting to `False` for parameters of `ParameterType` STRING. To override this behavior (or avoid this warning), specify `sort_values` during `ChoiceParameter` construction.\n", "\n" ] } ], "source": [ "choice_param = ChoiceParameter(\n", " name=\"choice\", values=[\"foo\", \"bar\"], parameter_type=ParameterType.STRING\n", ")\n", "fixed_param = FixedParameter(\n", " name=\"fixed\", value=[True], parameter_type=ParameterType.BOOL\n", ")" ] }, { "cell_type": "markdown", "id": "b3a6d6a6", "metadata": { "customInput": null, "originalKey": "75b46af0-9739-46a6-9b95-21c8e2e9e22a", "papermill": { "duration": 0.037353, "end_time": "2024-07-23T19:35:40.360258", "exception": false, "start_time": "2024-07-23T19:35:40.322905", "status": "completed" }, "showInput": false, "tags": [] }, "source": [ "Sum constraints enforce that the sum of a set of parameters is greater or less than some bound, and order constraints enforce that one parameter is smaller than the other. We won't use these either, but see two examples below.\n", "\n" ] }, { "cell_type": "code", "execution_count": 4, "id": "2b69bce3", "metadata": { "code_folding": [], "customInput": null, "execution": { "iopub.execute_input": "2024-07-23T19:35:40.435919Z", "iopub.status.busy": "2024-07-23T19:35:40.435346Z", "iopub.status.idle": "2024-07-23T19:35:40.439335Z", "shell.execute_reply": "2024-07-23T19:35:40.438757Z" }, "executionStartTime": 1646323256616, "executionStopTime": 1646323256621, "hidden_ranges": [], "originalKey": "b782e8cf-c11c-4f4e-a416-2577a56b4100", "papermill": { "duration": 0.043422, "end_time": "2024-07-23T19:35:40.440737", "exception": false, "start_time": "2024-07-23T19:35:40.397315", "status": "completed" }, "requestMsgId": "b782e8cf-c11c-4f4e-a416-2577a56b4100", "showInput": true, "tags": [] }, "outputs": [], "source": [ "sum_constraint = SumConstraint(\n", " parameters=[\n", " hartmann_search_space.parameters[\"x0\"],\n", " hartmann_search_space.parameters[\"x1\"],\n", " ],\n", " is_upper_bound=True,\n", " bound=5.0,\n", ")\n", "\n", "order_constraint = OrderConstraint(\n", " lower_parameter=hartmann_search_space.parameters[\"x0\"],\n", " upper_parameter=hartmann_search_space.parameters[\"x1\"],\n", ")" ] }, { "cell_type": "markdown", "id": "339e24fa", "metadata": { "code_folding": [], "hidden_ranges": [], "originalKey": "7bf887e2-2b02-4237-ba5e-6fa8beaa85fb", "papermill": { "duration": 0.036933, "end_time": "2024-07-23T19:35:40.514775", "exception": false, "start_time": "2024-07-23T19:35:40.477842", "status": "completed" }, "showInput": false, "tags": [] }, "source": [ "## 2. Create Optimization Config\n", "\n", "Second, we define the `optimization_config` with an `objective` and `outcome_constraints`.\n", "\n", "When doing the optimization, we will find points that minimize the objective while obeying the constraints (which in this case means `l2norm < 1.25`).\n", "\n", "Note: we are using `Hartmann6Metric` and `L2NormMetric` here, which have built in evaluation functions for testing. For creating your own cutom metrics, see [8. Defining custom metrics](#8.-Defining-custom-metrics)." ] }, { "cell_type": "code", "execution_count": 5, "id": "238366a9", "metadata": { "code_folding": [], "execution": { "iopub.execute_input": "2024-07-23T19:35:40.590920Z", "iopub.status.busy": "2024-07-23T19:35:40.590445Z", "iopub.status.idle": "2024-07-23T19:35:40.594788Z", "shell.execute_reply": "2024-07-23T19:35:40.594114Z" }, "executionStartTime": 1646323256629, "executionStopTime": 1646323256633, "hidden_ranges": [], "originalKey": "d0e2b580-bfb5-4a73-8db1-34a3c43c3ef2", "papermill": { "duration": 0.043953, "end_time": "2024-07-23T19:35:40.596086", "exception": false, "start_time": "2024-07-23T19:35:40.552133", "status": "completed" }, "requestMsgId": "d0e2b580-bfb5-4a73-8db1-34a3c43c3ef2", "tags": [] }, "outputs": [], "source": [ "param_names = [f\"x{i}\" for i in range(6)]\n", "optimization_config = OptimizationConfig(\n", " objective=Objective(\n", " metric=Hartmann6Metric(name=\"hartmann6\", param_names=param_names),\n", " minimize=True,\n", " ),\n", " outcome_constraints=[\n", " OutcomeConstraint(\n", " metric=L2NormMetric(name=\"l2norm\", param_names=param_names, noise_sd=0.2),\n", " op=ComparisonOp.LEQ,\n", " bound=1.25,\n", " relative=False,\n", " )\n", " ],\n", ")" ] }, { "cell_type": "markdown", "id": "9353ef8e", "metadata": { "code_folding": [], "customInput": null, "hidden_ranges": [], "originalKey": "ed80a5e4-4786-4961-979e-22a295bfa7f0", "papermill": { "duration": 0.037196, "end_time": "2024-07-23T19:35:40.670765", "exception": false, "start_time": "2024-07-23T19:35:40.633569", "status": "completed" }, "showInput": false, "tags": [] }, "source": [ "## 3. Define a Runner\n", "Before an experiment can collect data, it must have a Runner attached. A runner handles the deployment of trials. A trial must be \"run\" before it can be evaluated.\n", "\n", "Here, we have a dummy runner that does nothing. In practice, a runner might be in charge of pushing an experiment to production.\n", "\n", "The only method that needs to be defined for runner subclasses is run, which performs any necessary deployment logic, and returns a dictionary of resulting metadata. This metadata can later be accessed through the trial's `run_metadata` property." ] }, { "cell_type": "code", "execution_count": 6, "id": "39ab255e", "metadata": { "code_folding": [], "customInput": null, "execution": { "iopub.execute_input": "2024-07-23T19:35:40.747082Z", "iopub.status.busy": "2024-07-23T19:35:40.746497Z", "iopub.status.idle": "2024-07-23T19:35:40.750369Z", "shell.execute_reply": "2024-07-23T19:35:40.749729Z" }, "executionStartTime": 1646323256641, "executionStopTime": 1646323256645, "hidden_ranges": [], "originalKey": "c9862804-4c0c-4691-be2c-5cb0eb778460", "papermill": { "duration": 0.043564, "end_time": "2024-07-23T19:35:40.751659", "exception": false, "start_time": "2024-07-23T19:35:40.708095", "status": "completed" }, "requestMsgId": "c9862804-4c0c-4691-be2c-5cb0eb778460", "showInput": true, "tags": [] }, "outputs": [], "source": [ "from ax import Runner\n", "\n", "\n", "class MyRunner(Runner):\n", " def run(self, trial):\n", " trial_metadata = {\"name\": str(trial.index)}\n", " return trial_metadata" ] }, { "cell_type": "markdown", "id": "e9bb2ecd", "metadata": { "code_folding": [], "customInput": null, "hidden_ranges": [], "originalKey": "131ab2a9-e2c7-4752-99a3-547c7dbe42ec", "papermill": { "duration": 0.037612, "end_time": "2024-07-23T19:35:40.826658", "exception": false, "start_time": "2024-07-23T19:35:40.789046", "status": "completed" }, "showInput": false, "tags": [] }, "source": [ "## 4. Create Experiment\n", "Next, we make an `Experiment` with our search space, runner, and optimization config." ] }, { "cell_type": "code", "execution_count": 7, "id": "daccd2b6", "metadata": { "code_folding": [], "customInput": null, "execution": { "iopub.execute_input": "2024-07-23T19:35:40.903352Z", "iopub.status.busy": "2024-07-23T19:35:40.902836Z", "iopub.status.idle": "2024-07-23T19:35:40.906383Z", "shell.execute_reply": "2024-07-23T19:35:40.905728Z" }, "executionStartTime": 1646323256653, "executionStopTime": 1646323256658, "hidden_ranges": [], "originalKey": "18ce7d69-d556-48f5-9945-c75bedb362bb", "papermill": { "duration": 0.043564, "end_time": "2024-07-23T19:35:40.907740", "exception": false, "start_time": "2024-07-23T19:35:40.864176", "status": "completed" }, "requestMsgId": "18ce7d69-d556-48f5-9945-c75bedb362bb", "showInput": true, "tags": [] }, "outputs": [], "source": [ "exp = Experiment(\n", " name=\"test_hartmann\",\n", " search_space=hartmann_search_space,\n", " optimization_config=optimization_config,\n", " runner=MyRunner(),\n", ")" ] }, { "cell_type": "markdown", "id": "30b9b264", "metadata": { "code_folding": [], "hidden_ranges": [], "originalKey": "8a04eba9-97f2-45f7-8b10-7216fe9c0101", "papermill": { "duration": 0.037405, "end_time": "2024-07-23T19:35:40.982859", "exception": false, "start_time": "2024-07-23T19:35:40.945454", "status": "completed" }, "showInput": true, "tags": [] }, "source": [ "## 5. Perform Optimization\n", "\n", "Run the optimization using the settings defined on the experiment. We will create 5 random sobol points for exploration followed by 15 points generated using the GPEI optimizer.\n", "\n", "Instead of a member of the `Models` enum to produce generator runs, users can leverage a `GenerationStrategy`. See the [Generation Strategy Tutorial](https://ax.dev/tutorials/generation_strategy.html) for more info." ] }, { "cell_type": "code", "execution_count": 8, "id": "8d1985aa", "metadata": { "code_folding": [], "execution": { "iopub.execute_input": "2024-07-23T19:35:41.059519Z", "iopub.status.busy": "2024-07-23T19:35:41.058890Z", "iopub.status.idle": "2024-07-23T19:37:45.066125Z", "shell.execute_reply": "2024-07-23T19:37:45.064831Z" }, "executionStartTime": 1646323256665, "executionStopTime": 1646323714923, "hidden_ranges": [], "originalKey": "b48e26da-57e7-4b81-baf0-122a71f0bb72", "papermill": { "duration": 124.047403, "end_time": "2024-07-23T19:37:45.067871", "exception": false, "start_time": "2024-07-23T19:35:41.020468", "status": "completed" }, "requestMsgId": "b48e26da-57e7-4b81-baf0-122a71f0bb72", "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Running Sobol initialization trials...\n", "Running BO trial 6/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running BO trial 7/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running BO trial 8/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running BO trial 9/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running BO trial 10/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running BO trial 11/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running BO trial 12/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running BO trial 13/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running BO trial 14/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running BO trial 15/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running BO trial 16/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running BO trial 17/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running BO trial 18/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running BO trial 19/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running BO trial 20/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Done!\n" ] } ], "source": [ "from ax.modelbridge.registry import Models\n", "\n", "NUM_SOBOL_TRIALS = 5\n", "NUM_BOTORCH_TRIALS = 15\n", "\n", "print(f\"Running Sobol initialization trials...\")\n", "sobol = Models.SOBOL(search_space=exp.search_space)\n", "\n", "for i in range(NUM_SOBOL_TRIALS):\n", " # Produce a GeneratorRun from the model, which contains proposed arm(s) and other metadata\n", " generator_run = sobol.gen(n=1)\n", " # Add generator run to a trial to make it part of the experiment and evaluate arm(s) in it\n", " trial = exp.new_trial(generator_run=generator_run)\n", " # Start trial run to evaluate arm(s) in the trial\n", " trial.run()\n", " # Mark trial as completed to record when a trial run is completed\n", " # and enable fetching of data for metrics on the experiment\n", " # (by default, trials must be completed before metrics can fetch their data,\n", " # unless a metric is explicitly configured otherwise)\n", " trial.mark_completed()\n", "\n", "for i in range(NUM_BOTORCH_TRIALS):\n", " print(\n", " f\"Running BO trial {i + NUM_SOBOL_TRIALS + 1}/{NUM_SOBOL_TRIALS + NUM_BOTORCH_TRIALS}...\"\n", " )\n", " # Reinitialize GP+EI model at each step with updated data.\n", " gpei = Models.BOTORCH_MODULAR(experiment=exp, data=exp.fetch_data())\n", " generator_run = gpei.gen(n=1)\n", " trial = exp.new_trial(generator_run=generator_run)\n", " trial.run()\n", " trial.mark_completed()\n", "\n", "print(\"Done!\")" ] }, { "cell_type": "markdown", "id": "0b344647", "metadata": { "code_folding": [], "hidden_ranges": [], "originalKey": "f503e648-e3f2-419f-a60e-5bfcbc6775bd", "papermill": { "duration": 0.055006, "end_time": "2024-07-23T19:37:45.181209", "exception": false, "start_time": "2024-07-23T19:37:45.126203", "status": "completed" }, "showInput": true, "tags": [] }, "source": [ "## 6. Inspect trials' data\n", "\n", "Now we can inspect the `Experiment`'s data by calling `fetch_data()`, which retrieves evaluation data for all trials of the experiment.\n", "\n", "To fetch trial data, we need to run it and mark it completed. For most metrics in Ax, data is only available once the status of the trial is `COMPLETED`, since in real-worlds scenarios, metrics can typically only be fetched after the trial finished running.\n", "\n", "NOTE: Metrics classes may implement the `is_available_while_running` method. When this method returns `True`, data is available when trials are either `RUNNING` or `COMPLETED`. This can be used to obtain intermediate results from A/B test trials and other online experiments, or when metric values are available immediately, like in the case of synthetic problem metrics.\n", "\n", "We can also use the `fetch_trials_data` function to get evaluation data for a specific trials in the experiment, like so:" ] }, { "cell_type": "code", "execution_count": 9, "id": "2d7f24a8", "metadata": { "code_folding": [], "execution": { "iopub.execute_input": "2024-07-23T19:37:45.258804Z", "iopub.status.busy": "2024-07-23T19:37:45.258214Z", "iopub.status.idle": "2024-07-23T19:37:45.284626Z", "shell.execute_reply": "2024-07-23T19:37:45.283959Z" }, "executionStartTime": 1646323714950, "executionStopTime": 1646323715210, "hidden_ranges": [], "originalKey": "65113ac3-956a-4fcc-abf7-a9d45f8ecb72", "papermill": { "duration": 0.066973, "end_time": "2024-07-23T19:37:45.286042", "exception": false, "start_time": "2024-07-23T19:37:45.219069", "status": "completed" }, "requestMsgId": "65113ac3-956a-4fcc-abf7-a9d45f8ecb72", "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
arm_namemetric_namemeansemtrial_indexnfrac_nonnull
019_0hartmann6-2.4631940.01910000-2.463194
119_0l2norm1.0811310.219100001.081131
\n", "
" ], "text/plain": [ " arm_name metric_name mean sem trial_index n frac_nonnull\n", "0 19_0 hartmann6 -2.463194 0.0 19 10000 -2.463194\n", "1 19_0 l2norm 1.081131 0.2 19 10000 1.081131" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "trial_data = exp.fetch_trials_data([NUM_SOBOL_TRIALS + NUM_BOTORCH_TRIALS - 1])\n", "trial_data.df" ] }, { "cell_type": "markdown", "id": "e25a0731", "metadata": { "code_folding": [], "customInput": null, "hidden_ranges": [], "originalKey": "74fa0cc5-075c-4b34-98b9-cbf74ad5bb26", "papermill": { "duration": 0.038077, "end_time": "2024-07-23T19:37:45.362161", "exception": false, "start_time": "2024-07-23T19:37:45.324084", "status": "completed" }, "showInput": true, "tags": [] }, "source": [ "The below call to `exp.fetch_data()` also attaches data to the last trial, which because of the way we looped through Botorch trials in [5. Perform Optimization](5.-Perform-Optimization), would otherwise not have data attached. This is necessary to get `objective_means` in [7. Plot results](7.-Plot-results)." ] }, { "cell_type": "code", "execution_count": 10, "id": "877ce80b", "metadata": { "code_folding": [], "customInput": null, "execution": { "iopub.execute_input": "2024-07-23T19:37:45.440329Z", "iopub.status.busy": "2024-07-23T19:37:45.439849Z", "iopub.status.idle": "2024-07-23T19:37:45.605086Z", "shell.execute_reply": "2024-07-23T19:37:45.604345Z" }, "executionStartTime": 1646323715232, "executionStopTime": 1646323715950, "hidden_ranges": [], "originalKey": "88fb1408-0965-48f9-a211-140ea57f46a6", "papermill": { "duration": 0.206159, "end_time": "2024-07-23T19:37:45.606566", "exception": false, "start_time": "2024-07-23T19:37:45.400407", "status": "completed" }, "requestMsgId": "88fb1408-0965-48f9-a211-140ea57f46a6", "showInput": true, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
arm_namemetric_namemeansemtrial_indexnfrac_nonnull
00_0hartmann6-0.0376550.0010000-0.037655
10_0l2norm1.3583300.20100001.358330
21_0hartmann6-0.1213410.0110000-0.121341
31_0l2norm0.7772970.21100000.777297
42_0hartmann6-0.0007210.0210000-0.000721
52_0l2norm1.3342950.22100001.334295
63_0hartmann6-0.3050530.0310000-0.305053
73_0l2norm1.3900860.23100001.390086
84_0hartmann6-0.4801910.0410000-0.480191
94_0l2norm1.3349210.24100001.334921
105_0hartmann6-1.1975190.0510000-1.197519
115_0l2norm1.4314040.25100001.431404
126_0hartmann6-1.4728430.0610000-1.472843
136_0l2norm1.3449960.26100001.344996
147_0hartmann6-1.1884060.0710000-1.188406
157_0l2norm1.2190810.27100001.219081
168_0hartmann6-1.5498710.0810000-1.549871
178_0l2norm1.0211040.28100001.021104
189_0hartmann6-1.8180140.0910000-1.818014
199_0l2norm0.9719510.29100000.971951
2010_0hartmann6-1.5719630.01010000-1.571963
2110_0l2norm1.4262750.210100001.426275
2211_0hartmann6-1.4962970.01110000-1.496297
2311_0l2norm1.5511290.211100001.551129
2412_0hartmann6-1.9928560.01210000-1.992856
2512_0l2norm1.2893220.212100001.289322
2613_0hartmann6-1.9765300.01310000-1.976530
2713_0l2norm0.6809370.213100000.680937
2814_0hartmann6-2.1318600.01410000-2.131860
2914_0l2norm1.0068450.214100001.006845
3015_0hartmann6-2.3189800.01510000-2.318980
3115_0l2norm0.8622630.215100000.862263
3216_0hartmann6-2.5175730.01610000-2.517573
3316_0l2norm1.0039950.216100001.003995
3417_0hartmann6-2.6026050.01710000-2.602605
3517_0l2norm0.6594200.217100000.659420
3618_0hartmann6-2.4054130.01810000-2.405413
3718_0l2norm0.7996250.218100000.799625
3819_0hartmann6-2.4631940.01910000-2.463194
3919_0l2norm0.8722880.219100000.872288
\n", "
" ], "text/plain": [ " arm_name metric_name mean sem trial_index n frac_nonnull\n", "0 0_0 hartmann6 -0.037655 0.0 0 10000 -0.037655\n", "1 0_0 l2norm 1.358330 0.2 0 10000 1.358330\n", "2 1_0 hartmann6 -0.121341 0.0 1 10000 -0.121341\n", "3 1_0 l2norm 0.777297 0.2 1 10000 0.777297\n", "4 2_0 hartmann6 -0.000721 0.0 2 10000 -0.000721\n", "5 2_0 l2norm 1.334295 0.2 2 10000 1.334295\n", "6 3_0 hartmann6 -0.305053 0.0 3 10000 -0.305053\n", "7 3_0 l2norm 1.390086 0.2 3 10000 1.390086\n", "8 4_0 hartmann6 -0.480191 0.0 4 10000 -0.480191\n", "9 4_0 l2norm 1.334921 0.2 4 10000 1.334921\n", "10 5_0 hartmann6 -1.197519 0.0 5 10000 -1.197519\n", "11 5_0 l2norm 1.431404 0.2 5 10000 1.431404\n", "12 6_0 hartmann6 -1.472843 0.0 6 10000 -1.472843\n", "13 6_0 l2norm 1.344996 0.2 6 10000 1.344996\n", "14 7_0 hartmann6 -1.188406 0.0 7 10000 -1.188406\n", "15 7_0 l2norm 1.219081 0.2 7 10000 1.219081\n", "16 8_0 hartmann6 -1.549871 0.0 8 10000 -1.549871\n", "17 8_0 l2norm 1.021104 0.2 8 10000 1.021104\n", "18 9_0 hartmann6 -1.818014 0.0 9 10000 -1.818014\n", "19 9_0 l2norm 0.971951 0.2 9 10000 0.971951\n", "20 10_0 hartmann6 -1.571963 0.0 10 10000 -1.571963\n", "21 10_0 l2norm 1.426275 0.2 10 10000 1.426275\n", "22 11_0 hartmann6 -1.496297 0.0 11 10000 -1.496297\n", "23 11_0 l2norm 1.551129 0.2 11 10000 1.551129\n", "24 12_0 hartmann6 -1.992856 0.0 12 10000 -1.992856\n", "25 12_0 l2norm 1.289322 0.2 12 10000 1.289322\n", "26 13_0 hartmann6 -1.976530 0.0 13 10000 -1.976530\n", "27 13_0 l2norm 0.680937 0.2 13 10000 0.680937\n", "28 14_0 hartmann6 -2.131860 0.0 14 10000 -2.131860\n", "29 14_0 l2norm 1.006845 0.2 14 10000 1.006845\n", "30 15_0 hartmann6 -2.318980 0.0 15 10000 -2.318980\n", "31 15_0 l2norm 0.862263 0.2 15 10000 0.862263\n", "32 16_0 hartmann6 -2.517573 0.0 16 10000 -2.517573\n", "33 16_0 l2norm 1.003995 0.2 16 10000 1.003995\n", "34 17_0 hartmann6 -2.602605 0.0 17 10000 -2.602605\n", "35 17_0 l2norm 0.659420 0.2 17 10000 0.659420\n", "36 18_0 hartmann6 -2.405413 0.0 18 10000 -2.405413\n", "37 18_0 l2norm 0.799625 0.2 18 10000 0.799625\n", "38 19_0 hartmann6 -2.463194 0.0 19 10000 -2.463194\n", "39 19_0 l2norm 0.872288 0.2 19 10000 0.872288" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "exp.fetch_data().df" ] }, { "cell_type": "markdown", "id": "7d59a181", "metadata": { "code_folding": [], "hidden_ranges": [], "originalKey": "940865f9-af61-4668-aea0-b19ed5c5497d", "papermill": { "duration": 0.039017, "end_time": "2024-07-23T19:37:45.684370", "exception": false, "start_time": "2024-07-23T19:37:45.645353", "status": "completed" }, "showInput": false, "tags": [] }, "source": [ "## 7. Plot results\n", "Now we can plot the results of our optimization:" ] }, { "cell_type": "code", "execution_count": 11, "id": "38ad0dde", "metadata": { "code_folding": [], "execution": { "iopub.execute_input": "2024-07-23T19:37:45.763875Z", "iopub.status.busy": "2024-07-23T19:37:45.763222Z", "iopub.status.idle": "2024-07-23T19:37:45.992997Z", "shell.execute_reply": "2024-07-23T19:37:45.992266Z" }, "executionStartTime": 1646323715983, "executionStopTime": 1646323716634, "hidden_ranges": [], "originalKey": "5a4d2c4d-756a-492a-8938-d080a499b66c", "papermill": { "duration": 0.271838, "end_time": "2024-07-23T19:37:45.994908", "exception": false, "start_time": "2024-07-23T19:37:45.723070", "status": "completed" }, "requestMsgId": "5a4d2c4d-756a-492a-8938-d080a499b66c", "tags": [] }, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "linkText": "Export to plot.ly", "plotlyServerURL": "https://plot.ly", "showLink": false }, "data": [ { "hoverinfo": "none", "legendgroup": "", "line": { "width": 0 }, "mode": "lines", "showlegend": false, "type": "scatter", "x": [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 ], "y": [ -0.03765541731608217, -0.12134147630819689, -0.12134147630819689, -0.3050526079999184, -0.48019059707724854, -1.1975185346860686, -1.4728432773106108, -1.4728432773106108, -1.5498713140458125, -1.8180139638122486, -1.8180139638122486, -1.8180139638122486, -1.9928555311385057, -1.9928555311385057, -2.1318602129035353, -2.318979968143476, -2.517572633836711, -2.602604706033525, -2.602604706033525, -2.602604706033525 ] }, { "fill": "tonexty", "fillcolor": "rgba(128,177,211,0.3)", "legendgroup": "objective value", "line": { "color": "rgba(128,177,211,1)" }, "mode": "lines", "name": "objective value", "type": "scatter", "x": [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 ], "y": [ -0.03765541731608217, -0.12134147630819689, -0.12134147630819689, -0.3050526079999184, -0.48019059707724854, -1.1975185346860686, -1.4728432773106108, -1.4728432773106108, -1.5498713140458125, -1.8180139638122486, -1.8180139638122486, -1.8180139638122486, -1.9928555311385057, -1.9928555311385057, -2.1318602129035353, -2.318979968143476, -2.517572633836711, -2.602604706033525, -2.602604706033525, -2.602604706033525 ] }, { "fill": "tonexty", "fillcolor": "rgba(128,177,211,0.3)", "hoverinfo": "none", "legendgroup": "", "line": { "width": 0 }, "mode": "lines", "showlegend": false, "type": "scatter", "x": [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 ], "y": [ -0.03765541731608217, -0.12134147630819689, -0.12134147630819689, -0.3050526079999184, -0.48019059707724854, -1.1975185346860686, -1.4728432773106108, -1.4728432773106108, -1.5498713140458125, -1.8180139638122486, -1.8180139638122486, -1.8180139638122486, -1.9928555311385057, -1.9928555311385057, -2.1318602129035353, -2.318979968143476, -2.517572633836711, -2.602604706033525, -2.602604706033525, -2.602604706033525 ] }, { "line": { "color": "rgba(253,180,98,1)", "dash": "dash" }, "mode": "lines", "name": "Optimum", "type": "scatter", "x": [ 1, 20 ], "y": [ -3.32237, -3.32237 ] } ], "layout": { "showlegend": true, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "fillpattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "sequentialminus": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "title": { "text": "" }, "xaxis": { "title": { "text": "Iteration" } }, "yaxis": { "title": { "text": "" } } } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import numpy as np\n", "from ax.plot.trace import optimization_trace_single_method\n", "\n", "# `plot_single_method` expects a 2-d array of means, because it expects to average means from multiple\n", "# optimization runs, so we wrap out best objectives array in another array.\n", "objective_means = np.array([[trial.objective_mean for trial in exp.trials.values()]])\n", "best_objective_plot = optimization_trace_single_method(\n", " y=np.minimum.accumulate(objective_means, axis=1),\n", " optimum=-3.32237, # Known minimum objective for Hartmann6 function.\n", ")\n", "render(best_objective_plot)" ] }, { "cell_type": "markdown", "id": "1d088bd5", "metadata": { "code_folding": [], "customInput": null, "hidden_ranges": [], "originalKey": "934db3fd-1dce-421b-8228-820025f3821a", "papermill": { "duration": 0.044766, "end_time": "2024-07-23T19:37:46.086779", "exception": false, "start_time": "2024-07-23T19:37:46.042013", "status": "completed" }, "showInput": true, "tags": [] }, "source": [ "## 8. Defining custom metrics\n", "In order to perform an optimization, we also need to define an optimization config for the experiment. An optimization config is composed of an objective metric to be minimized or maximized in the experiment, and optionally a set of outcome constraints that place restrictions on how other metrics can be moved by the experiment.\n", "\n", "In order to define an objective or outcome constraint, we first need to subclass Metric. Metrics are used to evaluate trials, which are individual steps of the experiment sequence. Each trial contains one or more arms for which we will collect data at the same time.\n", "\n", "Our custom metric(s) will determine how, given a trial, to compute the mean and SEM of each of the trial's arms.\n", "\n", "The only method that needs to be defined for most metric subclasses is `fetch_trial_data`, which defines how a single trial is evaluated, and returns a pandas dataframe.\n", " \n", "The `is_available_while_running` method is optional and returns a boolean, specifying whether the trial data can be fetched before the trial is complete. See [6. Inspect trials' data](6.-Inspect-trials'-data) for more details." ] }, { "cell_type": "code", "execution_count": 12, "id": "c12cbd5b", "metadata": { "code_folding": [], "customInput": null, "execution": { "iopub.execute_input": "2024-07-23T19:37:46.176612Z", "iopub.status.busy": "2024-07-23T19:37:46.176009Z", "iopub.status.idle": "2024-07-23T19:37:46.181521Z", "shell.execute_reply": "2024-07-23T19:37:46.180832Z" }, "executionStartTime": 1646323716638, "executionStopTime": 1646323716697, "hidden_ranges": [], "originalKey": "7ec75ae4-1d7f-4ff4-8d9d-b77fdf28ccfe", "papermill": { "duration": 0.050168, "end_time": "2024-07-23T19:37:46.182860", "exception": false, "start_time": "2024-07-23T19:37:46.132692", "status": "completed" }, "requestMsgId": "7ec75ae4-1d7f-4ff4-8d9d-b77fdf28ccfe", "showInput": true, "tags": [] }, "outputs": [], "source": [ "from ax import Data\n", "import pandas as pd\n", "\n", "\n", "class BoothMetric(Metric):\n", " def fetch_trial_data(self, trial):\n", " records = []\n", " for arm_name, arm in trial.arms_by_name.items():\n", " params = arm.parameters\n", " records.append(\n", " {\n", " \"arm_name\": arm_name,\n", " \"metric_name\": self.name,\n", " \"trial_index\": trial.index,\n", " # in practice, the mean and sem will be looked up based on trial metadata\n", " # but for this tutorial we will calculate them\n", " \"mean\": (params[\"x1\"] + 2 * params[\"x2\"] - 7) ** 2\n", " + (2 * params[\"x1\"] + params[\"x2\"] - 5) ** 2,\n", " \"sem\": 0.0,\n", " }\n", " )\n", " return Data(df=pd.DataFrame.from_records(records))\n", "\n", " def is_available_while_running(self) -> bool:\n", " return True" ] }, { "cell_type": "markdown", "id": "7fe66f35", "metadata": { "code_folding": [], "customInput": null, "hidden_ranges": [], "originalKey": "92fcddf9-9d86-45cd-b9fb-a0a7acdb267d", "papermill": { "duration": 0.042185, "end_time": "2024-07-23T19:37:46.267485", "exception": false, "start_time": "2024-07-23T19:37:46.225300", "status": "completed" }, "showInput": false, "tags": [] }, "source": [ "## 9. Save to JSON or SQL\n", "At any point, we can also save our experiment to a JSON file. To ensure that our custom metrics and runner are saved properly, we first need to register them." ] }, { "cell_type": "code", "execution_count": 13, "id": "e715a23f", "metadata": { "code_folding": [], "customInput": null, "execution": { "iopub.execute_input": "2024-07-23T19:37:46.353811Z", "iopub.status.busy": "2024-07-23T19:37:46.353302Z", "iopub.status.idle": "2024-07-23T19:37:46.701232Z", "shell.execute_reply": "2024-07-23T19:37:46.700493Z" }, "executionStartTime": 1646324682655, "executionStopTime": 1646324682796, "hidden_ranges": [], "originalKey": "f57e11d7-cc68-4323-a0cd-ff6f464dcd97", "papermill": { "duration": 0.393205, "end_time": "2024-07-23T19:37:46.702901", "exception": false, "start_time": "2024-07-23T19:37:46.309696", "status": "completed" }, "requestMsgId": "f57e11d7-cc68-4323-a0cd-ff6f464dcd97", "showInput": true, "tags": [] }, "outputs": [], "source": [ "from ax.storage.registry_bundle import RegistryBundle\n", "\n", "bundle = RegistryBundle(\n", " metric_clss={BoothMetric: None, L2NormMetric: None, Hartmann6Metric: None},\n", " runner_clss={MyRunner: None},\n", ")\n", "\n", "from ax.storage.json_store.load import load_experiment\n", "from ax.storage.json_store.save import save_experiment\n", "\n", "save_experiment(exp, \"experiment.json\", encoder_registry=bundle.encoder_registry)" ] }, { "cell_type": "code", "execution_count": 14, "id": "551a153a", "metadata": { "code_folding": [], "customInput": null, "execution": { "iopub.execute_input": "2024-07-23T19:37:46.789956Z", "iopub.status.busy": "2024-07-23T19:37:46.789228Z", "iopub.status.idle": "2024-07-23T19:37:47.662601Z", "shell.execute_reply": "2024-07-23T19:37:47.661956Z" }, "executionStartTime": 1646324718153, "executionStopTime": 1646324720104, "hidden_ranges": [], "originalKey": "e19ec7fb-f266-417e-ad17-5662a53a9ae3", "papermill": { "duration": 0.918803, "end_time": "2024-07-23T19:37:47.664186", "exception": false, "start_time": "2024-07-23T19:37:46.745383", "status": "completed" }, "requestMsgId": "e19ec7fb-f266-417e-ad17-5662a53a9ae3", "showInput": true, "tags": [] }, "outputs": [], "source": [ "loaded_experiment = load_experiment(\n", " \"experiment.json\", decoder_registry=bundle.decoder_registry\n", ")" ] }, { "cell_type": "markdown", "id": "323ab6c3", "metadata": { "customInput": null, "originalKey": "dc1f6800-437e-45de-85d3-276ae5f8ca99", "papermill": { "duration": 0.042319, "end_time": "2024-07-23T19:37:47.748955", "exception": false, "start_time": "2024-07-23T19:37:47.706636", "status": "completed" }, "showInput": false, "tags": [] }, "source": [ "To save our experiment to SQL, we must first specify a connection to a database and create all necessary tables.\n", "\n" ] }, { "cell_type": "code", "execution_count": 15, "id": "cd25115b", "metadata": { "code_folding": [], "customInput": null, "execution": { "iopub.execute_input": "2024-07-23T19:37:47.835140Z", "iopub.status.busy": "2024-07-23T19:37:47.834609Z", "iopub.status.idle": "2024-07-23T19:37:47.871312Z", "shell.execute_reply": "2024-07-23T19:37:47.870756Z" }, "executionStartTime": 1646324834810, "executionStopTime": 1646324835293, "hidden_ranges": [], "originalKey": "a0376ade-9a26-430b-b08b-0b93e890539c", "papermill": { "duration": 0.08147, "end_time": "2024-07-23T19:37:47.872799", "exception": false, "start_time": "2024-07-23T19:37:47.791329", "status": "completed" }, "requestMsgId": "a0376ade-9a26-430b-b08b-0b93e890539c", "showInput": true, "tags": [] }, "outputs": [], "source": [ "from ax.storage.sqa_store.db import (\n", " init_engine_and_session_factory,\n", " get_engine,\n", " create_all_tables,\n", ")\n", "from ax.storage.sqa_store.load import load_experiment\n", "from ax.storage.sqa_store.save import save_experiment\n", "\n", "init_engine_and_session_factory(url=\"sqlite:///foo3.db\")\n", "\n", "engine = get_engine()\n", "create_all_tables(engine)" ] }, { "cell_type": "code", "execution_count": 16, "id": "9a2faf88", "metadata": { "code_folding": [], "customInput": null, "execution": { "iopub.execute_input": "2024-07-23T19:37:47.959915Z", "iopub.status.busy": "2024-07-23T19:37:47.959421Z", "iopub.status.idle": "2024-07-23T19:37:50.067082Z", "shell.execute_reply": "2024-07-23T19:37:50.066356Z" }, "executionStartTime": 1646324891053, "executionStopTime": 1646324897271, "hidden_ranges": [], "originalKey": "82f58ead-8f0d-44cf-9fa8-dd67f7c8c8df", "papermill": { "duration": 2.153341, "end_time": "2024-07-23T19:37:50.068920", "exception": false, "start_time": "2024-07-23T19:37:47.915579", "status": "completed" }, "requestMsgId": "82f58ead-8f0d-44cf-9fa8-dd67f7c8c8df", "showInput": true, "tags": [] }, "outputs": [], "source": [ "from ax.storage.sqa_store.sqa_config import SQAConfig\n", "\n", "exp.name = \"new\"\n", "\n", "sqa_config = SQAConfig(\n", " json_encoder_registry=bundle.encoder_registry,\n", " json_decoder_registry=bundle.decoder_registry,\n", " metric_registry=bundle.metric_registry,\n", " runner_registry=bundle.runner_registry,\n", ")\n", "\n", "save_experiment(exp, config=sqa_config)" ] }, { "cell_type": "code", "execution_count": 17, "id": "c4d376b3", "metadata": { "code_folding": [], "customInput": null, "execution": { "iopub.execute_input": "2024-07-23T19:37:50.155265Z", "iopub.status.busy": "2024-07-23T19:37:50.154968Z", "iopub.status.idle": "2024-07-23T19:37:51.516420Z", "shell.execute_reply": "2024-07-23T19:37:51.515790Z" }, "executionStartTime": 1646324904964, "executionStopTime": 1646324906901, "hidden_ranges": [], "originalKey": "ed1be69c-da92-4a1d-a5e8-e76bba42f0ba", "papermill": { "duration": 1.406288, "end_time": "2024-07-23T19:37:51.517750", "exception": false, "start_time": "2024-07-23T19:37:50.111462", "status": "completed" }, "requestMsgId": "ed1be69c-da92-4a1d-a5e8-e76bba42f0ba", "showInput": true, "tags": [] }, "outputs": [ { "data": { "text/plain": [ "Experiment(new)" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "load_experiment(exp.name, config=sqa_config)" ] }, { "cell_type": "code", "execution_count": null, "id": "f148d12b", "metadata": { "customInput": null, "originalKey": "d144e372-c212-4454-b507-564c825c1fc5", "papermill": { "duration": 0.042689, "end_time": "2024-07-23T19:37:51.603131", "exception": false, "start_time": "2024-07-23T19:37:51.560442", "status": "completed" }, "showInput": true, "tags": [] }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.14" }, "papermill": { "default_parameters": {}, "duration": 138.062411, "end_time": "2024-07-23T19:37:53.964154", "environment_variables": {}, "exception": null, "input_path": "/tmp/tmp.DL1QmpHQMI/Ax-main/tutorials/gpei_hartmann_developer.ipynb", "output_path": "/tmp/tmp.DL1QmpHQMI/Ax-main/tutorials/gpei_hartmann_developer.ipynb", "parameters": {}, "start_time": "2024-07-23T19:35:35.901743", "version": "2.6.0" } }, "nbformat": 4, "nbformat_minor": 5 }