{ "cells": [ { "cell_type": "markdown", "metadata": { "code_folding": [], "hidden_ranges": [], "originalKey": "08064d6a-453e-44d7-85dc-896d40b6303a", "showInput": true }, "source": [ "# Developer API Example on Hartmann6\n", "\n", "The Developer API is suitable when the user wants maximal customization of the optimization loop. This tutorial demonstrates optimization of a Hartmann6 function using the `Experiment` construct. In this example, trials will be evaluated synchronously." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "code_folding": [], "collapsed": false, "execution": { "iopub.execute_input": "2022-11-10T20:27:53.206629Z", "iopub.status.busy": "2022-11-10T20:27:53.206201Z", "iopub.status.idle": "2022-11-10T20:27:55.316899Z", "shell.execute_reply": "2022-11-10T20:27:55.316277Z" }, "executionStartTime": 1646323252842, "executionStopTime": 1646323256492, "hidden_ranges": [], "originalKey": "7b98b243-30da-468b-82c7-7e22dbce6b57", "requestMsgId": "7b98b243-30da-468b-82c7-7e22dbce6b57" }, "outputs": [ { "data": { "text/html": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 11-10 20:27:55] ax.utils.notebook.plotting: Injecting Plotly library into cell. Do not overwrite or delete cell.\n" ] }, { "data": { "text/html": [ " \n", " " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from ax import (\n", " ComparisonOp,\n", " ParameterType, \n", " RangeParameter,\n", " ChoiceParameter,\n", " FixedParameter,\n", " SearchSpace, \n", " Experiment, \n", " OutcomeConstraint, \n", " OrderConstraint,\n", " SumConstraint,\n", " OptimizationConfig,\n", " Objective,\n", " Metric,\n", ")\n", "from ax.utils.notebook.plotting import render, init_notebook_plotting\n", "\n", "init_notebook_plotting()" ] }, { "cell_type": "markdown", "metadata": { "code_folding": [], "hidden_ranges": [], "originalKey": "f522bb04-8372-4647-8c90-cffb8a664be3", "showInput": true }, "source": [ "## 1. Create Search Space\n", "\n", "First, we define a search space, which defines the type and allowed range for the parameters." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false, "execution": { "iopub.execute_input": "2022-11-10T20:27:55.376141Z", "iopub.status.busy": "2022-11-10T20:27:55.374858Z", "iopub.status.idle": "2022-11-10T20:27:55.382333Z", "shell.execute_reply": "2022-11-10T20:27:55.381229Z" }, "executionStartTime": 1646323256533, "executionStopTime": 1646323256546, "originalKey": "9b782d53-f9e2-4b13-a8ba-b7941aba802e", "requestMsgId": "9b782d53-f9e2-4b13-a8ba-b7941aba802e" }, "outputs": [], "source": [ "hartmann_search_space = SearchSpace(\n", " parameters=[\n", " RangeParameter(\n", " name=f\"x{i}\", parameter_type=ParameterType.FLOAT, lower=0.0, upper=1.0\n", " )\n", " for i in range(6)\n", " ]\n", ")" ] }, { "cell_type": "markdown", "metadata": { "customInput": null, "originalKey": "9e0c312c-e290-4e7b-bf9c-45bd5c360c25", "showInput": false }, "source": [ "Note that there are two other parameter classes, FixedParameter and ChoiceParameter. Although we won't use these in this example, you can create them as follows.\n", "\n" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "code_folding": [], "collapsed": false, "customInput": null, "execution": { "iopub.execute_input": "2022-11-10T20:27:55.385498Z", "iopub.status.busy": "2022-11-10T20:27:55.384966Z", "iopub.status.idle": "2022-11-10T20:27:55.389214Z", "shell.execute_reply": "2022-11-10T20:27:55.388649Z" }, "executionStartTime": 1646323256562, "executionStopTime": 1646323256584, "hidden_ranges": [], "originalKey": "e29cbb8f-9045-4d9c-8a57-aeff1cd91da6", "requestMsgId": "e29cbb8f-9045-4d9c-8a57-aeff1cd91da6", "showInput": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/home/runner/work/Ax/Ax/ax/core/parameter.py:481: UserWarning:\n", "\n", "`is_ordered` is not specified for `ChoiceParameter` \"choice\". Defaulting to `False` for parameters of `ParameterType` STRING. To override this behavior (or avoid this warning), specify `is_ordered` during `ChoiceParameter` construction.\n", "\n", "/home/runner/work/Ax/Ax/ax/core/parameter.py:481: UserWarning:\n", "\n", "`sort_values` is not specified for `ChoiceParameter` \"choice\". Defaulting to `False` for parameters of `ParameterType` STRING. To override this behavior (or avoid this warning), specify `sort_values` during `ChoiceParameter` construction.\n", "\n" ] } ], "source": [ "choice_param = ChoiceParameter(name=\"choice\", values=[\"foo\", \"bar\"], parameter_type=ParameterType.STRING)\n", "fixed_param = FixedParameter(name=\"fixed\", value=[True], parameter_type=ParameterType.BOOL)" ] }, { "cell_type": "markdown", "metadata": { "customInput": null, "originalKey": "75b46af0-9739-46a6-9b95-21c8e2e9e22a", "showInput": false }, "source": [ "Sum constraints enforce that the sum of a set of parameters is greater or less than some bound, and order constraints enforce that one parameter is smaller than the other. We won't use these either, but see two examples below.\n", "\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "code_folding": [], "collapsed": false, "customInput": null, "execution": { "iopub.execute_input": "2022-11-10T20:27:55.391887Z", "iopub.status.busy": "2022-11-10T20:27:55.391402Z", "iopub.status.idle": "2022-11-10T20:27:55.395527Z", "shell.execute_reply": "2022-11-10T20:27:55.394946Z" }, "executionStartTime": 1646323256616, "executionStopTime": 1646323256621, "hidden_ranges": [], "originalKey": "b782e8cf-c11c-4f4e-a416-2577a56b4100", "requestMsgId": "b782e8cf-c11c-4f4e-a416-2577a56b4100", "showInput": true }, "outputs": [], "source": [ "sum_constraint = SumConstraint(\n", " parameters=[hartmann_search_space.parameters['x0'], hartmann_search_space.parameters['x1']], \n", " is_upper_bound=True, \n", " bound=5.0,\n", ")\n", "\n", "order_constraint = OrderConstraint(\n", " lower_parameter = hartmann_search_space.parameters['x0'],\n", " upper_parameter = hartmann_search_space.parameters['x1'],\n", ")" ] }, { "cell_type": "markdown", "metadata": { "code_folding": [], "hidden_ranges": [], "originalKey": "7bf887e2-2b02-4237-ba5e-6fa8beaa85fb", "showInput": false }, "source": [ "## 2. Create Optimization Config\n", "\n", "Second, we define the `optimization_config` with an `objective` and `outcome_constraints`.\n", "\n", "When doing the optimization, we will find points that minimize the objective while obeying the constraints (which in this case means `l2norm < 1.25`).\n", "\n", "Note: we are using `Hartmann6Metric` and `L2NormMetric` here, which have built in evaluation functions for testing. For creating your own cutom metrics, see [8. Defining custom metrics](#8.-Defining-custom-metrics)." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "code_folding": [], "collapsed": false, "execution": { "iopub.execute_input": "2022-11-10T20:27:55.398494Z", "iopub.status.busy": "2022-11-10T20:27:55.398118Z", "iopub.status.idle": "2022-11-10T20:27:55.402839Z", "shell.execute_reply": "2022-11-10T20:27:55.402122Z" }, "executionStartTime": 1646323256629, "executionStopTime": 1646323256633, "hidden_ranges": [], "originalKey": "d0e2b580-bfb5-4a73-8db1-34a3c43c3ef2", "requestMsgId": "d0e2b580-bfb5-4a73-8db1-34a3c43c3ef2" }, "outputs": [], "source": [ "from ax.metrics.l2norm import L2NormMetric\n", "from ax.metrics.hartmann6 import Hartmann6Metric\n", "\n", "param_names = [f\"x{i}\" for i in range(6)]\n", "optimization_config = OptimizationConfig(\n", " objective = Objective(\n", " metric=Hartmann6Metric(name=\"hartmann6\", param_names=param_names), \n", " minimize=True,\n", " ),\n", " outcome_constraints=[\n", " OutcomeConstraint(\n", " metric=L2NormMetric(\n", " name=\"l2norm\", param_names=param_names, noise_sd=0.2\n", " ),\n", " op=ComparisonOp.LEQ,\n", " bound=1.25,\n", " relative=False,\n", " )\n", " ],\n", ")" ] }, { "cell_type": "markdown", "metadata": { "code_folding": [], "customInput": null, "hidden_ranges": [], "originalKey": "ed80a5e4-4786-4961-979e-22a295bfa7f0", "showInput": false }, "source": [ "## 3. Define a Runner\n", "Before an experiment can collect data, it must have a Runner attached. A runner handles the deployment of trials. A trial must be \"run\" before it can be evaluated.\n", "\n", "Here, we have a dummy runner that does nothing. In practice, a runner might be in charge of pushing an experiment to production.\n", "\n", "The only method that needs to be defined for runner subclasses is run, which performs any necessary deployment logic, and returns a dictionary of resulting metadata. This metadata can later be accessed through the trial's `run_metadata` property." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "code_folding": [], "collapsed": false, "customInput": null, "execution": { "iopub.execute_input": "2022-11-10T20:27:55.405066Z", "iopub.status.busy": "2022-11-10T20:27:55.404757Z", "iopub.status.idle": "2022-11-10T20:27:55.408208Z", "shell.execute_reply": "2022-11-10T20:27:55.407627Z" }, "executionStartTime": 1646323256641, "executionStopTime": 1646323256645, "hidden_ranges": [], "originalKey": "c9862804-4c0c-4691-be2c-5cb0eb778460", "requestMsgId": "c9862804-4c0c-4691-be2c-5cb0eb778460", "showInput": true }, "outputs": [], "source": [ "from ax import Runner\n", "\n", "class MyRunner(Runner):\n", " def run(self, trial):\n", " trial_metadata = {\"name\": str(trial.index)}\n", " return trial_metadata" ] }, { "cell_type": "markdown", "metadata": { "code_folding": [], "customInput": null, "hidden_ranges": [], "originalKey": "131ab2a9-e2c7-4752-99a3-547c7dbe42ec", "showInput": false }, "source": [ "## 4. Create Experiment\n", "Next, we make an `Experiment` with our search space, runner, and optimization config." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "code_folding": [], "collapsed": false, "customInput": null, "execution": { "iopub.execute_input": "2022-11-10T20:27:55.411229Z", "iopub.status.busy": "2022-11-10T20:27:55.410833Z", "iopub.status.idle": "2022-11-10T20:27:55.414513Z", "shell.execute_reply": "2022-11-10T20:27:55.413909Z" }, "executionStartTime": 1646323256653, "executionStopTime": 1646323256658, "hidden_ranges": [], "originalKey": "18ce7d69-d556-48f5-9945-c75bedb362bb", "requestMsgId": "18ce7d69-d556-48f5-9945-c75bedb362bb", "showInput": true }, "outputs": [], "source": [ "exp = Experiment(\n", " name=\"test_hartmann\",\n", " search_space=hartmann_search_space,\n", " optimization_config=optimization_config,\n", " runner=MyRunner(),\n", ")" ] }, { "cell_type": "markdown", "metadata": { "code_folding": [], "hidden_ranges": [], "originalKey": "8a04eba9-97f2-45f7-8b10-7216fe9c0101", "showInput": true }, "source": [ "## 5. Perform Optimization\n", "\n", "Run the optimization using the settings defined on the experiment. We will create 5 random sobol points for exploration followed by 15 points generated using the GPEI optimizer.\n", "\n", "Instead of a member of the `Models` enum to produce generator runs, users can leverage a `GenerationStrategy`. See the [Generation Strategy Tutorial](https://ax.dev/tutorials/generation_strategy.html) for more info." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "code_folding": [], "collapsed": false, "execution": { "iopub.execute_input": "2022-11-10T20:27:55.417177Z", "iopub.status.busy": "2022-11-10T20:27:55.416827Z", "iopub.status.idle": "2022-11-10T20:31:57.564552Z", "shell.execute_reply": "2022-11-10T20:31:57.564017Z" }, "executionStartTime": 1646323256665, "executionStopTime": 1646323714923, "hidden_ranges": [], "originalKey": "b48e26da-57e7-4b81-baf0-122a71f0bb72", "requestMsgId": "b48e26da-57e7-4b81-baf0-122a71f0bb72" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Running Sobol initialization trials...\n", "Running GP+EI optimization trial 6/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running GP+EI optimization trial 7/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running GP+EI optimization trial 8/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running GP+EI optimization trial 9/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running GP+EI optimization trial 10/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running GP+EI optimization trial 11/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running GP+EI optimization trial 12/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running GP+EI optimization trial 13/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running GP+EI optimization trial 14/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running GP+EI optimization trial 15/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running GP+EI optimization trial 16/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running GP+EI optimization trial 17/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running GP+EI optimization trial 18/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running GP+EI optimization trial 19/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running GP+EI optimization trial 20/20...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Done!\n" ] } ], "source": [ "from ax.modelbridge.registry import Models\n", "\n", "NUM_SOBOL_TRIALS = 5\n", "NUM_BOTORCH_TRIALS = 15\n", "\n", "print(f\"Running Sobol initialization trials...\")\n", "sobol = Models.SOBOL(search_space=exp.search_space)\n", " \n", "for i in range(NUM_SOBOL_TRIALS):\n", " # Produce a GeneratorRun from the model, which contains proposed arm(s) and other metadata\n", " generator_run = sobol.gen(n=1)\n", " # Add generator run to a trial to make it part of the experiment and evaluate arm(s) in it\n", " trial = exp.new_trial(generator_run=generator_run)\n", " # Start trial run to evaluate arm(s) in the trial\n", " trial.run()\n", " # Mark trial as completed to record when a trial run is completed \n", " # and enable fetching of data for metrics on the experiment \n", " # (by default, trials must be completed before metrics can fetch their data,\n", " # unless a metric is explicitly configured otherwise)\n", " trial.mark_completed()\n", "\n", "for i in range(NUM_BOTORCH_TRIALS):\n", " print(\n", " f\"Running GP+EI optimization trial {i + NUM_SOBOL_TRIALS + 1}/{NUM_SOBOL_TRIALS + NUM_BOTORCH_TRIALS}...\"\n", " )\n", " # Reinitialize GP+EI model at each step with updated data.\n", " gpei = Models.BOTORCH(experiment=exp, data=exp.fetch_data())\n", " generator_run = gpei.gen(n=1)\n", " trial = exp.new_trial(generator_run=generator_run)\n", " trial.run()\n", " trial.mark_completed()\n", " \n", "print(\"Done!\")" ] }, { "cell_type": "markdown", "metadata": { "code_folding": [], "hidden_ranges": [], "originalKey": "f503e648-e3f2-419f-a60e-5bfcbc6775bd", "showInput": true }, "source": [ "## 6. Inspect trials' data\n", "\n", "Now we can inspect the `Experiment`'s data by calling `fetch_data()`, which retrieves evaluation data for all trials of the experiment.\n", "\n", "To fetch trial data, we need to run it and mark it completed. For most metrics in Ax, data is only available once the status of the trial is `COMPLETED`, since in real-worlds scenarios, metrics can typically only be fetched after the trial finished running.\n", "\n", "NOTE: Metrics classes may implement the `is_available_while_running` method. When this method returns `True`, data is available when trials are either `RUNNING` or `COMPLETED`. This can be used to obtain intermediate results from A/B test trials and other online experiments, or when metric values are available immediately, like in the case of synthetic problem metrics.\n", "\n", "We can also use the `fetch_trials_data` function to get evaluation data for a specific trials in the experiment, like so:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "code_folding": [], "collapsed": false, "execution": { "iopub.execute_input": "2022-11-10T20:31:57.568685Z", "iopub.status.busy": "2022-11-10T20:31:57.567849Z", "iopub.status.idle": "2022-11-10T20:31:57.602344Z", "shell.execute_reply": "2022-11-10T20:31:57.601759Z" }, "executionStartTime": 1646323714950, "executionStopTime": 1646323715210, "hidden_ranges": [], "originalKey": "65113ac3-956a-4fcc-abf7-a9d45f8ecb72", "requestMsgId": "65113ac3-956a-4fcc-abf7-a9d45f8ecb72" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
arm_namemetric_namemeansemtrial_indexnfrac_nonnull
019_0hartmann6-2.7189710.01910000-2.718971
119_0l2norm1.2414250.219100001.241425
\n", "
" ], "text/plain": [ " arm_name metric_name mean sem trial_index n frac_nonnull\n", "0 19_0 hartmann6 -2.718971 0.0 19 10000 -2.718971\n", "1 19_0 l2norm 1.241425 0.2 19 10000 1.241425" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "trial_data = exp.fetch_trials_data([NUM_SOBOL_TRIALS + NUM_BOTORCH_TRIALS - 1])\n", "trial_data.df" ] }, { "cell_type": "markdown", "metadata": { "code_folding": [], "customInput": null, "hidden_ranges": [], "originalKey": "74fa0cc5-075c-4b34-98b9-cbf74ad5bb26", "showInput": true }, "source": [ "The below call to `exp.fetch_data()` also attaches data to the last trial, which because of the way we looped through Botorch trials in [5. Perform Optimization](5.-Perform-Optimization), would otherwise not have data attached. This is necessary to get `objective_means` in [7. Plot results](7.-Plot-results)." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "code_folding": [], "collapsed": false, "customInput": null, "execution": { "iopub.execute_input": "2022-11-10T20:31:57.605735Z", "iopub.status.busy": "2022-11-10T20:31:57.604960Z", "iopub.status.idle": "2022-11-10T20:31:57.827720Z", "shell.execute_reply": "2022-11-10T20:31:57.826684Z" }, "executionStartTime": 1646323715232, "executionStopTime": 1646323715950, "hidden_ranges": [], "originalKey": "88fb1408-0965-48f9-a211-140ea57f46a6", "requestMsgId": "88fb1408-0965-48f9-a211-140ea57f46a6", "showInput": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
arm_namemetric_namemeansemtrial_indexnfrac_nonnull
00_0hartmann6-0.0000530.0010000-0.000053
10_0l2norm2.2221870.20100002.222187
21_0hartmann6-0.1249240.0110000-0.124924
31_0l2norm1.4874310.21100001.487431
42_0hartmann6-0.3785130.0210000-0.378513
52_0l2norm0.8395590.22100000.839559
63_0hartmann6-0.0267590.0310000-0.026759
73_0l2norm1.5405450.23100001.540545
84_0hartmann6-0.1204230.0410000-0.120423
94_0l2norm1.7796490.24100001.779649
105_0hartmann6-0.1834820.0510000-0.183482
115_0l2norm0.6128680.25100000.612868
126_0hartmann6-0.4912490.0610000-0.491249
136_0l2norm1.0127870.26100001.012787
147_0hartmann6-0.5721320.0710000-0.572132
157_0l2norm0.9146190.27100000.914619
168_0hartmann6-0.5419540.0810000-0.541954
178_0l2norm0.6537390.28100000.653739
189_0hartmann6-0.5346030.0910000-0.534603
199_0l2norm0.7814750.29100000.781475
2010_0hartmann6-0.6590690.01010000-0.659069
2110_0l2norm1.0867940.210100001.086794
2211_0hartmann6-0.7583330.01110000-0.758333
2311_0l2norm1.1698970.211100001.169897
2412_0hartmann6-0.6059260.01210000-0.605926
2512_0l2norm1.1446250.212100001.144625
2613_0hartmann6-0.9943580.01310000-0.994358
2713_0l2norm1.1411650.213100001.141165
2814_0hartmann6-1.5545690.01410000-1.554569
2914_0l2norm1.3476040.214100001.347604
3015_0hartmann6-1.9905530.01510000-1.990553
3115_0l2norm1.0927040.215100001.092704
3216_0hartmann6-1.6995810.01610000-1.699581
3316_0l2norm1.4744150.216100001.474415
3417_0hartmann6-2.3544530.01710000-2.354453
3517_0l2norm0.7610100.217100000.761010
3618_0hartmann6-2.8767370.01810000-2.876737
3718_0l2norm1.2846120.218100001.284612
3819_0hartmann6-2.7189710.01910000-2.718971
3919_0l2norm1.3094550.219100001.309455
\n", "
" ], "text/plain": [ " arm_name metric_name mean sem trial_index n frac_nonnull\n", "0 0_0 hartmann6 -0.000053 0.0 0 10000 -0.000053\n", "1 0_0 l2norm 2.222187 0.2 0 10000 2.222187\n", "2 1_0 hartmann6 -0.124924 0.0 1 10000 -0.124924\n", "3 1_0 l2norm 1.487431 0.2 1 10000 1.487431\n", "4 2_0 hartmann6 -0.378513 0.0 2 10000 -0.378513\n", "5 2_0 l2norm 0.839559 0.2 2 10000 0.839559\n", "6 3_0 hartmann6 -0.026759 0.0 3 10000 -0.026759\n", "7 3_0 l2norm 1.540545 0.2 3 10000 1.540545\n", "8 4_0 hartmann6 -0.120423 0.0 4 10000 -0.120423\n", "9 4_0 l2norm 1.779649 0.2 4 10000 1.779649\n", "10 5_0 hartmann6 -0.183482 0.0 5 10000 -0.183482\n", "11 5_0 l2norm 0.612868 0.2 5 10000 0.612868\n", "12 6_0 hartmann6 -0.491249 0.0 6 10000 -0.491249\n", "13 6_0 l2norm 1.012787 0.2 6 10000 1.012787\n", "14 7_0 hartmann6 -0.572132 0.0 7 10000 -0.572132\n", "15 7_0 l2norm 0.914619 0.2 7 10000 0.914619\n", "16 8_0 hartmann6 -0.541954 0.0 8 10000 -0.541954\n", "17 8_0 l2norm 0.653739 0.2 8 10000 0.653739\n", "18 9_0 hartmann6 -0.534603 0.0 9 10000 -0.534603\n", "19 9_0 l2norm 0.781475 0.2 9 10000 0.781475\n", "20 10_0 hartmann6 -0.659069 0.0 10 10000 -0.659069\n", "21 10_0 l2norm 1.086794 0.2 10 10000 1.086794\n", "22 11_0 hartmann6 -0.758333 0.0 11 10000 -0.758333\n", "23 11_0 l2norm 1.169897 0.2 11 10000 1.169897\n", "24 12_0 hartmann6 -0.605926 0.0 12 10000 -0.605926\n", "25 12_0 l2norm 1.144625 0.2 12 10000 1.144625\n", "26 13_0 hartmann6 -0.994358 0.0 13 10000 -0.994358\n", "27 13_0 l2norm 1.141165 0.2 13 10000 1.141165\n", "28 14_0 hartmann6 -1.554569 0.0 14 10000 -1.554569\n", "29 14_0 l2norm 1.347604 0.2 14 10000 1.347604\n", "30 15_0 hartmann6 -1.990553 0.0 15 10000 -1.990553\n", "31 15_0 l2norm 1.092704 0.2 15 10000 1.092704\n", "32 16_0 hartmann6 -1.699581 0.0 16 10000 -1.699581\n", "33 16_0 l2norm 1.474415 0.2 16 10000 1.474415\n", "34 17_0 hartmann6 -2.354453 0.0 17 10000 -2.354453\n", "35 17_0 l2norm 0.761010 0.2 17 10000 0.761010\n", "36 18_0 hartmann6 -2.876737 0.0 18 10000 -2.876737\n", "37 18_0 l2norm 1.284612 0.2 18 10000 1.284612\n", "38 19_0 hartmann6 -2.718971 0.0 19 10000 -2.718971\n", "39 19_0 l2norm 1.309455 0.2 19 10000 1.309455" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "exp.fetch_data().df" ] }, { "cell_type": "markdown", "metadata": { "code_folding": [], "hidden_ranges": [], "originalKey": "940865f9-af61-4668-aea0-b19ed5c5497d", "showInput": false }, "source": [ "## 7. Plot results\n", "Now we can plot the results of our optimization:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "code_folding": [], "collapsed": false, "execution": { "iopub.execute_input": "2022-11-10T20:31:57.832220Z", "iopub.status.busy": "2022-11-10T20:31:57.831098Z", "iopub.status.idle": "2022-11-10T20:31:58.128289Z", "shell.execute_reply": "2022-11-10T20:31:58.127150Z" }, "executionStartTime": 1646323715983, "executionStopTime": 1646323716634, "hidden_ranges": [], "originalKey": "5a4d2c4d-756a-492a-8938-d080a499b66c", "requestMsgId": "5a4d2c4d-756a-492a-8938-d080a499b66c" }, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "linkText": "Export to plot.ly", "plotlyServerURL": "https://plot.ly", "showLink": false }, "data": [ { "hoverinfo": "none", "legendgroup": "", "line": { "width": 0 }, "mode": "lines", "showlegend": false, "type": "scatter", "x": [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 ], "y": [ -5.319252145905967e-05, -0.12492399020593482, -0.37851275126100115, -0.37851275126100115, -0.37851275126100115, -0.37851275126100115, -0.4912491917162606, -0.5721322171674978, -0.5721322171674978, -0.5721322171674978, -0.6590689426459395, -0.7583328692417235, -0.7583328692417235, -0.9943582082838394, -1.5545691374509978, -1.9905529256853942, -1.9905529256853942, -2.354452608063184, -2.8767366204640115, -2.8767366204640115 ] }, { "fill": "tonexty", "fillcolor": "rgba(128,177,211,0.3)", "legendgroup": "objective value", "line": { "color": "rgba(128,177,211,1)" }, "mode": "lines", "name": "objective value", "type": "scatter", "x": [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 ], "y": [ -5.319252145905967e-05, -0.12492399020593482, -0.37851275126100115, -0.37851275126100115, -0.37851275126100115, -0.37851275126100115, -0.4912491917162606, -0.5721322171674978, -0.5721322171674978, -0.5721322171674978, -0.6590689426459395, -0.7583328692417235, -0.7583328692417235, -0.9943582082838394, -1.5545691374509978, -1.9905529256853942, -1.9905529256853942, -2.354452608063184, -2.8767366204640115, -2.8767366204640115 ] }, { "fill": "tonexty", "fillcolor": "rgba(128,177,211,0.3)", "hoverinfo": "none", "legendgroup": "", "line": { "width": 0 }, "mode": "lines", "showlegend": false, "type": "scatter", "x": [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 ], "y": [ -5.319252145905967e-05, -0.12492399020593482, -0.37851275126100115, -0.37851275126100115, -0.37851275126100115, -0.37851275126100115, -0.4912491917162606, -0.5721322171674978, -0.5721322171674978, -0.5721322171674978, -0.6590689426459395, -0.7583328692417235, -0.7583328692417235, -0.9943582082838394, -1.5545691374509978, -1.9905529256853942, -1.9905529256853942, -2.354452608063184, -2.8767366204640115, -2.8767366204640115 ] }, { "line": { "color": "rgba(253,180,98,1)", "dash": "dash" }, "mode": "lines", "name": "Optimum", "type": "scatter", "x": [ 1, 20 ], "y": [ -3.32237, -3.32237 ] } ], "layout": { "showlegend": true, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "fillpattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "sequentialminus": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "title": { "text": "" }, "xaxis": { "title": { "text": "Iteration" } }, "yaxis": { "title": { "text": "" } } } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import numpy as np\n", "from ax.plot.trace import optimization_trace_single_method\n", "\n", "# `plot_single_method` expects a 2-d array of means, because it expects to average means from multiple \n", "# optimization runs, so we wrap out best objectives array in another array.\n", "objective_means = np.array([[trial.objective_mean for trial in exp.trials.values()]])\n", "best_objective_plot = optimization_trace_single_method(\n", " y=np.minimum.accumulate(objective_means, axis=1),\n", " optimum=-3.32237, # Known minimum objective for Hartmann6 function.\n", ")\n", "render(best_objective_plot)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": { "code_folding": [], "customInput": null, "hidden_ranges": [], "originalKey": "934db3fd-1dce-421b-8228-820025f3821a", "showInput": true }, "source": [ "## 8. Defining custom metrics\n", "In order to perform an optimization, we also need to define an optimization config for the experiment. An optimization config is composed of an objective metric to be minimized or maximized in the experiment, and optionally a set of outcome constraints that place restrictions on how other metrics can be moved by the experiment.\n", "\n", "In order to define an objective or outcome constraint, we first need to subclass Metric. Metrics are used to evaluate trials, which are individual steps of the experiment sequence. Each trial contains one or more arms for which we will collect data at the same time.\n", "\n", "Our custom metric(s) will determine how, given a trial, to compute the mean and SEM of each of the trial's arms.\n", "\n", "The only method that needs to be defined for most metric subclasses is `fetch_trial_data`, which defines how a single trial is evaluated, and returns a pandas dataframe.\n", " \n", "The `is_available_while_running` method is optional and returns a boolean, specifying whether the trial data can be fetched before the trial is complete. See [6. Inspect trials' data](6.-Inspect-trials'-data) for more details." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "code_folding": [], "collapsed": false, "customInput": null, "execution": { "iopub.execute_input": "2022-11-10T20:31:58.132735Z", "iopub.status.busy": "2022-11-10T20:31:58.132047Z", "iopub.status.idle": "2022-11-10T20:31:58.138649Z", "shell.execute_reply": "2022-11-10T20:31:58.137978Z" }, "executionStartTime": 1646323716638, "executionStopTime": 1646323716697, "hidden_ranges": [], "originalKey": "7ec75ae4-1d7f-4ff4-8d9d-b77fdf28ccfe", "requestMsgId": "7ec75ae4-1d7f-4ff4-8d9d-b77fdf28ccfe", "showInput": true }, "outputs": [], "source": [ "from ax import Data\n", "import pandas as pd\n", "\n", "\n", "class BoothMetric(Metric):\n", " def fetch_trial_data(self, trial): \n", " records = []\n", " for arm_name, arm in trial.arms_by_name.items():\n", " params = arm.parameters\n", " records.append({\n", " \"arm_name\": arm_name,\n", " \"metric_name\": self.name,\n", " \"trial_index\": trial.index,\n", " # in practice, the mean and sem will be looked up based on trial metadata\n", " # but for this tutorial we will calculate them\n", " \"mean\": (params[\"x1\"] + 2*params[\"x2\"] - 7)**2 + (2*params[\"x1\"] + params[\"x2\"] - 5)**2,\n", " \"sem\": 0.0,\n", " })\n", " return Data(df=pd.DataFrame.from_records(records))\n", "\n", " def is_available_while_running(self) -> bool:\n", " return True" ] }, { "cell_type": "markdown", "metadata": { "code_folding": [], "customInput": null, "hidden_ranges": [], "originalKey": "92fcddf9-9d86-45cd-b9fb-a0a7acdb267d", "showInput": false }, "source": [ "## 9. Save to JSON or SQL\n", "At any point, we can also save our experiment to a JSON file. To ensure that our custom metrics and runner are saved properly, we first need to register them." ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "code_folding": [], "collapsed": false, "customInput": null, "execution": { "iopub.execute_input": "2022-11-10T20:31:58.141424Z", "iopub.status.busy": "2022-11-10T20:31:58.141013Z", "iopub.status.idle": "2022-11-10T20:31:58.345697Z", "shell.execute_reply": "2022-11-10T20:31:58.344667Z" }, "executionStartTime": 1646324682655, "executionStopTime": 1646324682796, "hidden_ranges": [], "originalKey": "f57e11d7-cc68-4323-a0cd-ff6f464dcd97", "requestMsgId": "f57e11d7-cc68-4323-a0cd-ff6f464dcd97", "showInput": true }, "outputs": [], "source": [ "from ax.storage.registry_bundle import RegistryBundle\n", "\n", "bundle = RegistryBundle(\n", " metric_clss={BoothMetric: None, L2NormMetric: None, Hartmann6Metric: None},\n", " runner_clss={MyRunner: None}\n", ")\n", "\n", "from ax.storage.json_store.load import load_experiment\n", "from ax.storage.json_store.save import save_experiment\n", "\n", "save_experiment(exp, \"experiment.json\", encoder_registry=bundle.encoder_registry)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "code_folding": [], "collapsed": false, "customInput": null, "execution": { "iopub.execute_input": "2022-11-10T20:31:58.350845Z", "iopub.status.busy": "2022-11-10T20:31:58.350304Z", "iopub.status.idle": "2022-11-10T20:31:59.458547Z", "shell.execute_reply": "2022-11-10T20:31:59.457568Z" }, "executionStartTime": 1646324718153, "executionStopTime": 1646324720104, "hidden_ranges": [], "originalKey": "e19ec7fb-f266-417e-ad17-5662a53a9ae3", "requestMsgId": "e19ec7fb-f266-417e-ad17-5662a53a9ae3", "showInput": true }, "outputs": [], "source": [ "loaded_experiment = load_experiment(\"experiment.json\", decoder_registry=bundle.decoder_registry)" ] }, { "cell_type": "markdown", "metadata": { "customInput": null, "originalKey": "dc1f6800-437e-45de-85d3-276ae5f8ca99", "showInput": false }, "source": [ "To save our experiment to SQL, we must first specify a connection to a database and create all necessary tables.\n", "\n" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "code_folding": [], "collapsed": false, "customInput": null, "execution": { "iopub.execute_input": "2022-11-10T20:31:59.463403Z", "iopub.status.busy": "2022-11-10T20:31:59.463053Z", "iopub.status.idle": "2022-11-10T20:31:59.501697Z", "shell.execute_reply": "2022-11-10T20:31:59.500991Z" }, "executionStartTime": 1646324834810, "executionStopTime": 1646324835293, "hidden_ranges": [], "originalKey": "a0376ade-9a26-430b-b08b-0b93e890539c", "requestMsgId": "a0376ade-9a26-430b-b08b-0b93e890539c", "showInput": true }, "outputs": [], "source": [ "from ax.storage.sqa_store.db import init_engine_and_session_factory, get_engine, create_all_tables\n", "from ax.storage.sqa_store.load import load_experiment\n", "from ax.storage.sqa_store.save import save_experiment\n", "\n", "init_engine_and_session_factory(url='sqlite:///foo3.db')\n", "\n", "engine = get_engine()\n", "create_all_tables(engine)" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "code_folding": [], "collapsed": false, "customInput": null, "execution": { "iopub.execute_input": "2022-11-10T20:31:59.505279Z", "iopub.status.busy": "2022-11-10T20:31:59.504393Z", "iopub.status.idle": "2022-11-10T20:32:02.221756Z", "shell.execute_reply": "2022-11-10T20:32:02.220183Z" }, "executionStartTime": 1646324891053, "executionStopTime": 1646324897271, "hidden_ranges": [], "originalKey": "82f58ead-8f0d-44cf-9fa8-dd67f7c8c8df", "requestMsgId": "82f58ead-8f0d-44cf-9fa8-dd67f7c8c8df", "showInput": true }, "outputs": [], "source": [ "from ax.storage.sqa_store.sqa_config import SQAConfig\n", "\n", "exp.name = \"new\"\n", "\n", "sqa_config = SQAConfig(\n", " json_encoder_registry=bundle.encoder_registry,\n", " json_decoder_registry=bundle.decoder_registry,\n", " metric_registry=bundle.metric_registry,\n", " runner_registry=bundle.runner_registry,\n", ")\n", "\n", "save_experiment(exp, config=sqa_config)" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "code_folding": [], "collapsed": false, "customInput": null, "execution": { "iopub.execute_input": "2022-11-10T20:32:02.226594Z", "iopub.status.busy": "2022-11-10T20:32:02.226068Z", "iopub.status.idle": "2022-11-10T20:32:03.922218Z", "shell.execute_reply": "2022-11-10T20:32:03.921087Z" }, "executionStartTime": 1646324904964, "executionStopTime": 1646324906901, "hidden_ranges": [], "originalKey": "ed1be69c-da92-4a1d-a5e8-e76bba42f0ba", "requestMsgId": "ed1be69c-da92-4a1d-a5e8-e76bba42f0ba", "showInput": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/home/runner/work/Ax/Ax/ax/storage/sqa_store/load.py:246: SAWarning:\n", "\n", "TypeDecorator JSONEncodedText() will not produce a cache key because the ``cache_ok`` attribute is not set to True. This can have significant performance implications including some performance degradations in comparison to prior SQLAlchemy versions. Set this attribute to True if this type object's state is safe to use in a cache key, or False to disable this warning. (Background on this error at: https://sqlalche.me/e/14/cprf)\n", "\n" ] }, { "data": { "text/plain": [ "Experiment(new)" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "load_experiment(exp.name, config=sqa_config)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "customInput": null, "originalKey": "d144e372-c212-4454-b507-564c825c1fc5", "showInput": true }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "python3", "language": "python", "metadata": null, "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.14" }, "last_base_url": "https://1619.od.fbinfra.net:443/", "last_kernel_id": "40cd7306-bfa1-451e-844e-64f818e20933", "last_msg_id": "54be6184-05a034649b7d6d1610bf2359_61", "last_server_session_id": "b4ee8410-839e-4375-8756-2cd91dd4c478", "outputWidgetContext": {} }, "nbformat": 4, "nbformat_minor": 2 }