{ "cells": [ { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "# Factorial design with empirical Bayes and Thompson Sampling" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "This tutorial illustrates how to run a factorial experiment. In such an experiment, each parameter (factor) can be assigned one of multiple discrete values (levels). A full-factorial experiment design explores all possible combinations of factors and levels.\n", "\n", "For instance, consider a banner with a title and an image. We are considering two different titles and three different images. A full-factorial experiment will compare all 2*3=6 possible combinations of title and image, to see which version of the banner performs the best.\n", "\n", "In this example, we first run an exploratory batch to collect data on all possible combinations. Then we use empirical Bayes to model the data and shrink noisy estimates toward the mean. Next, we use Thompson Sampling to suggest a set of arms (combinations of factors and levels) on which to collect more data. We repeat the process until we have identified the best performing combination(s)." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "ExecuteTime": { "end_time": "2019-04-01T16:59:07.844357Z", "start_time": "2019-04-01T09:59:06.377921-07:00" }, "collapsed": true }, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import sklearn as skl\n", "from typing import Dict, Optional, Tuple, Union\n", "from ax import Arm, ChoiceParameter, Models, ParameterType, SearchSpace, SimpleExperiment\n", "from ax.plot.scatter import plot_fitted\n", "from ax.utils.notebook.plotting import render, init_notebook_plotting\n", "from ax.utils.stats.statstools import agresti_coull_sem" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "ExecuteTime": { "end_time": "2019-04-01T16:59:07.852360Z", "start_time": "2019-04-01T09:59:07.846655-07:00" } }, "outputs": [ { "data": { "text/html": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 10-01 16:21:47] ax.utils.notebook.plotting: Injecting Plotly library into cell. Do not overwrite or delete cell.\n" ] }, { "data": { "text/html": [ " \n", " " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "init_notebook_plotting()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Define the search space" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "First, we define our search space. A factorial search space contains a ChoiceParameter for each factor, where the values of the parameter are its levels." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "ExecuteTime": { "end_time": "2019-04-01T16:59:07.861686Z", "start_time": "2019-04-01T09:59:07.854353-07:00" }, "collapsed": true }, "outputs": [], "source": [ "search_space = SearchSpace(\n", " parameters=[\n", " ChoiceParameter(\n", " name=\"factor1\",\n", " parameter_type=ParameterType.STRING,\n", " values=[\"level11\", \"level12\", \"level13\"],\n", " ),\n", " ChoiceParameter(\n", " name=\"factor2\",\n", " parameter_type=ParameterType.STRING,\n", " values=[\"level21\", \"level22\"],\n", " ),\n", " ChoiceParameter(\n", " name=\"factor3\",\n", " parameter_type=ParameterType.STRING,\n", " values=[\"level31\", \"level32\", \"level33\", \"level34\"],\n", " ),\n", " ]\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Define the evaluation function" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "Second, we define an evaluation function, which is responsible for computing\n", "the mean and standard error of a given arm.\n", "\n", "In this example, each possible parameter value is given a coefficient. The higher the level, the higher the coefficient, and the higher the coefficients, the greater the mean.\n", "\n", "The standard error of each arm is determined by the weight passed into the evaluation function, which represents the size of the population on which this arm was evaluated. The higher the weight, the greater the sample size, and thus the lower the standard error." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "ExecuteTime": { "end_time": "2019-04-01T16:59:07.871141Z", "start_time": "2019-04-01T09:59:07.863475-07:00" }, "collapsed": true }, "outputs": [], "source": [ "one_hot_encoder = skl.preprocessing.OneHotEncoder(\n", " categories=[par.values for par in search_space.parameters.values()], \n", ")\n", "\n", "def factorial_evaluation_function(\n", " # `parameterization` is a dict of parameter names to values of those parameters.\n", " parameterization: Dict[str, Optional[Union[str, bool, float]]],\n", " # `weight` is the weight of the parameterization, \n", " # which is used to determine the variance of the estimate.\n", " weight: Optional[float] = None,\n", ") -> Dict[str, Tuple[float, float]]: # Mapping of metric names to tuple of mean and standard error.\n", " batch_size = 10000\n", " noise_level = 0.0\n", " weight = weight if weight is not None else 1.0\n", " coefficients = np.array([\n", " 0.1, 0.2, 0.3,\n", " 0.1, 0.2,\n", " 0.1, 0.2, 0.3, 0.4\n", " ])\n", " features = np.array(list(parameterization.values())).reshape(1, -1)\n", " encoded_features = one_hot_encoder.fit_transform(features)\n", " z = coefficients @ encoded_features.T + np.sqrt(noise_level) * np.random.randn()\n", " p = np.exp(z) / (1 + np.exp(z))\n", " plays = np.random.binomial(batch_size, weight)\n", " successes = np.random.binomial(plays, p)\n", " mean = float(successes) / plays\n", " sem = agresti_coull_sem(successes, plays)\n", " return {\n", " \"success_metric\": (mean, sem)\n", " }" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Define the experiment" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "We now set up our experiment and define the status quo arm, in which each parameter is assigned to the lowest level." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "ExecuteTime": { "end_time": "2019-04-01T16:59:07.876425Z", "start_time": "2019-04-01T09:59:07.872766-07:00" }, "collapsed": true }, "outputs": [], "source": [ "exp = SimpleExperiment(\n", " name=\"my_factorial_closed_loop_experiment\",\n", " search_space=search_space,\n", " evaluation_function=factorial_evaluation_function,\n", " objective_name=\"success_metric\",\n", ")\n", "exp.status_quo = Arm(\n", " parameters={\"factor1\": \"level11\", \"factor2\": \"level21\", \"factor3\": \"level31\"}\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Run an exploratory batch" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "We then generate an a set of arms that covers the full space of the factorial design, including the status quo. There are three parameters, with two, three, and four values, respectively, so there are 24 possible arms." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "24\n" ] } ], "source": [ "factorial = Models.FACTORIAL(search_space=exp.search_space)\n", "factorial_run = factorial.gen(n=-1) # Number of arms to generate is derived from the search space.\n", "print(len(factorial_run.arms))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we create a trial including all of these arms, so that we can collect data and evaluate the performance of each." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": true }, "outputs": [], "source": [ "trial = (\n", " exp.new_batch_trial(optimize_for_power=True)\n", " .add_generator_run(factorial_run, multiplier=1)\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By default, the weight of each arm in `factorial_run` will be 1. However, to optimize for power on the contrasts of `k` groups against the status quo, the status quo should be `sqrt(k)` larger than any of the treatment groups. Since we have 24 different arms in our search space, the status quo should be roughly five times larger. That larger weight is automatically set by Ax under the hood if `optimize_for_power` kwarg is set to True on new batched trial creation." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "4.795831523312719" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "trial._status_quo_weight_override" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Iterate using Thompson Sampling" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "Next, we run multiple trials (iterations of the experiment) to hone in on the optimal arm(s). \n", "\n", "In each iteration, we first collect data about all arms in that trial by calling `exp.eval_trial()`. Then we run Thompson Sampling, which assigns a weight to each arm that is proportional to the probability of that arm being the best. Arms whose weight exceed `min_weight` are added to the next trial, so that we can gather more data on their performance." ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "ExecuteTime": { "end_time": "2019-04-01T16:59:08.480646Z", "start_time": "2019-04-01T09:59:07.908822-07:00" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Running trial 1...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running trial 2...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running trial 3...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running trial 4...\n" ] } ], "source": [ "models = []\n", "for i in range(4):\n", " print(f\"Running trial {i+1}...\")\n", " data = exp.eval_trial(trial)\n", " thompson = Models.THOMPSON(\n", " experiment=exp, data=data, min_weight=0.01\n", " )\n", " models.append(thompson)\n", " thompson_run = thompson.gen(n=-1)\n", " trial = exp.new_batch_trial(optimize_for_power=True).add_generator_run(thompson_run)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Plot 1: Predicted outcomes for each arm in initial trial" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "The plot below shows the mean and standard error for each arm in the first trial. We can see that the standard error for the status quo is the smallest, since this arm was assigned 5x weight." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "ExecuteTime": { "end_time": "2019-04-01T16:59:08.534814Z", "start_time": "2019-04-01T09:59:08.482576-07:00" }, "scrolled": false }, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "linkText": "Export to plot.ly", "plotlyServerURL": "https://plot.ly", "showLink": false }, "data": [ { "error_y": { "array": [ 0.05154247283575067, 0.04930063520902043, 0.04790452560645175, 0.05205959642958938, 0.04937502522644063, 0.048435137164434726, 0.04839263532809516, 0.04874081832726382, 0.046720391081045004, 0.04767360839632559, 0.04713023430482041, 0.04949338337313541, 0.04855053168709971, 0.048315111883455476, 0.047035061512192215, 0.04672676618257923, 0.049309778559697225, 0.0507877959325867, 0.053246647004775334, 0.04683293455062465, 0.05135224796234841, 0.04907046881754487, 0.05068515600321663, 0.02338764872246525 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "hoverinfo": "text", "legendgroup": "In-sample", "marker": { "color": "rgba(128,177,211,1)" }, "mode": "markers", "name": "In-sample", "showlegend": true, "text": [ "Arm 0_0

success_metric: 0.645 [0.593, 0.696]

Parameterization:
factor1: level11
factor2: level21
factor3: level32", "Arm 0_1

success_metric: 0.635 [0.586, 0.684]

Parameterization:
factor1: level11
factor2: level21
factor3: level33", "Arm 0_10

success_metric: 0.649 [0.601, 0.697]

Parameterization:
factor1: level12
factor2: level21
factor3: level34", "Arm 0_11

success_metric: 0.598 [0.546, 0.650]

Parameterization:
factor1: level12
factor2: level22
factor3: level31", "Arm 0_12

success_metric: 0.661 [0.612, 0.710]

Parameterization:
factor1: level12
factor2: level22
factor3: level32", "Arm 0_13

success_metric: 0.683 [0.634, 0.731]

Parameterization:
factor1: level12
factor2: level22
factor3: level33", "Arm 0_14

success_metric: 0.69 [0.642, 0.739]

Parameterization:
factor1: level12
factor2: level22
factor3: level34", "Arm 0_15

success_metric: 0.658 [0.609, 0.706]

Parameterization:
factor1: level13
factor2: level21
factor3: level31", "Arm 0_16

success_metric: 0.654 [0.607, 0.701]

Parameterization:
factor1: level13
factor2: level21
factor3: level32", "Arm 0_17

success_metric: 0.677 [0.629, 0.724]

Parameterization:
factor1: level13
factor2: level21
factor3: level33", "Arm 0_18

success_metric: 0.702 [0.655, 0.750]

Parameterization:
factor1: level13
factor2: level21
factor3: level34", "Arm 0_19

success_metric: 0.671 [0.622, 0.721]

Parameterization:
factor1: level13
factor2: level22
factor3: level31", "Arm 0_2

success_metric: 0.661 [0.613, 0.710]

Parameterization:
factor1: level11
factor2: level21
factor3: level34", "Arm 0_20

success_metric: 0.652 [0.604, 0.701]

Parameterization:
factor1: level13
factor2: level22
factor3: level32", "Arm 0_21

success_metric: 0.71 [0.663, 0.757]

Parameterization:
factor1: level13
factor2: level22
factor3: level33", "Arm 0_22

success_metric: 0.706 [0.659, 0.752]

Parameterization:
factor1: level13
factor2: level22
factor3: level34", "Arm 0_3

success_metric: 0.592 [0.542, 0.641]

Parameterization:
factor1: level11
factor2: level22
factor3: level31", "Arm 0_4

success_metric: 0.64 [0.589, 0.690]

Parameterization:
factor1: level11
factor2: level22
factor3: level32", "Arm 0_5

success_metric: 0.598 [0.545, 0.651]

Parameterization:
factor1: level11
factor2: level22
factor3: level33", "Arm 0_6

success_metric: 0.679 [0.632, 0.726]

Parameterization:
factor1: level11
factor2: level22
factor3: level34", "Arm 0_7

success_metric: 0.576 [0.524, 0.627]

Parameterization:
factor1: level12
factor2: level21
factor3: level31", "Arm 0_8

success_metric: 0.666 [0.617, 0.715]

Parameterization:
factor1: level12
factor2: level21
factor3: level32", "Arm 0_9

success_metric: 0.63 [0.580, 0.681]

Parameterization:
factor1: level12
factor2: level21
factor3: level33", "Arm status_quo

success_metric: 0.563 [0.540, 0.586]

Parameterization:
factor1: level11
factor2: level21
factor3: level31" ], "type": "scatter", "visible": true, "x": [ "0_0", "0_1", "0_10", "0_11", "0_12", "0_13", "0_14", "0_15", "0_16", "0_17", "0_18", "0_19", "0_2", "0_20", "0_21", "0_22", "0_3", "0_4", "0_5", "0_6", "0_7", "0_8", "0_9", "status_quo" ], "y": [ 0.6445783132530121, 0.6348773841961853, 0.6492146596858639, 0.5982404692082112, 0.6610169491525424, 0.6825842696629213, 0.6903409090909091, 0.6575342465753424, 0.6541353383458647, 0.6765498652291105, 0.7024793388429752, 0.6714697406340058, 0.6612021857923497, 0.6524064171122995, 0.7103064066852368, 0.7057220708446866, 0.5916230366492147, 0.6395348837209303, 0.598159509202454, 0.6788511749347258, 0.5758426966292135, 0.6657303370786517, 0.6303724928366762, 0.5630787037037037 ] } ], "layout": { "annotations": [ { "showarrow": false, "text": "Sort By", "x": 1.18, "xref": "paper", "y": 0.72, "yanchor": "middle", "yref": "paper" } ], "font": { "size": 10 }, "hovermode": "closest", "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "sequentialminus": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "title": { "text": "Predicted Outcomes" }, "updatemenus": [ { "buttons": [ { "args": [ { "xaxis.categoryarray": [ "status_quo", "0_0", "0_1", "0_2", "0_3", "0_4", "0_5", "0_6", "0_7", "0_8", "0_9", "0_10", "0_11", "0_12", "0_13", "0_14", "0_15", "0_16", "0_17", "0_18", "0_19", "0_20", "0_21", "0_22" ], "xaxis.categoryorder": "array" } ], "label": "Name", "method": "relayout" }, { "args": [ { "xaxis.categoryarray": [ "status_quo", "0_7", "0_3", "0_5", "0_11", "0_9", "0_1", "0_4", "0_0", "0_10", "0_20", "0_16", "0_15", "0_12", "0_2", "0_8", "0_19", "0_17", "0_6", "0_13", "0_14", "0_18", "0_22", "0_21" ], "xaxis.categoryorder": "array" } ], "label": "Effect Size", "method": "relayout" } ], "x": 1.25, "xanchor": "left", "y": 0.67, "yanchor": "middle" } ], "xaxis": { "categoryarray": [ "status_quo", "0_0", "0_1", "0_2", "0_3", "0_4", "0_5", "0_6", "0_7", "0_8", "0_9", "0_10", "0_11", "0_12", "0_13", "0_14", "0_15", "0_16", "0_17", "0_18", "0_19", "0_20", "0_21", "0_22" ], "categoryorder": "array", "tickangle": 45 }, "yaxis": { "title": { "text": "success_metric" }, "zerolinecolor": "red" } } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "render(plot_fitted(models[0], metric=\"success_metric\", rel=False))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Plot 2: Predicted outcomes for arms in last trial" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The following plot below shows the mean and standard error for each arm that made it to the last trial (as well as the status quo, which appears throughout). " ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "linkText": "Export to plot.ly", "plotlyServerURL": "https://plot.ly", "showLink": false }, "data": [ { "error_y": { "array": [ 0.02057800388584073, 0.05193654918868695, 0.07075473879769907, 0.017393595952410478, 0.013843355759376837 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "hoverinfo": "text", "legendgroup": "In-sample", "marker": { "color": "rgba(128,177,211,1)" }, "mode": "markers", "name": "In-sample", "showlegend": true, "text": [ "Arm 0_14

success_metric: 0.704 [0.683, 0.725]

Parameterization:
factor1: level12
factor2: level22
factor3: level34", "Arm 0_18

success_metric: 0.692 [0.640, 0.744]

Parameterization:
factor1: level13
factor2: level21
factor3: level34", "Arm 0_21

success_metric: 0.695 [0.624, 0.766]

Parameterization:
factor1: level13
factor2: level22
factor3: level33", "Arm 0_22

success_metric: 0.698 [0.681, 0.716]

Parameterization:
factor1: level13
factor2: level22
factor3: level34", "Arm status_quo

success_metric: 0.564 [0.550, 0.578]

Parameterization:
factor1: level11
factor2: level21
factor3: level31" ], "type": "scatter", "visible": true, "x": [ "0_14", "0_18", "0_21", "0_22", "status_quo" ], "y": [ 0.7040169133192389, 0.6918032786885245, 0.6951219512195121, 0.6984304932735426, 0.5642118076688983 ] } ], "layout": { "annotations": [ { "showarrow": false, "text": "Sort By", "x": 1.18, "xref": "paper", "y": 0.72, "yanchor": "middle", "yref": "paper" } ], "font": { "size": 10 }, "hovermode": "closest", "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "sequentialminus": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "title": { "text": "Predicted Outcomes" }, "updatemenus": [ { "buttons": [ { "args": [ { "xaxis.categoryarray": [ "status_quo", "0_14", "0_18", "0_21", "0_22" ], "xaxis.categoryorder": "array" } ], "label": "Name", "method": "relayout" }, { "args": [ { "xaxis.categoryarray": [ "status_quo", "0_18", "0_21", "0_22", "0_14" ], "xaxis.categoryorder": "array" } ], "label": "Effect Size", "method": "relayout" } ], "x": 1.25, "xanchor": "left", "y": 0.67, "yanchor": "middle" } ], "xaxis": { "categoryarray": [ "status_quo", "0_14", "0_18", "0_21", "0_22" ], "categoryorder": "array", "tickangle": 45 }, "yaxis": { "title": { "text": "success_metric" }, "zerolinecolor": "red" } } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "render(plot_fitted(models[-1], metric=\"success_metric\", rel=False))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "As expected given our evaluation function, arms with higher levels\n", "perform better and are given higher weight. Below we see the arms\n", "that made it to the final trial." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "ExecuteTime": { "end_time": "2019-04-01T16:59:08.548754Z", "start_time": "2019-04-01T09:59:08.536758-07:00" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " values weight\n", "0 level13,level22,level33 0.1663\n", "1 level12,level22,level34 0.1570\n", "2 level13,level21,level34 0.1080\n", "3 level13,level22,level34 0.0687\n", "4 level11,level21,level31 0.5000\n" ] } ], "source": [ "results = pd.DataFrame(\n", " [\n", " {\"values\": \",\".join(arm.parameters.values()), \"weight\": weight}\n", " for arm, weight in trial.normalized_arm_weights().items()\n", " ]\n", ")\n", "print(results)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Plot 3: Rollout Process" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also visualize the progression of the experience in the following rollout chart. Each bar represents a trial, and the width of the bands within a bar are proportional to the weight of the arms in that trial. \n", "\n", "In the first trial, all arms appear with equal weight, except for the status quo. By the last trial, we have narrowed our focus to only four arms, with arm 0_22 (the arm with the highest levels) having the greatest weight." ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "ExecuteTime": { "end_time": "2019-04-01T16:59:08.569844Z", "start_time": "2019-04-01T09:59:08.550440-07:00" } }, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "linkText": "Export to plot.ly", "plotlyServerURL": "https://plot.ly", "showLink": false }, "data": [ { "hoverinfo": "name+text", "marker": { "color": "rgba(2,63,165,1)" }, "name": "status_quo", "text": [ "17.25%", "50.00%", "50.00%", "50.00%", "50.00%" ], "type": "bar", "width": 0.5, "x": [ "Round 0", "Round 1", "Round 2", "Round 3", "Round 4" ], "y": [ 17.25377965142145, 49.99999999999999, 50.0, 50.0, 50.0 ] }, { "hoverinfo": "name+text", "marker": { "color": "rgba(125,135,185,1)" }, "name": "0_0", "text": [ "3.60%" ], "type": "bar", "width": 0.5, "x": [ "Round 0" ], "y": [ 3.5976617542860234 ] }, { "hoverinfo": "name+text", "marker": { "color": "rgba(190,193,212,1)" }, "name": "0_1", "text": [ "3.60%" ], "type": "bar", "width": 0.5, "x": [ "Round 0" ], "y": [ 3.5976617542860234 ] }, { "hoverinfo": "name+text", "marker": { "color": "rgba(214,188,192,1)" }, "name": "0_2", "text": [ "3.60%" ], "type": "bar", "width": 0.5, "x": [ "Round 0" ], "y": [ 3.5976617542860234 ] }, { "hoverinfo": "name+text", "marker": { "color": "rgba(187,119,132,1)" }, "name": "0_3", "text": [ "3.60%" ], "type": "bar", "width": 0.5, "x": [ "Round 0" ], "y": [ 3.5976617542860234 ] }, { "hoverinfo": "name+text", "marker": { "color": "rgba(142,6,59,1)" }, "name": "0_4", "text": [ "3.60%" ], "type": "bar", "width": 0.5, "x": [ "Round 0" ], "y": [ 3.5976617542860234 ] }, { "hoverinfo": "name+text", "marker": { "color": "rgba(74,111,227,1)" }, "name": "0_5", "text": [ "3.60%" ], "type": "bar", "width": 0.5, "x": [ "Round 0" ], "y": [ 3.5976617542860234 ] }, { "hoverinfo": "name+text", "marker": { "color": "rgba(133,149,225,1)" }, "name": "0_6", "text": [ "3.60%", "1.83%" ], "type": "bar", "width": 0.5, "x": [ "Round 0", "Round 1" ], "y": [ 3.5976617542860234, 1.8269330297070694 ] }, { "hoverinfo": "name+text", "marker": { "color": "rgba(181,187,227,1)" }, "name": "0_7", "text": [ "3.60%" ], "type": "bar", "width": 0.5, "x": [ "Round 0" ], "y": [ 3.5976617542860234 ] }, { "hoverinfo": "name+text", "marker": { "color": "rgba(230,175,185,1)" }, "name": "0_8", "text": [ "3.60%", "0.67%", "3.28%" ], "type": "bar", "width": 0.5, "x": [ "Round 0", "Round 1", "Round 2" ], "y": [ 3.5976617542860234, 0.6676327502328951, 3.27991987981973 ] }, { "hoverinfo": "name+text", "marker": { "color": "rgba(224,123,145,1)" }, "name": "0_9", "text": [ "3.60%" ], "type": "bar", "width": 0.5, "x": [ "Round 0" ], "y": [ 3.5976617542860234 ] }, { "hoverinfo": "name+text", "marker": { "color": "rgba(211,63,106,1)" }, "name": "0_10", "text": [ "3.60%" ], "type": "bar", "width": 0.5, "x": [ "Round 0" ], "y": [ 3.5976617542860234 ] }, { "hoverinfo": "name+text", "marker": { "color": "rgba(17,198,56,1)" }, "name": "0_11", "text": [ "3.60%" ], "type": "bar", "width": 0.5, "x": [ "Round 0" ], "y": [ 3.5976617542860234 ] }, { "hoverinfo": "name+text", "marker": { "color": "rgba(141,213,147,1)" }, "name": "0_12", "text": [ "3.60%" ], "type": "bar", "width": 0.5, "x": [ "Round 0" ], "y": [ 3.5976617542860234 ] }, { "hoverinfo": "name+text", "marker": { "color": "rgba(198,222,199,1)" }, "name": "0_13", "text": [ "3.60%", "2.57%", "20.78%" ], "type": "bar", "width": 0.5, "x": [ "Round 0", "Round 1", "Round 2" ], "y": [ 3.5976617542860234, 2.5721974950833246, 20.776164246369554 ] }, { "hoverinfo": "name+text", "marker": { "color": "rgba(234,211,198,1)" }, "name": "0_14", "text": [ "3.60%", "4.72%", "8.59%", "18.14%", "15.70%" ], "type": "bar", "width": 0.5, "x": [ "Round 0", "Round 1", "Round 2", "Round 3", "Round 4" ], "y": [ 3.5976617542860234, 4.720008280716282, 8.587881822734102, 18.143821130023166, 15.7 ] }, { "hoverinfo": "name+text", "marker": { "color": "rgba(240,185,141,1)" }, "name": "0_15", "text": [ "3.60%" ], "type": "bar", "width": 0.5, "x": [ "Round 0" ], "y": [ 3.5976617542860234 ] }, { "hoverinfo": "name+text", "marker": { "color": "rgba(239,151,8,1)" }, "name": "0_16", "text": [ "3.60%" ], "type": "bar", "width": 0.5, "x": [ "Round 0" ], "y": [ 3.5976617542860234 ] }, { "hoverinfo": "name+text", "marker": { "color": "rgba(15,207,192,1)" }, "name": "0_17", "text": [ "3.60%", "1.47%" ], "type": "bar", "width": 0.5, "x": [ "Round 0", "Round 1" ], "y": [ 3.5976617542860234, 1.4698271400476142 ] }, { "hoverinfo": "name+text", "marker": { "color": "rgba(156,222,214,1)" }, "name": "0_18", "text": [ "3.60%", "9.97%", "1.51%", "3.26%", "10.80%" ], "type": "bar", "width": 0.5, "x": [ "Round 0", "Round 1", "Round 2", "Round 3", "Round 4" ], "y": [ 3.5976617542860234, 9.967912224407412, 1.5122684026039062, 3.258132742471548, 10.8 ] }, { "hoverinfo": "name+text", "marker": { "color": "rgba(213,234,231,1)" }, "name": "0_19", "text": [ "3.60%", "1.12%", "4.01%" ], "type": "bar", "width": 0.5, "x": [ "Round 0", "Round 1", "Round 2" ], "y": [ 3.5976617542860234, 1.1178966980643827, 4.006009013520281 ] }, { "hoverinfo": "name+text", "marker": { "color": "rgba(243,225,235,1)" }, "name": "0_20", "text": [ "3.60%" ], "type": "bar", "width": 0.5, "x": [ "Round 0" ], "y": [ 3.5976617542860234 ] }, { "hoverinfo": "name+text", "marker": { "color": "rgba(246,196,225,1)" }, "name": "0_21", "text": [ "3.60%", "15.78%", "5.04%", "1.54%", "16.63%" ], "type": "bar", "width": 0.5, "x": [ "Round 0", "Round 1", "Round 2", "Round 3", "Round 4" ], "y": [ 3.5976617542860234, 15.779939964806958, 5.042563845768653, 1.5409406788196194, 16.63 ] }, { "hoverinfo": "name+text", "marker": { "color": "rgba(247,156,212,1)" }, "name": "0_22", "text": [ "3.60%", "11.88%", "6.80%", "27.06%", "6.87%" ], "type": "bar", "width": 0.5, "x": [ "Round 0", "Round 1", "Round 2", "Round 3", "Round 4" ], "y": [ 3.5976617542860234, 11.877652416934065, 6.795192789183775, 27.057105448685668, 6.87 ] } ], "layout": { "barmode": "stack", "margin": { "r": 40 }, "showlegend": false, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "sequentialminus": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "title": { "text": "Rollout Process
Bandit Weight Graph" }, "xaxis": { "categoryarray": [ "Round 0", "Round 1", "Round 2", "Round 3", "Round 4" ], "categoryorder": "array", "title": { "text": "Rounds" }, "zeroline": false }, "yaxis": { "showline": false, "title": { "text": "Percent" } } } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from ax.plot.bandit_rollout import plot_bandit_rollout\n", "from ax.utils.notebook.plotting import render\n", "\n", "render(plot_bandit_rollout(exp))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Plot 4: Marginal Effects" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, we can examine which parameter values had the greatest effect on the overall arm value. As we see in the diagram below, arms whose parameters were assigned the lower level values (such as `levell1`, `levell2`, `level31` and `level32`) performed worse than average, whereas arms with higher levels performed better than average." ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "ExecuteTime": { "end_time": "2019-04-01T17:03:56.645223Z", "start_time": "2019-04-01T10:03:56.563655-07:00" } }, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "linkText": "Export to plot.ly", "plotlyServerURL": "https://plot.ly", "showLink": false }, "data": [ { "error_y": { "array": [ 1.167979003812773, 1.3963353150901396, 1.342906986629675 ], "type": "data" }, "name": "factor1", "type": "bar", "x": [ "level11", "level12", "level13" ], "xaxis": "x", "y": [ -5.123443101200899, 0.8195550883922715, 6.015007538455098 ], "yaxis": "y" }, { "error_y": { "array": [ 0.9930932353864158, 1.1276078130532206 ], "type": "data" }, "name": "factor2", "type": "bar", "x": [ "level21", "level22" ], "xaxis": "x2", "y": [ -2.2355182215534053, 2.8821397000885445 ], "yaxis": "y2" }, { "error_y": { "array": [ 1.2888754484509461, 1.600214519550456, 1.6016495910168187, 1.5464880030754977 ], "type": "data" }, "name": "factor3", "type": "bar", "x": [ "level31", "level32", "level33", "level34" ], "xaxis": "x3", "y": [ -7.477148378810608, 1.9417266571436331, 2.7511690320551305, 6.386380208202293 ], "yaxis": "y3" } ], "layout": { "annotations": [ { "font": { "size": 16 }, "showarrow": false, "text": "factor1", "x": 0.14444444444444446, "xanchor": "center", "xref": "paper", "y": 1.0, "yanchor": "bottom", "yref": "paper" }, { "font": { "size": 16 }, "showarrow": false, "text": "factor2", "x": 0.5, "xanchor": "center", "xref": "paper", "y": 1.0, "yanchor": "bottom", "yref": "paper" }, { "font": { "size": 16 }, "showarrow": false, "text": "factor3", "x": 0.8555555555555556, "xanchor": "center", "xref": "paper", "y": 1.0, "yanchor": "bottom", "yref": "paper" } ], "showlegend": false, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "sequentialminus": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "title": { "text": "Marginal Effects by Factor" }, "xaxis": { "anchor": "y", "domain": [ 0.0, 0.2888888888888889 ] }, "xaxis2": { "anchor": "y2", "domain": [ 0.35555555555555557, 0.6444444444444445 ] }, "xaxis3": { "anchor": "y3", "domain": [ 0.7111111111111111, 1.0 ] }, "yaxis": { "hoverformat": ".3f", "title": { "text": "% better than experiment average" } }, "yaxis2": { "anchor": "x2", "domain": [ 0.0, 1.0 ], "matches": "y", "showticklabels": false }, "yaxis3": { "anchor": "x3", "domain": [ 0.0, 1.0 ], "matches": "y", "showticklabels": false } } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from ax.plot.marginal_effects import plot_marginal_effects\n", "render(plot_marginal_effects(models[0], 'success_metric'))" ] } ], "metadata": { "kernelspec": { "display_name": "python3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.1" } }, "nbformat": 4, "nbformat_minor": 2 }