{ "cells": [ { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "# Using Ax for Human-in-the-loop Experimentation¶" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "While Ax can be used in as a fully automated service, generating and deploying candidates Ax can be also used in a trial-by-trial fashion, allowing for human oversight. \n", "\n", "Typically, human intervention in Ax is necessary when there are clear tradeoffs between multiple metrics of interest. Condensing multiple outcomes of interest into a single scalar quantity can be really challenging. Instead, it can be useful to specify an objective and constraints, and tweak these based on the information from the experiment. \n", "\n", "To facilitate this, Ax provides the following key features:\n", "\n", "1. Constrained optimization\n", "2. Interfaces for easily modifying optimization goals\n", "3. Utilities for visualizing and deploying new trials composed of multiple optimizations. \n", "\n", "\n", "In this tutorial, we'll demonstrate how Ax enables users to explore these tradeoffs. With an understanding of the tradeoffs present in our data, we'll then make use of the constrained optimization utilities to generate candidates from multiple different optimization objectives, and create a conglomerate batch, with all of these candidates in together in one trial. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Experiment Setup\n", "\n", "For this tutorial, we will assume our experiment has already been created." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 11-04 19:17:50] ax.utils.notebook.plotting: Injecting Plotly library into cell. Do not overwrite or delete cell.\n" ] }, { "data": { "text/html": [ " \n", " " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from ax import Data, Metric, OptimizationConfig, Objective, OutcomeConstraint, ComparisonOp, load\n", "from ax.modelbridge.cross_validation import cross_validate\n", "from ax.modelbridge.factory import get_GPEI\n", "from ax.plot.diagnostic import tile_cross_validation\n", "from ax.plot.scatter import plot_multiple_metrics, tile_fitted\n", "from ax.utils.notebook.plotting import render, init_notebook_plotting\n", "\n", "import pandas as pd\n", "\n", "init_notebook_plotting()" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "experiment = load('hitl_exp.json')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Initial Sobol Trial" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Bayesian Optimization experiments almost always begin with a set of random points. In this experiment, these points were chosen via a Sobol sequence, accessible via the `ModelBridge` factory.\n", "\n", "A collection of points run and analyzed together form a `BatchTrial`. A `Trial` object provides metadata pertaining to the deployment of these points, including details such as when they were deployed, and the current status of their experiment. \n", "\n", "Here, we see an initial experiment has finished running (COMPLETED status)." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "BatchTrial(experiment_name='human_in_the_loop_tutorial', index=0, status=TrialStatus.COMPLETED)" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "experiment.trials[0]" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "datetime.datetime(2019, 3, 29, 18, 10, 6)" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "experiment.trials[0].time_created" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "65" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Number of arms in first experiment, including status_quo\n", "len(experiment.trials[0].arms)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "Arm(name='0_0', parameters={'x_excellent': 0.9715802669525146, 'x_good': 0.8615524768829346, 'x_moderate': 0.7668091654777527, 'x_poor': 0.34871453046798706, 'x_unknown': 0.7675797343254089, 'y_excellent': 2.900710028409958, 'y_good': 1.5137152910232545, 'y_moderate': 0.6775947093963622, 'y_poor': 0.4974367544054985, 'y_unknown': 1.0852564811706542, 'z_excellent': 517803.49761247635, 'z_good': 607874.5171427727, 'z_moderate': 1151881.2023103237, 'z_poor': 2927449.2621421814, 'z_unknown': 2068407.6935052872})" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Sample arm configuration\n", "experiment.trials[0].arms[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Experiment Analysis\n", "\n", "**Optimization Config**\n", "\n", "An important construct for analyzing an experiment is an OptimizationConfig. An OptimizationConfig contains an objective, and outcome constraints. Experiment's can have a default OptimizationConfig, but models can also take an OptimizationConfig as input independent of the default.\n", "\n", "**Objective:** A metric to optimize, along with a direction to optimize (default: maximize)\n", "\n", "**Outcome Constraint:** A metric to constrain, along with a constraint direction (<= or >=), as well as a bound. \n", "\n", "Let's start with a simple OptimizationConfig. By default, our objective metric will be maximized, but can be minimized by setting the `minimize` flag. Our outcome constraint will, by default, be evaluated as a relative percentage change. This percentage change is computed relative to the experiment's status quo arm. " ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Arm(name='status_quo', parameters={'x_excellent': 0, 'x_good': 0, 'x_moderate': 0, 'x_poor': 0, 'x_unknown': 0, 'y_excellent': 1, 'y_good': 1, 'y_moderate': 1, 'y_poor': 1, 'y_unknown': 1, 'z_excellent': 1000000, 'z_good': 1000000, 'z_moderate': 1000000, 'z_poor': 1000000, 'z_unknown': 1000000})" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "experiment.status_quo" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "collapsed": true }, "outputs": [], "source": [ "objective_metric = Metric(name=\"metric_1\")\n", "constraint_metric = Metric(name=\"metric_2\")\n", "\n", "experiment.optimization_config = OptimizationConfig(\n", " objective=Objective(objective_metric),\n", " outcome_constraints=[\n", " OutcomeConstraint(metric=constraint_metric, op=ComparisonOp.LEQ, bound=5),\n", " ]\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Data**\n", "\n", "Another critical piece of analysis is data itself! Ax data follows a standard format, shown below. This format is imposed upon the underlying data structure, which is a Pandas DataFrame. \n", "\n", "A key set of fields are required for all data, for use with Ax models. \n", "\n", "It's a good idea to double check our data before fitting models -- let's make sure all of our expected metrics and arms are present." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
arm_namemetric_namemeansemtrial_indexstart_timeend_timen
00_1metric_1495.7630482.62164102019-03-302019-04-031599994
10_23metric_1524.3677122.73164702019-03-302019-04-031596356
20_14metric_221.4602440.06945702019-03-302019-04-031600182
30_53metric_221.4374330.06994102019-03-302019-04-031601081
40_53metric_1548.3876912.89348602019-03-302019-04-031601081
\n", "
" ], "text/plain": [ " arm_name metric_name mean sem trial_index start_time \\\n", "0 0_1 metric_1 495.763048 2.621641 0 2019-03-30 \n", "1 0_23 metric_1 524.367712 2.731647 0 2019-03-30 \n", "2 0_14 metric_2 21.460244 0.069457 0 2019-03-30 \n", "3 0_53 metric_2 21.437433 0.069941 0 2019-03-30 \n", "4 0_53 metric_1 548.387691 2.893486 0 2019-03-30 \n", "\n", " end_time n \n", "0 2019-04-03 1599994 \n", "1 2019-04-03 1596356 \n", "2 2019-04-03 1600182 \n", "3 2019-04-03 1601081 \n", "4 2019-04-03 1601081 " ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data = Data(pd.read_json('hitl_data.json'))\n", "data.df.head()" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array(['0_1', '0_23', '0_14', '0_53', '0_0', '0_54', '0_55', '0_56',\n", " '0_27', '0_57', '0_58', '0_13', '0_59', '0_6', '0_60', '0_61',\n", " '0_62', '0_63', '0_7', '0_28', '0_15', '0_16', '0_17', '0_18',\n", " '0_19', '0_29', '0_2', '0_20', '0_21', '0_22', '0_3', '0_30',\n", " '0_8', '0_10', '0_31', '0_24', '0_32', '0_33', '0_34', '0_35',\n", " '0_36', '0_37', '0_38', '0_9', '0_39', '0_4', '0_25', '0_11',\n", " '0_40', '0_41', '0_42', '0_43', '0_44', '0_45', 'status_quo',\n", " '0_46', '0_47', '0_48', '0_26', '0_49', '0_12', '0_5', '0_50',\n", " '0_51', '0_52'], dtype=object)" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.df['arm_name'].unique()" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array(['metric_1', 'metric_2'], dtype=object)" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.df['metric_name'].unique()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Search Space** \n", "\n", "The final component necessary for human-in-the-loop optimization is a SearchSpace. A SearchSpace defines the feasible region for our parameters, as well as their types.\n", "\n", "Here, we have both parameters and a set of constraints on those parameters. \n", "\n", "Without a SearchSpace, our models are unable to generate new candidates. By default, the models will read the search space off of the experiment, when they are told to generate candidates. SearchSpaces can also be specified by the user at this time. Sometimes, the first round of an experiment is too restrictive--perhaps the experimenter was too cautious when defining their initial ranges for exploration! In this case, it can be useful to generate candidates from new, expanded search spaces, beyond that specified in the experiment. " ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'x_excellent': RangeParameter(name='x_excellent', parameter_type=FLOAT, range=[0.0, 1.0]),\n", " 'x_good': RangeParameter(name='x_good', parameter_type=FLOAT, range=[0.0, 1.0]),\n", " 'x_moderate': RangeParameter(name='x_moderate', parameter_type=FLOAT, range=[0.0, 1.0]),\n", " 'x_poor': RangeParameter(name='x_poor', parameter_type=FLOAT, range=[0.0, 1.0]),\n", " 'x_unknown': RangeParameter(name='x_unknown', parameter_type=FLOAT, range=[0.0, 1.0]),\n", " 'y_excellent': RangeParameter(name='y_excellent', parameter_type=FLOAT, range=[0.1, 3.0]),\n", " 'y_good': RangeParameter(name='y_good', parameter_type=FLOAT, range=[0.1, 3.0]),\n", " 'y_moderate': RangeParameter(name='y_moderate', parameter_type=FLOAT, range=[0.1, 3.0]),\n", " 'y_poor': RangeParameter(name='y_poor', parameter_type=FLOAT, range=[0.1, 3.0]),\n", " 'y_unknown': RangeParameter(name='y_unknown', parameter_type=FLOAT, range=[0.1, 3.0]),\n", " 'z_excellent': RangeParameter(name='z_excellent', parameter_type=FLOAT, range=[50000.0, 5000000.0]),\n", " 'z_good': RangeParameter(name='z_good', parameter_type=FLOAT, range=[50000.0, 5000000.0]),\n", " 'z_moderate': RangeParameter(name='z_moderate', parameter_type=FLOAT, range=[50000.0, 5000000.0]),\n", " 'z_poor': RangeParameter(name='z_poor', parameter_type=FLOAT, range=[50000.0, 5000000.0]),\n", " 'z_unknown': RangeParameter(name='z_unknown', parameter_type=FLOAT, range=[50000.0, 5000000.0])}" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "experiment.search_space.parameters" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[OrderConstraint(x_poor <= x_moderate),\n", " OrderConstraint(x_moderate <= x_good),\n", " OrderConstraint(x_good <= x_excellent),\n", " OrderConstraint(y_poor <= y_moderate),\n", " OrderConstraint(y_moderate <= y_good),\n", " OrderConstraint(y_good <= y_excellent)]" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "experiment.search_space.parameter_constraints" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Model Fit\n", "\n", "Fitting BoTorch's GPEI will allow us to predict new candidates based on our first Sobol batch. \n", "Here, we make use of the default settings for GP-EI defined in the ModelBridge factory. " ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "gp = get_GPEI(\n", " experiment=experiment,\n", " data=data,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can validate the model fits using cross validation, shown below for each metric of interest. Here, our model fits leave something to be desired--the tail ends of each metric are hard to model. In this situation, there are three potential actions to take: \n", "\n", "1. Increase the amount of traffic in this experiment, to reduce the measurement noise.\n", "2. Increase the number of points run in the random batch, to assist the GP in covering the space.\n", "3. Reduce the number of parameters tuned at one time. \n", "\n", "However, away from the tail effects, the fits do show a strong correlations, so we will proceed with candidate generation. " ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "linkText": "Export to plot.ly", "plotlyServerURL": "https://plot.ly", "showLink": false }, "data": [ { "hoverinfo": "none", "line": { "color": "black", "dash": "dot", "width": 2 }, "mode": "lines", "showlegend": false, "type": "scatter", "visible": true, "x": [ 448.38976507062, 606.3074654917367 ], "xaxis": "x", "y": [ 448.38976507062, 606.3074654917367 ], "yaxis": "y" }, { "error_x": { "array": [ 6.190927355172, 5.772626468364, 5.886846170632, 5.671232086071999, 5.167176563472, 5.237828343804, 5.8864067304, 5.86323027038, 5.988926380052, 5.798868959716, 5.627416608492, 5.7142095143, 5.3540288342240006, 4.67603093818, 6.096293581475999, 4.810065405488, 5.885906806136, 6.3328913268600004, 5.325433822296, 5.572223061803999, 5.827235964648, 5.06591711934, 5.118090228804, 5.545166264932, 5.802951513688, 6.11435071338, 5.45615506856, 5.533871155176, 5.797301127492, 5.498551454004, 5.301745460256, 5.519124127916, 5.343143744904, 5.89784668788, 5.574751998452, 5.848627122408, 5.848267544336, 5.3381252827, 5.424114778432, 5.65094921678, 6.102434032156, 5.7883311724, 5.1384162492600005, 5.984116165104, 5.650589316092, 5.526892887992, 5.769850742824, 5.639094136983999, 5.602080697892, 5.947329837483999, 6.133309401379999, 5.710890680796, 5.520030807255999, 5.778172100988, 6.08159215636, 5.573027659248, 4.938383116492, 5.8656403100959995, 5.418048856555999, 5.617999044868, 2.0637137267320003, 5.29582149628, 5.708466052016, 5.7305319832879995, 5.583339289807999 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "error_y": { "array": [ 37.89303221103355, 37.59317024758884, 35.1197959336435, 35.46838566231181, 38.57151944124726, 33.17315854098841, 39.39595904451878, 31.101041208324585, 37.30202001962297, 34.5878600646512, 36.26519867144207, 37.63586291776306, 38.11946017482805, 37.53116412920291, 29.105903005451104, 34.85064943313749, 34.30766196011062, 38.700229782054585, 36.72081989490793, 33.830638728964786, 36.51452407196543, 38.79747044466231, 33.15207360297466, 34.14097022339198, 36.707177650820306, 36.45133848044904, 33.5420716813542, 37.245284281611305, 34.39586656621052, 36.032476687441125, 35.79897306581593, 33.800645011459046, 34.641395462752065, 30.56643846529243, 38.44883573561593, 32.659821860763735, 33.209728266359214, 35.9219369267721, 34.57120505456969, 37.2415504581762, 32.876453620550855, 32.101134551016365, 33.110484542676026, 33.20329983506272, 34.710555576433336, 34.98166828373731, 36.81696240150535, 32.22260803164157, 41.07595634674203, 38.166501799607516, 34.69962267674952, 34.36403063881525, 36.204314551290715, 34.99101083468862, 32.04643569438623, 29.847273520737627, 40.10828530608979, 26.92441375702, 35.10286104142972, 37.16526957883767, 42.06307597615629, 34.8140134231506, 39.12156100882193, 33.61389953918892, 31.71856240113689 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "hoverinfo": "text", "marker": { "color": "rgba(128,177,211,1)" }, "mode": "markers", "name": "In-sample", "showlegend": true, "text": [ "Arm 0_44

Actual: 585.784 [579.593, 591.975]
Predicted: 568.414 [530.521, 606.307]
Parameterization has too many items to render on hover (15).", "Arm 0_52

Actual: 556.643 [550.870, 562.416]
Predicted: 544.149 [506.555, 581.742]
Parameterization has too many items to render on hover (15).", "Arm 0_24

Actual: 565.626 [559.740, 571.513]
Predicted: 552.301 [517.181, 587.420]
Parameterization has too many items to render on hover (15).", "Arm 0_53

Actual: 548.388 [542.716, 554.059]
Predicted: 551.169 [515.701, 586.637]
Parameterization has too many items to render on hover (15).", "Arm 0_63

Actual: 506.364 [501.196, 511.531]
Predicted: 525.303 [486.731, 563.874]
Parameterization has too many items to render on hover (15).", "Arm 0_40

Actual: 512.301 [507.063, 517.538]
Predicted: 500.449 [467.276, 533.622]
Parameterization has too many items to render on hover (15).", "Arm 0_37

Actual: 565.501 [559.615, 571.388]
Predicted: 545.194 [505.798, 584.590]
Parameterization has too many items to render on hover (15).", "Arm 0_56

Actual: 561.015 [555.152, 566.879]
Predicted: 567.996 [536.895, 599.097]
Parameterization has too many items to render on hover (15).", "Arm 0_59

Actual: 572.732 [566.743, 578.721]
Predicted: 559.351 [522.049, 596.653]
Parameterization has too many items to render on hover (15).", "Arm 0_26

Actual: 562.617 [556.819, 568.416]
Predicted: 553.884 [519.296, 588.471]
Parameterization has too many items to render on hover (15).", "Arm 0_6

Actual: 540.741 [535.114, 546.369]
Predicted: 539.339 [503.074, 575.604]
Parameterization has too many items to render on hover (15).", "Arm 0_5

Actual: 549.051 [543.337, 554.765]
Predicted: 540.624 [502.988, 578.260]
Parameterization has too many items to render on hover (15).", "Arm 0_23

Actual: 524.368 [519.014, 529.722]
Predicted: 541.671 [503.551, 579.790]
Parameterization has too many items to render on hover (15).", "Arm 0_31

Actual: 453.066 [448.390, 457.742]
Predicted: 505.223 [467.692, 542.754]
Parameterization has too many items to render on hover (15).", "Arm 0_16

Actual: 574.527 [568.431, 580.624]
Predicted: 575.567 [546.461, 604.673]
Parameterization has too many items to render on hover (15).", "Arm 0_0

Actual: 464.447 [459.637, 469.257]
Predicted: 510.085 [475.235, 544.936]
Parameterization has too many items to render on hover (15).", "Arm 0_36

Actual: 563.958 [558.072, 569.843]
Predicted: 565.307 [530.999, 599.614]
Parameterization has too many items to render on hover (15).", "Arm 0_61

Actual: 591.711 [585.378, 598.044]
Predicted: 555.718 [517.018, 594.418]
Parameterization has too many items to render on hover (15).", "Arm 0_57

Actual: 515.547 [510.222, 520.872]
Predicted: 522.756 [486.035, 559.477]
Parameterization has too many items to render on hover (15).", "Arm 0_4

Actual: 544.43 [538.857, 550.002]
Predicted: 546.191 [512.360, 580.021]
Parameterization has too many items to render on hover (15).", "Arm 0_50

Actual: 562.709 [556.882, 568.536]
Predicted: 554.558 [518.043, 591.072]
Parameterization has too many items to render on hover (15).", "Arm 0_62

Actual: 486.998 [481.932, 492.064]
Predicted: 526.493 [487.696, 565.291]
Parameterization has too many items to render on hover (15).", "Arm 0_10

Actual: 496.954 [491.835, 502.072]
Predicted: 520.963 [487.811, 554.115]
Parameterization has too many items to render on hover (15).", "Arm 0_43

Actual: 540.156 [534.611, 545.701]
Predicted: 545.957 [511.816, 580.098]
Parameterization has too many items to render on hover (15).", "Arm 0_39

Actual: 559.704 [553.901, 565.507]
Predicted: 552.974 [516.267, 589.682]
Parameterization has too many items to render on hover (15).", "Arm 0_32

Actual: 579.876 [573.761, 585.990]
Predicted: 560.202 [523.751, 596.653]
Parameterization has too many items to render on hover (15).", "Arm 0_51

Actual: 534.877 [529.421, 540.333]
Predicted: 529.891 [496.349, 563.434]
Parameterization has too many items to render on hover (15).", "Arm 0_7

Actual: 537.722 [532.188, 543.256]
Predicted: 527.147 [489.902, 564.392]
Parameterization has too many items to render on hover (15).", "Arm 0_49

Actual: 558.937 [553.140, 564.735]
Predicted: 563.219 [528.823, 597.615]
Parameterization has too many items to render on hover (15).", "Arm 0_54

Actual: 540.355 [534.856, 545.853]
Predicted: 540.526 [504.494, 576.559]
Parameterization has too many items to render on hover (15).", "Arm 0_9

Actual: 515.907 [510.606, 521.209]
Predicted: 520.387 [484.589, 556.186]
Parameterization has too many items to render on hover (15).", "Arm 0_30

Actual: 534.884 [529.365, 540.403]
Predicted: 521.171 [487.371, 554.972]
Parameterization has too many items to render on hover (15).", "Arm 0_46

Actual: 521.844 [516.500, 527.187]
Predicted: 502.48 [467.839, 537.122]
Parameterization has too many items to render on hover (15).", "Arm 0_35

Actual: 564.092 [558.194, 569.989]
Predicted: 563.111 [532.545, 593.678]
Parameterization has too many items to render on hover (15).", "Arm 0_45

Actual: 543.492 [537.917, 549.066]
Predicted: 531.651 [493.202, 570.100]
Parameterization has too many items to render on hover (15).", "Arm 0_20

Actual: 561.147 [555.298, 566.995]
Predicted: 556.969 [524.310, 589.629]
Parameterization has too many items to render on hover (15).", "Arm 0_25

Actual: 564.735 [558.886, 570.583]
Predicted: 557.441 [524.231, 590.651]
Parameterization has too many items to render on hover (15).", "Arm 0_27

Actual: 521.489 [516.150, 526.827]
Predicted: 538.702 [502.780, 574.624]
Parameterization has too many items to render on hover (15).", "Arm 0_15

Actual: 517.029 [511.605, 522.453]
Predicted: 555.143 [520.572, 589.715]
Parameterization has too many items to render on hover (15).", "Arm 0_8

Actual: 546.857 [541.206, 552.508]
Predicted: 544.046 [506.805, 581.288]
Parameterization has too many items to render on hover (15).", "Arm 0_11

Actual: 577.68 [571.578, 583.783]
Predicted: 568.135 [535.259, 601.012]
Parameterization has too many items to render on hover (15).", "Arm 0_17

Actual: 553.942 [548.153, 559.730]
Predicted: 561.003 [528.902, 593.104]
Parameterization has too many items to render on hover (15).", "Arm 0_1

Actual: 495.763 [490.625, 500.901]
Predicted: 518.369 [485.258, 551.479]
Parameterization has too many items to render on hover (15).", "Arm 0_22

Actual: 570.823 [564.839, 576.807]
Predicted: 556.592 [523.389, 589.796]
Parameterization has too many items to render on hover (15).", "Arm 0_58

Actual: 547.101 [541.451, 552.752]
Predicted: 539.387 [504.677, 574.098]
Parameterization has too many items to render on hover (15).", "Arm 0_42

Actual: 533.3 [527.773, 538.826]
Predicted: 534.356 [499.374, 569.338]
Parameterization has too many items to render on hover (15).", "Arm 0_34

Actual: 538.84 [533.071, 544.610]
Predicted: 547.981 [511.164, 584.798]
Parameterization has too many items to render on hover (15).", "Arm 0_2

Actual: 542.303 [536.664, 547.942]
Predicted: 552.658 [520.435, 584.880]
Parameterization has too many items to render on hover (15).", "Arm 0_47

Actual: 541.149 [535.547, 546.751]
Predicted: 545.814 [504.738, 586.890]
Parameterization has too many items to render on hover (15).", "Arm 0_29

Actual: 569.49 [563.542, 575.437]
Predicted: 543.629 [505.463, 581.796]
Parameterization has too many items to render on hover (15).", "Arm 0_33

Actual: 579.549 [573.416, 585.682]
Predicted: 564.186 [529.486, 598.886]
Parameterization has too many items to render on hover (15).", "Arm 0_3

Actual: 548.235 [542.524, 553.945]
Predicted: 538.905 [504.541, 573.270]
Parameterization has too many items to render on hover (15).", "Arm 0_48

Actual: 535.163 [529.643, 540.683]
Predicted: 547.861 [511.657, 584.066]
Parameterization has too many items to render on hover (15).", "Arm 0_38

Actual: 553.54 [547.762, 559.318]
Predicted: 559.895 [524.904, 594.886]
Parameterization has too many items to render on hover (15).", "Arm 0_60

Actual: 577.466 [571.384, 583.548]
Predicted: 570.427 [538.381, 602.474]
Parameterization has too many items to render on hover (15).", "Arm 0_12

Actual: 543.44 [537.867, 549.013]
Predicted: 517.098 [487.251, 546.946]
Parameterization has too many items to render on hover (15).", "Arm 0_19

Actual: 477.576 [472.637, 482.514]
Predicted: 513.492 [473.384, 553.601]
Parameterization has too many items to render on hover (15).", "Arm 0_21

Actual: 563.654 [557.788, 569.520]
Predicted: 561.694 [534.770, 588.618]
Parameterization has too many items to render on hover (15).", "Arm 0_41

Actual: 531.739 [526.321, 537.157]
Predicted: 540.968 [505.865, 576.071]
Parameterization has too many items to render on hover (15).", "Arm 0_55

Actual: 543.351 [537.733, 548.969]
Predicted: 549.509 [512.344, 586.674]
Parameterization has too many items to render on hover (15).", "Arm status_quo

Actual: 560.647 [558.584, 562.711]
Predicted: 540.299 [498.236, 582.362]
Parameterization has too many items to render on hover (15).", "Arm 0_28

Actual: 518.569 [513.273, 523.865]
Predicted: 527.575 [492.760, 562.389]
Parameterization has too many items to render on hover (15).", "Arm 0_18

Actual: 553.026 [547.317, 558.734]
Predicted: 534.067 [494.946, 573.189]
Parameterization has too many items to render on hover (15).", "Arm 0_14

Actual: 556.068 [550.338, 561.799]
Predicted: 556.202 [522.588, 589.816]
Parameterization has too many items to render on hover (15).", "Arm 0_13

Actual: 544.589 [539.006, 550.172]
Predicted: 549.964 [518.246, 581.683]
Parameterization has too many items to render on hover (15)." ], "type": "scatter", "visible": true, "x": [ 585.7843090537, 556.6430987863, 565.6264647098, 548.3876911099, 506.3635257112, 512.3005012676, 565.5011648274, 561.0153373251, 572.7317928028, 562.6174250382, 540.7413033532, 549.0511518835, 524.3677121973, 453.0657960088, 574.5273877209, 464.4467049594, 563.9575508896, 591.7112903438, 515.5469981054, 544.4295676658, 562.7090597954, 486.9978083981, 496.9535891264, 540.1556866168, 559.7036228384, 579.8755075795, 534.8772454669, 537.7218273947, 558.9373312742, 540.354941559, 515.9073035981, 534.8840893416, 521.8436058271, 564.0915622365, 543.4917462892, 561.14681338, 564.7346483405, 521.4885673544, 517.0287055503, 546.8571018679, 577.6802807769, 553.9417568228, 495.7630483864, 570.8228707945, 547.1011628546, 533.2995099946, 538.8404308226, 542.3029350722, 541.1493906184, 569.4895133769, 579.5490838314, 548.2345843739, 535.162617066, 553.5400395602, 577.466025923, 543.4399676676, 477.5755207251, 563.6541000894, 531.7389392235, 543.3509256103, 560.6473614869, 518.5691087639, 553.0256150304, 556.068091252, 544.5891065864 ], "xaxis": "x", "y": [ 568.4144332807032, 544.1486215115954, 552.3006925620916, 551.168921575117, 525.302676643966, 500.44871401316254, 545.1936329360982, 567.9964451037938, 559.3511139050994, 553.8835399982524, 539.3391847131048, 540.6243105400042, 541.6705298081439, 505.22295992292163, 575.5666759976448, 510.08544792985583, 565.306831137206, 555.7177384542981, 522.7558408398207, 546.1905921624231, 554.5577524033392, 526.4934489547479, 520.9629182964444, 545.9570046392935, 552.9744536964504, 560.2021460648713, 529.8914672031704, 527.147052019302, 563.2193314831068, 540.5261066664567, 520.3874859601214, 521.171175030216, 502.48047423720226, 563.1114330916406, 531.650779345191, 556.9693821163283, 557.4409344670077, 538.7020497898658, 555.1433575320291, 544.0463540737123, 568.1352323133023, 561.0027164849203, 518.368823754035, 556.5922172582771, 539.3873494524552, 534.356138725886, 547.9813788805153, 552.6575155782202, 545.8135574269817, 543.6290160487216, 564.1859638651191, 538.9054723851467, 547.8613698539359, 559.8950418005089, 570.4272569618711, 517.0983796033333, 513.4922239132426, 561.6940374565252, 540.9683452122575, 549.5091163704695, 540.2987750765238, 527.5745091221713, 534.0672913864701, 556.2020843495693, 549.9642215923072 ], "yaxis": "y" }, { "hoverinfo": "none", "line": { "color": "black", "dash": "dot", "width": 2 }, "mode": "lines", "showlegend": false, "type": "scatter", "visible": true, "x": [ 20.084380635715515, 24.136214740024 ], "xaxis": "x2", "y": [ 20.084380635715515, 24.136214740024 ], "yaxis": "y2" }, { "error_x": { "array": [ 0.15260062336400002, 0.13819307614, 0.138841771264, 0.13708526454, 0.13616042913200002, 0.13504200491600002, 0.145560420376, 0.14083460146000001, 0.139001731568, 0.138426719508, 0.135587722228, 0.13689072062399998, 0.13550375720000002, 0.134213041928, 0.143115790972, 0.13613213418, 0.139859693148, 0.160122590824, 0.136175133836, 0.135737426244, 0.140573637652, 0.13729417114, 0.137623368232, 0.136089003008, 0.13648318488000002, 0.139931757056, 0.140151500888, 0.137380934068, 0.139130825792, 0.13503340306400002, 0.13506536498, 0.136899248388, 0.136497777276, 0.135432008852, 0.137450593644, 0.135963966376, 0.13729121683200002, 0.13685765209200002, 0.13440878771599998, 0.13639236083200001, 0.141532092156, 0.13729187578399998, 0.135580132716, 0.137857664476, 0.13852454526400002, 0.139328979048, 0.13799847656, 0.137405700628, 0.145277608056, 0.14735822214400002, 0.14215547878, 0.137566380644, 0.134035104308, 0.13840536609200002, 0.140269432324, 0.13943522770800001, 0.135733025064, 0.13902685347600002, 0.133955590832, 0.13758017238, 0.048670612592, 0.136269576044, 0.13627733, 0.13613659514, 0.137215147664 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "error_y": { "array": [ 0.8393080129497453, 0.8283477189639717, 0.7259075761312267, 0.7744591797484566, 0.834301553330492, 0.7328446592013276, 0.865582341387274, 0.6990124169573776, 0.8068599620219472, 0.7439006731985106, 0.7943495024307955, 0.809971171814888, 0.8503622565384642, 0.8144860911914966, 0.6195885594082874, 0.7750692963782035, 0.7112301837895003, 0.7924566096886022, 0.8161999418871051, 0.7474835895568889, 0.8090851977146871, 0.8601888389191574, 0.7139933659804036, 0.7408829665418449, 0.8043943040253139, 0.78821563012587, 0.7428085653952547, 0.789776822519969, 0.7265679514649412, 0.7712943700640611, 0.789644715847749, 0.7250771688483412, 0.7498754145631465, 0.6262879507996786, 0.8491192763049722, 0.7036519663688661, 0.7393702677069675, 0.7916709876098769, 0.7779315243532644, 0.8012249882478357, 0.7126471068033279, 0.6631352087804409, 0.7479691888731921, 0.6982100958286395, 0.7675129224727351, 0.7784819570221259, 0.803452654363432, 0.7036923942763025, 0.9304210828999834, 0.8635261520966099, 0.7252487479945205, 0.744187528240209, 0.7858705900538769, 0.7427885101476709, 0.6743945185347757, 0.6871864082034645, 0.8716391256706476, 0.5804013752250926, 0.7509102695706406, 0.8180831164690537, 0.9564389360474348, 0.7556461643808257, 0.860351051948185, 0.7062686003645899, 0.6623191663384792 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "hoverinfo": "text", "marker": { "color": "rgba(128,177,211,1)" }, "mode": "markers", "name": "In-sample", "showlegend": true, "text": [ "Arm 0_44

Actual: 23.293 [23.140, 23.446]
Predicted: 22.348 [21.508, 23.187]
Parameterization has too many items to render on hover (15).", "Arm 0_52

Actual: 21.832 [21.693, 21.970]
Predicted: 21.701 [20.873, 22.530]
Parameterization has too many items to render on hover (15).", "Arm 0_24

Actual: 22.061 [21.923, 22.200]
Predicted: 21.666 [20.940, 22.392]
Parameterization has too many items to render on hover (15).", "Arm 0_53

Actual: 21.437 [21.300, 21.575]
Predicted: 21.555 [20.780, 22.329]
Parameterization has too many items to render on hover (15).", "Arm 0_63

Actual: 21.039 [20.903, 21.175]
Predicted: 21.304 [20.470, 22.138]
Parameterization has too many items to render on hover (15).", "Arm 0_40

Actual: 20.931 [20.796, 21.066]
Predicted: 20.817 [20.084, 21.550]
Parameterization has too many items to render on hover (15).", "Arm 0_37

Actual: 22.637 [22.492, 22.783]
Predicted: 21.915 [21.050, 22.781]
Parameterization has too many items to render on hover (15).", "Arm 0_56

Actual: 21.876 [21.735, 22.017]
Predicted: 22.093 [21.394, 22.792]
Parameterization has too many items to render on hover (15).", "Arm 0_59

Actual: 21.852 [21.713, 21.991]
Predicted: 21.893 [21.086, 22.700]
Parameterization has too many items to render on hover (15).", "Arm 0_26

Actual: 21.88 [21.741, 22.018]
Predicted: 21.622 [20.878, 22.366]
Parameterization has too many items to render on hover (15).", "Arm 0_6

Actual: 21.351 [21.216, 21.487]
Predicted: 21.407 [20.613, 22.202]
Parameterization has too many items to render on hover (15).", "Arm 0_5

Actual: 21.473 [21.336, 21.610]
Predicted: 21.412 [20.602, 22.222]
Parameterization has too many items to render on hover (15).", "Arm 0_23

Actual: 20.947 [20.812, 21.083]
Predicted: 21.336 [20.486, 22.186]
Parameterization has too many items to render on hover (15).", "Arm 0_31

Actual: 20.369 [20.235, 20.504]
Predicted: 20.971 [20.156, 21.785]
Parameterization has too many items to render on hover (15).", "Arm 0_16

Actual: 22.191 [22.048, 22.334]
Predicted: 22.313 [21.694, 22.933]
Parameterization has too many items to render on hover (15).", "Arm 0_0

Actual: 20.394 [20.258, 20.530]
Predicted: 20.891 [20.116, 21.666]
Parameterization has too many items to render on hover (15).", "Arm 0_36

Actual: 21.916 [21.776, 22.056]
Predicted: 22.183 [21.472, 22.894]
Parameterization has too many items to render on hover (15).", "Arm 0_61

Actual: 23.976 [23.816, 24.136]
Predicted: 22.055 [21.262, 22.847]
Parameterization has too many items to render on hover (15).", "Arm 0_57

Actual: 20.938 [20.802, 21.074]
Predicted: 21.056 [20.240, 21.872]
Parameterization has too many items to render on hover (15).", "Arm 0_4

Actual: 21.303 [21.168, 21.439]
Predicted: 21.364 [20.616, 22.111]
Parameterization has too many items to render on hover (15).", "Arm 0_50

Actual: 22.12 [21.979, 22.261]
Predicted: 21.898 [21.089, 22.707]
Parameterization has too many items to render on hover (15).", "Arm 0_62

Actual: 20.807 [20.669, 20.944]
Predicted: 21.314 [20.454, 22.174]
Parameterization has too many items to render on hover (15).", "Arm 0_10

Actual: 20.856 [20.719, 20.994]
Predicted: 21.354 [20.640, 22.068]
Parameterization has too many items to render on hover (15).", "Arm 0_43

Actual: 21.338 [21.202, 21.475]
Predicted: 21.41 [20.669, 22.151]
Parameterization has too many items to render on hover (15).", "Arm 0_39

Actual: 21.664 [21.528, 21.800]
Predicted: 21.59 [20.786, 22.395]
Parameterization has too many items to render on hover (15).", "Arm 0_32

Actual: 22.329 [22.189, 22.469]
Predicted: 22.079 [21.291, 22.867]
Parameterization has too many items to render on hover (15).", "Arm 0_51

Actual: 21.341 [21.200, 21.481]
Predicted: 21.598 [20.856, 22.341]
Parameterization has too many items to render on hover (15).", "Arm 0_7

Actual: 21.296 [21.159, 21.434]
Predicted: 21.068 [20.279, 21.858]
Parameterization has too many items to render on hover (15).", "Arm 0_49

Actual: 21.8 [21.661, 21.939]
Predicted: 21.822 [21.096, 22.549]
Parameterization has too many items to render on hover (15).", "Arm 0_54

Actual: 21.363 [21.228, 21.498]
Predicted: 21.388 [20.617, 22.159]
Parameterization has too many items to render on hover (15).", "Arm 0_9

Actual: 21.034 [20.899, 21.169]
Predicted: 21.033 [20.243, 21.823]
Parameterization has too many items to render on hover (15).", "Arm 0_30

Actual: 21.047 [20.910, 21.184]
Predicted: 21.167 [20.442, 21.892]
Parameterization has too many items to render on hover (15).", "Arm 0_46

Actual: 20.975 [20.838, 21.111]
Predicted: 20.876 [20.126, 21.626]
Parameterization has too many items to render on hover (15).", "Arm 0_35

Actual: 21.779 [21.643, 21.914]
Predicted: 21.632 [21.006, 22.259]
Parameterization has too many items to render on hover (15).", "Arm 0_45

Actual: 21.161 [21.023, 21.298]
Predicted: 21.294 [20.444, 22.143]
Parameterization has too many items to render on hover (15).", "Arm 0_20

Actual: 21.622 [21.486, 21.758]
Predicted: 21.686 [20.982, 22.389]
Parameterization has too many items to render on hover (15).", "Arm 0_25

Actual: 21.64 [21.503, 21.777]
Predicted: 21.737 [20.998, 22.476]
Parameterization has too many items to render on hover (15).", "Arm 0_27

Actual: 21.77 [21.634, 21.907]
Predicted: 22.139 [21.347, 22.930]
Parameterization has too many items to render on hover (15).", "Arm 0_15

Actual: 21.116 [20.982, 21.251]
Predicted: 21.891 [21.113, 22.669]
Parameterization has too many items to render on hover (15).", "Arm 0_8

Actual: 21.166 [21.029, 21.302]
Predicted: 21.431 [20.630, 22.232]
Parameterization has too many items to render on hover (15).", "Arm 0_11

Actual: 22.181 [22.040, 22.323]
Predicted: 22.381 [21.668, 23.094]
Parameterization has too many items to render on hover (15).", "Arm 0_17

Actual: 21.713 [21.575, 21.850]
Predicted: 21.615 [20.952, 22.278]
Parameterization has too many items to render on hover (15).", "Arm 0_1

Actual: 20.867 [20.731, 21.002]
Predicted: 21.132 [20.384, 21.880]
Parameterization has too many items to render on hover (15).", "Arm 0_22

Actual: 21.622 [21.484, 21.760]
Predicted: 21.742 [21.044, 22.440]
Parameterization has too many items to render on hover (15).", "Arm 0_58

Actual: 21.321 [21.183, 21.460]
Predicted: 21.402 [20.635, 22.170]
Parameterization has too many items to render on hover (15).", "Arm 0_42

Actual: 21.394 [21.254, 21.533]
Predicted: 21.551 [20.772, 22.329]
Parameterization has too many items to render on hover (15).", "Arm 0_34

Actual: 22.082 [21.944, 22.220]
Predicted: 21.876 [21.072, 22.679]
Parameterization has too many items to render on hover (15).", "Arm 0_2

Actual: 21.419 [21.282, 21.557]
Predicted: 21.529 [20.825, 22.232]
Parameterization has too many items to render on hover (15).", "Arm 0_47

Actual: 22.506 [22.360, 22.651]
Predicted: 22.057 [21.127, 22.988]
Parameterization has too many items to render on hover (15).", "Arm 0_29

Actual: 22.928 [22.781, 23.075]
Predicted: 22.065 [21.201, 22.929]
Parameterization has too many items to render on hover (15).", "Arm 0_33

Actual: 22.492 [22.350, 22.634]
Predicted: 21.891 [21.166, 22.616]
Parameterization has too many items to render on hover (15).", "Arm 0_3

Actual: 21.546 [21.409, 21.684]
Predicted: 21.393 [20.648, 22.137]
Parameterization has too many items to render on hover (15).", "Arm 0_48

Actual: 21.384 [21.250, 21.518]
Predicted: 21.346 [20.560, 22.132]
Parameterization has too many items to render on hover (15).", "Arm 0_38

Actual: 21.472 [21.334, 21.611]
Predicted: 21.635 [20.893, 22.378]
Parameterization has too many items to render on hover (15).", "Arm 0_60

Actual: 22.243 [22.103, 22.383]
Predicted: 22.155 [21.480, 22.829]
Parameterization has too many items to render on hover (15).", "Arm 0_12

Actual: 21.508 [21.369, 21.648]
Predicted: 21.353 [20.666, 22.040]
Parameterization has too many items to render on hover (15).", "Arm 0_19

Actual: 20.915 [20.779, 21.051]
Predicted: 21.2 [20.328, 22.071]
Parameterization has too many items to render on hover (15).", "Arm 0_21

Actual: 21.66 [21.521, 21.799]
Predicted: 21.52 [20.940, 22.101]
Parameterization has too many items to render on hover (15).", "Arm 0_41

Actual: 21.26 [21.127, 21.394]
Predicted: 21.039 [20.288, 21.789]
Parameterization has too many items to render on hover (15).", "Arm 0_55

Actual: 21.409 [21.271, 21.546]
Predicted: 21.445 [20.627, 22.263]
Parameterization has too many items to render on hover (15).", "Arm status_quo

Actual: 21.637 [21.589, 21.686]
Predicted: 22.012 [21.056, 22.969]
Parameterization has too many items to render on hover (15).", "Arm 0_28

Actual: 21.085 [20.949, 21.222]
Predicted: 21.542 [20.786, 22.297]
Parameterization has too many items to render on hover (15).", "Arm 0_18

Actual: 21.354 [21.217, 21.490]
Predicted: 21.308 [20.447, 22.168]
Parameterization has too many items to render on hover (15).", "Arm 0_14

Actual: 21.46 [21.324, 21.596]
Predicted: 21.399 [20.693, 22.106]
Parameterization has too many items to render on hover (15).", "Arm 0_13

Actual: 21.418 [21.281, 21.556]
Predicted: 21.204 [20.542, 21.867]
Parameterization has too many items to render on hover (15)." ], "type": "scatter", "visible": true, "x": [ 23.2929146801, 21.8315714862, 22.061401753, 21.4374334625, 21.038950512, 20.9308294427, 22.6371209753, 21.8761495501, 21.8518919135, 21.8797036224, 21.3514435661, 21.4727492046, 20.9474084514, 20.3694726715, 22.1907197586, 20.3938551623, 21.9157166512, 23.9760921492, 20.9381701672, 21.3033959051, 22.1199857861, 20.8066326138, 20.8561277107, 21.338490998, 21.6640149689, 22.3289678016, 21.3405492897, 21.2963493601, 21.8001539366, 21.3634470762, 21.0336997163, 21.0473200141, 20.9745011961, 21.7785559743, 21.1609294892, 21.6224324583, 21.6400346563, 21.7704708828, 21.1164428415, 21.1658724048, 22.1811563552, 21.7126315645, 20.8668886394, 21.6216898678, 21.3210724987, 21.3937173897, 22.0823489199, 21.4194272163, 22.5055156411, 22.9278848853, 22.4920081911, 21.5463005234, 21.3843847146, 21.4723590682, 22.2431650654, 21.5082145841, 20.9149831146, 21.6598472233, 21.2604964675, 21.4088999606, 21.6372636518, 21.0854461784, 21.3535329606, 21.4602436992, 21.4183918127 ], "xaxis": "x2", "y": [ 22.347671928530836, 21.701386713671145, 21.666081175729907, 21.55483824271535, 21.304154729823768, 20.81722529491684, 21.91531131853578, 22.09298031947607, 21.893268379130085, 21.622008936772968, 21.407270106078997, 21.411858310886814, 21.336089129723522, 20.97060240541131, 22.31332168298832, 20.89098937895223, 22.182991270217215, 22.054583020734913, 21.055709976986027, 21.36367142935505, 21.897658273718683, 21.313798130384388, 21.35435436796898, 21.409963222943396, 21.59018636188756, 22.07916054014311, 21.598435963373255, 21.06849444380679, 21.822482084797212, 21.388068440075546, 21.032965709973585, 21.166921332718086, 20.876295538068362, 21.632490212502038, 21.293536076573343, 21.685783927069213, 21.73702744359255, 22.13870622540406, 21.89060296979386, 21.430825176742236, 22.380865716762084, 21.614795422085923, 21.132259271563427, 21.741838575132338, 21.402261506350847, 21.550511251683155, 21.875937893880746, 21.528544405182746, 22.057478388987565, 22.06502608122708, 21.891235833529876, 21.392680934460785, 21.345640271178265, 21.635324553558764, 22.154843883025617, 21.352876543077254, 21.199591218062743, 21.52020242004916, 21.03854742467888, 21.4446476536615, 22.012294806071544, 21.541696191904283, 21.307679992481948, 21.39928823020686, 21.204328596998312 ], "yaxis": "y2" } ], "layout": { "annotations": [ { "font": { "size": 12 }, "showarrow": false, "text": "metric_1", "x": 0.2125, "xanchor": "center", "xref": "paper", "y": 1.0, "yanchor": "bottom", "yref": "paper" }, { "font": { "size": 12 }, "showarrow": false, "text": "metric_2", "x": 0.7875, "xanchor": "center", "xref": "paper", "y": 1.0, "yanchor": "bottom", "yref": "paper" } ], "font": { "size": 10 }, "height": 400, "hovermode": "closest", "showlegend": false, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "sequentialminus": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "title": { "text": "Cross-Validation" }, "width": 800, "xaxis": { "anchor": "y", "domain": [ 0.0, 0.425 ], "linecolor": "black", "linewidth": 0.5, "mirror": true, "title": { "text": "Actual Outcome" } }, "xaxis2": { "anchor": "y2", "domain": [ 0.575, 1.0 ], "linecolor": "black", "linewidth": 0.5, "mirror": true, "title": { "text": "Actual Outcome" } }, "yaxis": { "anchor": "x", "domain": [ 0.0, 1.0 ], "linecolor": "black", "linewidth": 0.5, "mirror": true, "title": { "text": "Predicted Outcome" } }, "yaxis2": { "anchor": "x2", "domain": [ 0.0, 1.0 ], "linecolor": "black", "linewidth": 0.5, "mirror": true, "title": { "text": "Predicted Outcome" } } } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "cv_result = cross_validate(gp)\n", "render(tile_cross_validation(cv_result))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The parameters from the initial batch have a wide range of effects on the metrics of interest, as shown from the outcomes from our fitted GP model. " ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "linkText": "Export to plot.ly", "plotlyServerURL": "https://plot.ly", "showLink": false }, "data": [ { "error_y": { "array": [ 0.6547387343317975, 0.6532033115837151, 0.6609466315731528, 0.6813554440847576, 0.6694705491295897, 0.6591303137747925, 0.6563086390178189, 0.6508039407989954, 0.6839216136881153, 0.6604406759568969, 0.6601787231575951, 0.6565366929811817, 0.6613093240694986, 0.6561140874958142, 0.6633706122047732, 0.6637799634962176, 0.6555405967295402, 0.6704566164608853, 0.66273573692496, 0.6685871803108022, 0.6628627361853555, 0.6572002148831203, 0.712446900341419, 0.6636937933944925, 0.6586242603445244, 0.6475500126287197, 0.6773760670047011, 0.6859237814996093, 0.6688072097037889, 0.6516465341587597, 0.6738024004862555, 0.7036162417194723, 0.6668939573219893, 0.6610547090506331, 0.6553518681013905, 0.6508051033775936, 0.64797757405811, 0.6713714106339492, 0.656723211631772, 0.7346090086077302, 0.6642524778308558, 0.6574536033119194, 0.7031723757695748, 0.6494774398002832, 0.6704885519899918, 0.6622056608350378, 0.6797639514603396, 0.6735895146091605, 0.6692892385552365, 0.6620882696925839, 0.6531942830594355, 0.6650013545172943, 0.6769982685390781, 0.6574731612170758, 0.6674033123702455, 0.6722373506713744, 0.6559893752039715, 0.6748097809173098, 0.7697444470145935, 0.6626605822603224, 0.6581382444319371, 0.663286850538724, 0.658955602798785, 0.6526018616070007, 0.3176861573292605 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "hoverinfo": "text", "legendgroup": "In-sample", "marker": { "color": "rgba(128,177,211,1)" }, "mode": "markers", "name": "In-sample", "showlegend": true, "text": [ "Arm 0_0

metric_2: -5.683% [-6.338%, -5.028%]
", "Arm 0_1

metric_2: -3.526% [-4.180%, -2.873%]
", "Arm 0_10

metric_2: -3.535% [-4.196%, -2.874%]
", "Arm 0_11

metric_2: 2.542% [1.861%, 3.224%]
", "Arm 0_12

metric_2: -0.629% [-1.298%, 0.041%]
", "Arm 0_13

metric_2: -1.055% [-1.714%, -0.396%]
", "Arm 0_14

metric_2: -0.832% [-1.488%, -0.176%]
", "Arm 0_15

metric_2: -2.319% [-2.970%, -1.669%]
", "Arm 0_16

metric_2: 2.582% [1.898%, 3.266%]
", "Arm 0_17

metric_2: 0.326% [-0.334%, 0.987%]
", "Arm 0_18

metric_2: -1.321% [-1.981%, -0.661%]
", "Arm 0_19

metric_2: -3.315% [-3.971%, -2.658%]
", "Arm 0_2

metric_2: -0.992% [-1.653%, -0.331%]
", "Arm 0_20

metric_2: -0.063% [-0.719%, 0.594%]
", "Arm 0_21

metric_2: 0.066% [-0.597%, 0.730%]
", "Arm 0_22

metric_2: -0.055% [-0.719%, 0.608%]
", "Arm 0_23

metric_2: -3.15% [-3.805%, -2.494%]
", "Arm 0_24

metric_2: 1.893% [1.223%, 2.564%]
", "Arm 0_25

metric_2: 0.024% [-0.639%, 0.686%]
", "Arm 0_26

metric_2: 1.077% [0.409%, 1.746%]
", "Arm 0_27

metric_2: 0.651% [-0.012%, 1.314%]
", "Arm 0_28

metric_2: -2.49% [-3.147%, -1.832%]
", "Arm 0_29

metric_2: 5.858% [5.145%, 6.570%]
", "Arm 0_3

metric_2: -0.448% [-1.112%, 0.215%]
", "Arm 0_30

metric_2: -2.712% [-3.371%, -2.054%]
", "Arm 0_31

metric_2: -5.792% [-6.440%, -5.145%]
", "Arm 0_32

metric_2: 3.158% [2.481%, 3.835%]
", "Arm 0_33

metric_2: 3.853% [3.167%, 4.539%]
", "Arm 0_34

metric_2: 2.026% [1.357%, 2.695%]
", "Arm 0_35

metric_2: 0.619% [-0.033%, 1.270%]
", "Arm 0_36

metric_2: 1.328% [0.654%, 2.002%]
", "Arm 0_37

metric_2: 4.539% [3.836%, 5.243%]
", "Arm 0_38

metric_2: -0.74% [-1.407%, -0.074%]
", "Arm 0_39

metric_2: 0.11% [-0.551%, 0.771%]
", "Arm 0_4

metric_2: -1.538% [-2.193%, -0.883%]
", "Arm 0_40

metric_2: -3.287% [-3.937%, -2.636%]
", "Arm 0_41

metric_2: -1.776% [-2.424%, -1.128%]
", "Arm 0_42

metric_2: -1.107% [-1.779%, -0.436%]
", "Arm 0_43

metric_2: -1.374% [-2.031%, -0.717%]
", "Arm 0_44

metric_2: 7.531% [6.797%, 8.266%]
", "Arm 0_45

metric_2: -2.19% [-2.854%, -1.526%]
", "Arm 0_46

metric_2: -3.082% [-3.740%, -2.425%]
", "Arm 0_47

metric_2: 3.964% [3.261%, 4.667%]
", "Arm 0_48

metric_2: -1.178% [-1.827%, -0.528%]
", "Arm 0_49

metric_2: 0.752% [0.081%, 1.422%]
", "Arm 0_5

metric_2: -0.772% [-1.435%, -0.110%]
", "Arm 0_50

metric_2: 2.197% [1.517%, 2.877%]
", "Arm 0_51

metric_2: -1.337% [-2.011%, -0.664%]
", "Arm 0_52

metric_2: 0.878% [0.208%, 1.547%]
", "Arm 0_53

metric_2: -0.911% [-1.573%, -0.249%]
", "Arm 0_54

metric_2: -1.267% [-1.920%, -0.613%]
", "Arm 0_55

metric_2: -1.055% [-1.720%, -0.390%]
", "Arm 0_56

metric_2: 1.138% [0.461%, 1.815%]
", "Arm 0_57

metric_2: -3.221% [-3.878%, -2.563%]
", "Arm 0_58

metric_2: -1.454% [-2.122%, -0.787%]
", "Arm 0_59

metric_2: 0.993% [0.320%, 1.665%]
", "Arm 0_6

metric_2: -1.318% [-1.974%, -0.662%]
", "Arm 0_60

metric_2: 2.781% [2.106%, 3.455%]
", "Arm 0_61

metric_2: 10.553% [9.783%, 11.322%]
", "Arm 0_62

metric_2: -3.787% [-4.450%, -3.125%]
", "Arm 0_63

metric_2: -2.738% [-3.397%, -2.080%]
", "Arm 0_7

metric_2: -1.61% [-2.273%, -0.947%]
", "Arm 0_8

metric_2: -2.148% [-2.807%, -1.489%]
", "Arm 0_9

metric_2: -2.794% [-3.447%, -2.142%]
", "Arm status_quo

metric_2: -0.0% [-0.318%, 0.318%]
" ], "type": "scatter", "visible": true, "x": [ "0_0", "0_1", "0_10", "0_11", "0_12", "0_13", "0_14", "0_15", "0_16", "0_17", "0_18", "0_19", "0_2", "0_20", "0_21", "0_22", "0_23", "0_24", "0_25", "0_26", "0_27", "0_28", "0_29", "0_3", "0_30", "0_31", "0_32", "0_33", "0_34", "0_35", "0_36", "0_37", "0_38", "0_39", "0_4", "0_40", "0_41", "0_42", "0_43", "0_44", "0_45", "0_46", "0_47", "0_48", "0_49", "0_5", "0_50", "0_51", "0_52", "0_53", "0_54", "0_55", "0_56", "0_57", "0_58", "0_59", "0_6", "0_60", "0_61", "0_62", "0_63", "0_7", "0_8", "0_9", "status_quo" ], "xaxis": "x", "y": [ -5.682890950120342, -3.5263840662392654, -3.5354295227440935, 2.542378866552539, -0.6285418092402999, -1.0553405972573464, -0.8320014955517723, -2.3193462375707914, 2.58163977526919, 0.32621268542555526, -1.3207797795363851, -3.314816695063182, -0.9921710458941605, -0.06250694414329654, 0.06640049636248754, -0.05540222390315327, -3.1495291448321576, 1.8934882315080965, 0.023529017577203423, 1.0773772731169438, 0.6512642256730126, -2.489527283561809, 5.857858819488534, -0.4484318638610448, -2.712182537468451, -5.792314875460171, 3.1580957792386144, 3.8533176800717563, 2.026262991103158, 0.6187928699607773, 1.3280865831055046, 4.5394243735749775, -0.7404858515521753, 0.10968492064137063, -1.5381048845570704, -3.2865434005114005, -1.7761680789956928, -1.1071508425287317, -1.3739381694523336, 7.5312245476719, -2.1901762525545276, -3.082214357601481, 3.963995297169013, -1.1779086966890597, 0.7516740725998268, -0.772463613395467, 2.196834434560228, -1.3372360602549658, 0.8775653888918572, -0.9111761852726025, -1.2666733248842346, -1.055168296030082, 1.1378418002527275, -3.2209479427630807, -1.4541461401402338, 0.9926717601692114, -1.3177840022000746, 2.7806338967179443, 10.552746740900881, -3.7873810524577847, -2.7384540934203034, -1.6100825481553491, -2.147955641504198, -2.7941662700346765, -0.0 ], "yaxis": "y" }, { "error_y": { "array": [ 0.9032151733520527, 0.9624260346524522, 0.9593909284698995, 1.1347609507759298, 1.0408257278018285, 1.0435860695228432, 1.0711864083714089, 1.0156140087849477, 1.128257833835402, 1.0788991832923005, 1.0706037001289912, 0.9284671425890606, 1.0523829338844282, 1.0903373892267718, 1.0861328996665252, 1.1145865581026473, 1.0063514862723812, 1.0994776629118859, 1.091675914931962, 1.0844593315915725, 1.0017853270032084, 0.9935701990690048, 1.1127508961856405, 1.0668110378139575, 1.0328601627012277, 0.8797268687204242, 1.140518932271723, 1.1417536403033617, 1.0764096893237975, 1.0965566556283748, 1.0984998396729218, 1.102540333829891, 1.0794770222090497, 1.085962611129493, 1.0433251902342626, 0.9820278721895025, 1.0166480934308502, 1.0343546221526094, 1.0382944853121021, 1.1551456071793769, 1.0464931893207308, 1.0023861997986674, 1.0516802404518037, 1.0346896327290815, 1.08327143184596, 1.0698309404523143, 1.0904297395971863, 1.0222860350150513, 1.0810704079796398, 1.0613893596292459, 1.0320103439592427, 1.0528002249874768, 1.0912064123686547, 0.9988930110467326, 1.0571937806819418, 1.1193739547640122, 1.0530800272030827, 1.1303999126786766, 1.180499014903316, 0.9513079288502773, 0.9719033134076178, 1.03791308728753, 1.0588852902469934, 0.9945314913677392, 0.5199803498226312 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "hoverinfo": "text", "legendgroup": "In-sample", "marker": { "color": "rgba(128,177,211,1)" }, "mode": "markers", "name": "In-sample", "showlegend": false, "text": [ "Arm 0_0

metric_1: -17.022% [-17.925%, -16.119%]
", "Arm 0_1

metric_1: -11.473% [-12.436%, -10.511%]
", "Arm 0_10

metric_1: -11.26% [-12.220%, -10.301%]
", "Arm 0_11

metric_1: 2.99% [1.855%, 4.125%]
", "Arm 0_12

metric_1: -3.205% [-4.246%, -2.164%]
", "Arm 0_13

metric_1: -2.828% [-3.871%, -1.784%]
", "Arm 0_14

metric_1: -0.807% [-1.878%, 0.264%]
", "Arm 0_15

metric_1: -7.632% [-8.648%, -6.617%]
", "Arm 0_16

metric_1: 2.493% [1.365%, 3.621%]
", "Arm 0_17

metric_1: -1.148% [-2.227%, -0.069%]
", "Arm 0_18

metric_1: -1.417% [-2.488%, -0.346%]
", "Arm 0_19

metric_1: -14.735% [-15.664%, -13.807%]
", "Arm 0_2

metric_1: -3.211% [-4.263%, -2.158%]
", "Arm 0_20

metric_1: 0.073% [-1.017%, 1.164%]
", "Arm 0_21

metric_1: 0.528% [-0.558%, 1.614%]
", "Arm 0_22

metric_1: 1.746% [0.632%, 2.861%]
", "Arm 0_23

metric_1: -6.404% [-7.411%, -5.398%]
", "Arm 0_24

metric_1: 0.836% [-0.264%, 1.935%]
", "Arm 0_25

metric_1: 0.699% [-0.393%, 1.790%]
", "Arm 0_26

metric_1: 0.319% [-0.766%, 1.403%]
", "Arm 0_27

metric_1: -6.918% [-7.920%, -5.916%]
", "Arm 0_28

metric_1: -7.463% [-8.457%, -6.469%]
", "Arm 0_29

metric_1: 1.48% [0.367%, 2.592%]
", "Arm 0_3

metric_1: -2.25% [-3.317%, -1.183%]
", "Arm 0_30

metric_1: -4.65% [-5.683%, -3.617%]
", "Arm 0_31

metric_1: -19.06% [-19.939%, -18.180%]
", "Arm 0_32

metric_1: 3.344% [2.204%, 4.485%]
", "Arm 0_33

metric_1: 3.298% [2.156%, 4.440%]
", "Arm 0_34

metric_1: -3.845% [-4.922%, -2.769%]
", "Arm 0_35

metric_1: 0.616% [-0.480%, 1.713%]
", "Arm 0_36

metric_1: 0.606% [-0.493%, 1.704%]
", "Arm 0_37

metric_1: 0.801% [-0.301%, 1.904%]
", "Arm 0_38

metric_1: -1.23% [-2.310%, -0.151%]
", "Arm 0_39

metric_1: -0.189% [-1.275%, 0.897%]
", "Arm 0_4

metric_1: -2.877% [-3.920%, -1.834%]
", "Arm 0_40

metric_1: -8.667% [-9.649%, -7.685%]
", "Arm 0_41

metric_1: -5.112% [-6.129%, -4.096%]
", "Arm 0_42

metric_1: -4.866% [-5.900%, -3.831%]
", "Arm 0_43

metric_1: -3.621% [-4.659%, -2.582%]
", "Arm 0_44

metric_1: 4.416% [3.261%, 5.571%]
", "Arm 0_45

metric_1: -3.094% [-4.140%, -2.047%]
", "Arm 0_46

metric_1: -6.987% [-7.990%, -5.985%]
", "Arm 0_47

metric_1: -3.456% [-4.507%, -2.404%]
", "Arm 0_48

metric_1: -4.487% [-5.521%, -3.452%]
", "Arm 0_49

metric_1: -0.276% [-1.360%, 0.807%]
", "Arm 0_5

metric_1: -2.093% [-3.163%, -1.023%]
", "Arm 0_50

metric_1: 0.34% [-0.750%, 1.431%]
", "Arm 0_51

metric_1: -4.611% [-5.633%, -3.589%]
", "Arm 0_52

metric_1: -0.757% [-1.838%, 0.324%]
", "Arm 0_53

metric_1: -2.166% [-3.228%, -1.105%]
", "Arm 0_54

metric_1: -3.611% [-4.643%, -2.579%]
", "Arm 0_55

metric_1: -3.053% [-4.106%, -2.000%]
", "Arm 0_56

metric_1: 0.115% [-0.976%, 1.206%]
", "Arm 0_57

metric_1: -8.012% [-9.011%, -7.013%]
", "Arm 0_58

metric_1: -2.444% [-3.501%, -1.387%]
", "Arm 0_59

metric_1: 2.105% [0.986%, 3.225%]
", "Arm 0_6

metric_1: -3.549% [-4.602%, -2.496%]
", "Arm 0_60

metric_1: 2.966% [1.836%, 4.097%]
", "Arm 0_61

metric_1: 5.393% [4.212%, 6.573%]
", "Arm 0_62

metric_1: -13.026% [-13.978%, -12.075%]
", "Arm 0_63

metric_1: -9.619% [-10.591%, -8.647%]
", "Arm 0_7

metric_1: -4.118% [-5.156%, -3.080%]
", "Arm 0_8

metric_1: -2.464% [-3.523%, -1.405%]
", "Arm 0_9

metric_1: -7.957% [-8.951%, -6.962%]
", "Arm status_quo

metric_1: -0.0% [-0.520%, 0.520%]
" ], "type": "scatter", "visible": true, "x": [ "0_0", "0_1", "0_10", "0_11", "0_12", "0_13", "0_14", "0_15", "0_16", "0_17", "0_18", "0_19", "0_2", "0_20", "0_21", "0_22", "0_23", "0_24", "0_25", "0_26", "0_27", "0_28", "0_29", "0_3", "0_30", "0_31", "0_32", "0_33", "0_34", "0_35", "0_36", "0_37", "0_38", "0_39", "0_4", "0_40", "0_41", "0_42", "0_43", "0_44", "0_45", "0_46", "0_47", "0_48", "0_49", "0_5", "0_50", "0_51", "0_52", "0_53", "0_54", "0_55", "0_56", "0_57", "0_58", "0_59", "0_6", "0_60", "0_61", "0_62", "0_63", "0_7", "0_8", "0_9", "status_quo" ], "xaxis": "x2", "y": [ -17.022045315104542, -11.473276363575263, -11.260309485449103, 2.9899364287480865, -3.2049806778237757, -2.827647808086468, -0.8072357684589017, -7.632201738791567, 2.4931910046363135, -1.1483177933819395, -1.4169105153422092, -14.735353419604191, -3.2108750242520334, 0.07332105300986574, 0.52794229128138, 1.7461298706241524, -6.404301715961119, 0.8357526716703064, 0.6988214644260046, 0.3185581570295636, -6.918118515914437, -7.462960459390675, 1.4797223030792654, -2.250057847895754, -4.649788400829341, -19.059723742205165, 3.3442377220374424, 3.2977920613814318, -3.845344844296581, 0.6160918095707203, 0.605873558955114, 0.8012482568804604, -1.2301194692314896, -0.18910424701708023, -2.8771572632426596, -8.666686434035148, -5.1123185935816275, -4.86569050603379, -3.620616821502621, 4.416040423353074, -3.093977759193243, -6.987479150965008, -3.45561072825807, -4.486522144334903, -0.27632651192689084, -2.092939160334957, 0.34040916740729205, -4.6111158309365, -0.7572230824447213, -2.1663524739549853, -3.611088388568022, -3.053027739941141, 0.11513485293184113, -8.011705772759294, -2.4442287384702572, 2.1052356784416553, -3.5492459592908614, 2.966172129108271, 5.392515165139946, -13.026404058841306, -9.619176461732291, -4.118044655666446, -2.4636655602657878, -7.956569241984238, -0.0 ], "yaxis": "y2" } ], "layout": { "annotations": [ { "font": { "size": 10 }, "showarrow": false, "text": "metric_2", "x": 0.2375, "xanchor": "center", "xref": "paper", "y": 1.0, "yanchor": "bottom", "yref": "paper" }, { "font": { "size": 10 }, "showarrow": false, "text": "metric_1", "x": 0.7625, "xanchor": "center", "xref": "paper", "y": 1.0, "yanchor": "bottom", "yref": "paper" }, { "font": { "size": 10 }, "showarrow": false, "text": "Predicted Outcomes", "x": 0.5, "xanchor": "center", "xref": "paper", "y": 1.4, "yanchor": "middle", "yref": "paper" }, { "font": { "size": 10 }, "showarrow": false, "text": "Sort By", "x": 0.05, "xanchor": "left", "xref": "paper", "y": 1.4, "yanchor": "middle", "yref": "paper" } ], "font": { "size": 10 }, "height": 300, "hovermode": "closest", "legend": { "orientation": "h", "x": 0, "xanchor": "left", "y": 1.2, "yanchor": "middle" }, "margin": { "t": 0 }, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "sequentialminus": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "updatemenus": [ { "buttons": [ { "args": [ { "xaxis.categoryarray": [ "status_quo", "0_0", "0_1", "0_2", "0_3", "0_4", "0_5", "0_6", "0_7", "0_8", "0_9", "0_10", "0_11", "0_12", "0_13", "0_14", "0_15", "0_16", "0_17", "0_18", "0_19", "0_20", "0_21", "0_22", "0_23", "0_24", "0_25", "0_26", "0_27", "0_28", "0_29", "0_30", "0_31", "0_32", "0_33", "0_34", "0_35", "0_36", "0_37", "0_38", "0_39", "0_40", "0_41", "0_42", "0_43", "0_44", "0_45", "0_46", "0_47", "0_48", "0_49", "0_50", "0_51", "0_52", "0_53", "0_54", "0_55", "0_56", "0_57", "0_58", "0_59", "0_60", "0_61", "0_62", "0_63" ], "xaxis.categoryorder": "array", "xaxis2.categoryarray": [ "status_quo", "0_0", "0_1", "0_2", "0_3", "0_4", "0_5", "0_6", "0_7", "0_8", "0_9", "0_10", "0_11", "0_12", "0_13", "0_14", "0_15", "0_16", "0_17", "0_18", "0_19", "0_20", "0_21", "0_22", "0_23", "0_24", "0_25", "0_26", "0_27", "0_28", "0_29", "0_30", "0_31", "0_32", "0_33", "0_34", "0_35", "0_36", "0_37", "0_38", "0_39", "0_40", "0_41", "0_42", "0_43", "0_44", "0_45", "0_46", "0_47", "0_48", "0_49", "0_50", "0_51", "0_52", "0_53", "0_54", "0_55", "0_56", "0_57", "0_58", "0_59", "0_60", "0_61", "0_62", "0_63" ], "xaxis2.categoryorder": "array" } ], "label": "Name", "method": "relayout" }, { "args": [ { "xaxis.categoryarray": [ "0_31", "0_0", "0_62", "0_10", "0_1", "0_19", "0_40", "0_57", "0_23", "0_46", "0_9", "0_63", "0_30", "0_28", "0_15", "0_45", "0_8", "0_41", "0_7", "0_4", "0_58", "0_43", "0_51", "0_18", "0_6", "0_54", "0_48", "0_42", "0_13", "0_55", "0_2", "0_53", "0_14", "0_5", "0_38", "0_12", "0_3", "0_20", "0_22", "status_quo", "0_25", "0_21", "0_39", "0_17", "0_35", "0_27", "0_49", "0_52", "0_59", "0_26", "0_56", "0_36", "0_24", "0_34", "0_50", "0_11", "0_16", "0_60", "0_32", "0_33", "0_47", "0_37", "0_29", "0_44", "0_61" ], "xaxis.categoryorder": "array", "xaxis2.categoryarray": [ "0_31", "0_0", "0_19", "0_62", "0_1", "0_10", "0_63", "0_40", "0_57", "0_9", "0_15", "0_28", "0_46", "0_27", "0_23", "0_41", "0_42", "0_30", "0_51", "0_48", "0_7", "0_34", "0_43", "0_54", "0_6", "0_47", "0_2", "0_12", "0_45", "0_55", "0_4", "0_13", "0_8", "0_58", "0_3", "0_53", "0_5", "0_18", "0_38", "0_17", "0_14", "0_52", "0_49", "0_39", "status_quo", "0_20", "0_56", "0_26", "0_50", "0_21", "0_36", "0_35", "0_25", "0_37", "0_24", "0_29", "0_22", "0_59", "0_16", "0_60", "0_11", "0_33", "0_32", "0_44", "0_61" ], "xaxis2.categoryorder": "array" } ], "label": "Effect Size", "method": "relayout" } ], "x": 0.15, "xanchor": "left", "y": 1.4, "yanchor": "middle" } ], "width": 950, "xaxis": { "anchor": "y", "categoryarray": [ "status_quo", "0_0", "0_1", "0_2", "0_3", "0_4", "0_5", "0_6", "0_7", "0_8", "0_9", "0_10", "0_11", "0_12", "0_13", "0_14", "0_15", "0_16", "0_17", "0_18", "0_19", "0_20", "0_21", "0_22", "0_23", "0_24", "0_25", "0_26", "0_27", "0_28", "0_29", "0_30", "0_31", "0_32", "0_33", "0_34", "0_35", "0_36", "0_37", "0_38", "0_39", "0_40", "0_41", "0_42", "0_43", "0_44", "0_45", "0_46", "0_47", "0_48", "0_49", "0_50", "0_51", "0_52", "0_53", "0_54", "0_55", "0_56", "0_57", "0_58", "0_59", "0_60", "0_61", "0_62", "0_63" ], "categoryorder": "array", "domain": [ 0.0, 0.475 ], "type": "category" }, "xaxis2": { "anchor": "y2", "categoryarray": [ "status_quo", "0_0", "0_1", "0_2", "0_3", "0_4", "0_5", "0_6", "0_7", "0_8", "0_9", "0_10", "0_11", "0_12", "0_13", "0_14", "0_15", "0_16", "0_17", "0_18", "0_19", "0_20", "0_21", "0_22", "0_23", "0_24", "0_25", "0_26", "0_27", "0_28", "0_29", "0_30", "0_31", "0_32", "0_33", "0_34", "0_35", "0_36", "0_37", "0_38", "0_39", "0_40", "0_41", "0_42", "0_43", "0_44", "0_45", "0_46", "0_47", "0_48", "0_49", "0_50", "0_51", "0_52", "0_53", "0_54", "0_55", "0_56", "0_57", "0_58", "0_59", "0_60", "0_61", "0_62", "0_63" ], "categoryorder": "array", "domain": [ 0.525, 1.0 ], "type": "category" }, "yaxis": { "anchor": "x", "domain": [ 0.0, 1.0 ], "ticksuffix": "%", "zerolinecolor": "red" }, "yaxis2": { "anchor": "x2", "domain": [ 0.0, 1.0 ], "ticksuffix": "%", "zerolinecolor": "red" } } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "render(tile_fitted(gp, rel=True))" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "linkText": "Export to plot.ly", "plotlyServerURL": "https://plot.ly", "showLink": false }, "data": [ { "error_x": { "array": [ 0.9105274690527578, 0.9725977092158347, 0.9694448615326413, 1.1526497268980418, 1.056129089525654, 1.0581150790455403, 1.0853729636494531, 1.025297982327309, 1.1509354346821015, 1.0946224534113276, 1.080994735603789, 0.9349800939958878, 1.0669782977880082, 1.1063382644746038, 1.109747945764228, 1.1312428838918263, 1.0151345627364643, 1.113745664637434, 1.1070645021428274, 1.098298041303658, 1.011825279703854, 1.004076551317031, 1.1247628081965457, 1.0803499998271806, 1.0451840940805415, 0.8854990204077908, 1.1551313262975833, 1.1582540625351196, 1.088250532740983, 1.1152606223383992, 1.1132227750465398, 1.1136443399827738, 1.0928257948335294, 1.098342295748459, 1.0562137210185467, 0.9929499124720941, 1.0275183054522712, 1.0461406507525532, 1.0507233488011212, 1.1693056777789836, 1.0564300559878494, 1.012746475490335, 1.060503126319525, 1.0453978622119475, 1.0972242260284542, 1.0810868080421863, 1.103084437746675, 1.0346109289273437, 1.0925727929561801, 1.0737167310801996, 1.042945064542918, 1.0636626539145406, 1.108765950358755, 1.0083792696638647, 1.069964804645196, 1.1324679783490406, 1.0646727177942992, 1.149093505549881, 1.19450723179394, 0.9584867213686696, 0.9797729070203455, 1.048287897726107, 1.0699714924908204, 1.0044799847095478, 0.5205646439608032 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "error_y": { "array": [ 0.6639175585103954, 0.6630929411298598, 0.6719874423233675, 0.6935680367786019, 0.6821107678411614, 0.6721157287254815, 0.6675598394585217, 0.6588395471546759, 0.7005079335820925, 0.6734693768172479, 0.6678033173208016, 0.6639242874696609, 0.6729502923953926, 0.6673739566339196, 0.6808476824923786, 0.675618453499239, 0.6630338514495195, 0.6814341339540937, 0.6732139305974362, 0.678993016475866, 0.6717812932415321, 0.6668482051137559, 0.7215454438634975, 0.6740882429740953, 0.6694676526234568, 0.6554367403862607, 0.6871145937679641, 0.6973627667199596, 0.677839082032618, 0.6656099293584739, 0.6853610253860684, 0.7127045342173167, 0.6774929784721044, 0.6697791158632792, 0.665276789680729, 0.6609617898538276, 0.6573670880847166, 0.6812573532252995, 0.6669304645971238, 0.7456805630690589, 0.6722619172607951, 0.6674666794827154, 0.71101954162863, 0.6581471114518421, 0.681784654518447, 0.6708890082180987, 0.6891795059009299, 0.6846723076532707, 0.6778079742730775, 0.6716150419025536, 0.6624185510214552, 0.6736751066980704, 0.6894763100114708, 0.6659339720000773, 0.6774968737338238, 0.6814011092512952, 0.6647910226422914, 0.6882834695714566, 0.7808801986391988, 0.6703813570659151, 0.666212972377905, 0.6724199160716084, 0.6676592507580481, 0.6614165175994707, 0.3181115760489757 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "hoverinfo": "text", "marker": { "color": "rgba(128,177,211,1)" }, "mode": "markers", "name": "In-sample", "showlegend": true, "text": [ "Arm 0_0

metric_1: -17.159% [-18.070%, -16.249%]
metric_2: -5.747% [-6.411%, -5.083%]
Parameterization has too many items to render on hover (15).", "Arm 0_1

metric_1: -11.573% [-12.546%, -10.601%]
metric_2: -3.561% [-4.224%, -2.897%]
Parameterization has too many items to render on hover (15).", "Arm 0_10

metric_1: -11.361% [-12.331%, -10.392%]
metric_2: -3.61% [-4.282%, -2.938%]
Parameterization has too many items to render on hover (15).", "Arm 0_11

metric_1: 3.038% [1.885%, 4.190%]
metric_2: 2.514% [1.820%, 3.207%]
Parameterization has too many items to render on hover (15).", "Arm 0_12

metric_1: -3.07% [-4.126%, -2.013%]
metric_2: -0.597% [-1.279%, 0.086%]
Parameterization has too many items to render on hover (15).", "Arm 0_13

metric_1: -2.865% [-3.923%, -1.806%]
metric_2: -1.012% [-1.684%, -0.340%]
Parameterization has too many items to render on hover (15).", "Arm 0_14

metric_1: -0.817% [-1.903%, 0.268%]
metric_2: -0.818% [-1.486%, -0.151%]
Parameterization has too many items to render on hover (15).", "Arm 0_15

metric_1: -7.78% [-8.806%, -6.755%]
metric_2: -2.407% [-3.066%, -1.748%]
Parameterization has too many items to render on hover (15).", "Arm 0_16

metric_1: 2.475% [1.324%, 3.626%]
metric_2: 2.558% [1.857%, 3.258%]
Parameterization has too many items to render on hover (15).", "Arm 0_17

metric_1: -1.196% [-2.291%, -0.102%]
metric_2: 0.348% [-0.325%, 1.022%]
Parameterization has too many items to render on hover (15).", "Arm 0_18

metric_1: -1.36% [-2.441%, -0.279%]
metric_2: -1.311% [-1.979%, -0.644%]
Parameterization has too many items to render on hover (15).", "Arm 0_19

metric_1: -14.817% [-15.752%, -13.882%]
metric_2: -3.338% [-4.002%, -2.674%]
Parameterization has too many items to render on hover (15).", "Arm 0_2

metric_1: -3.272% [-4.339%, -2.205%]
metric_2: -1.007% [-1.680%, -0.334%]
Parameterization has too many items to render on hover (15).", "Arm 0_20

metric_1: 0.089% [-1.018%, 1.195%]
metric_2: -0.069% [-0.736%, 0.599%]
Parameterization has too many items to render on hover (15).", "Arm 0_21

metric_1: 0.536% [-0.574%, 1.646%]
metric_2: 0.104% [-0.577%, 0.785%]
Parameterization has too many items to render on hover (15).", "Arm 0_22

metric_1: 1.815% [0.683%, 2.946%]
metric_2: -0.072% [-0.748%, 0.604%]
Parameterization has too many items to render on hover (15).", "Arm 0_23

metric_1: -6.471% [-7.486%, -5.456%]
metric_2: -3.188% [-3.851%, -2.525%]
Parameterization has too many items to render on hover (15).", "Arm 0_24

metric_1: 0.888% [-0.226%, 2.001%]
metric_2: 1.96% [1.279%, 2.642%]
Parameterization has too many items to render on hover (15).", "Arm 0_25

metric_1: 0.729% [-0.378%, 1.836%]
metric_2: 0.013% [-0.661%, 0.686%]
Parameterization has too many items to render on hover (15).", "Arm 0_26

metric_1: 0.351% [-0.747%, 1.449%]
metric_2: 1.12% [0.441%, 1.799%]
Parameterization has too many items to render on hover (15).", "Arm 0_27

metric_1: -6.985% [-7.997%, -5.973%]
metric_2: 0.616% [-0.056%, 1.287%]
Parameterization has too many items to render on hover (15).", "Arm 0_28

metric_1: -7.506% [-8.510%, -6.502%]
metric_2: -2.55% [-3.217%, -1.884%]
Parameterization has too many items to render on hover (15).", "Arm 0_29

metric_1: 1.577% [0.452%, 2.702%]
metric_2: 5.965% [5.243%, 6.686%]
Parameterization has too many items to render on hover (15).", "Arm 0_3

metric_1: -2.214% [-3.295%, -1.134%]
metric_2: -0.421% [-1.095%, 0.254%]
Parameterization has too many items to render on hover (15).", "Arm 0_30

metric_1: -4.596% [-5.641%, -3.550%]
metric_2: -2.727% [-3.396%, -2.057%]
Parameterization has too many items to render on hover (15).", "Arm 0_31

metric_1: -19.189% [-20.075%, -18.304%]
metric_2: -5.859% [-6.515%, -5.204%]
Parameterization has too many items to render on hover (15).", "Arm 0_32

metric_1: 3.429% [2.274%, 4.584%]
metric_2: 3.197% [2.510%, 3.884%]
Parameterization has too many items to render on hover (15).", "Arm 0_33

metric_1: 3.371% [2.213%, 4.529%]
metric_2: 3.95% [3.253%, 4.648%]
Parameterization has too many items to render on hover (15).", "Arm 0_34

metric_1: -3.89% [-4.978%, -2.802%]
metric_2: 2.057% [1.379%, 2.735%]
Parameterization has too many items to render on hover (15).", "Arm 0_35

metric_1: 0.614% [-0.501%, 1.729%]
metric_2: 0.653% [-0.013%, 1.318%]
Parameterization has too many items to render on hover (15).", "Arm 0_36

metric_1: 0.59% [-0.523%, 1.703%]
metric_2: 1.287% [0.601%, 1.972%]
Parameterization has too many items to render on hover (15).", "Arm 0_37

metric_1: 0.865% [-0.248%, 1.979%]
metric_2: 4.621% [3.908%, 5.334%]
Parameterization has too many items to render on hover (15).", "Arm 0_38

metric_1: -1.268% [-2.361%, -0.175%]
metric_2: -0.762% [-1.440%, -0.085%]
Parameterization has too many items to render on hover (15).", "Arm 0_39

metric_1: -0.169% [-1.267%, 0.930%]
metric_2: 0.124% [-0.546%, 0.793%]
Parameterization has too many items to render on hover (15).", "Arm 0_4

metric_1: -2.893% [-3.949%, -1.837%]
metric_2: -1.543% [-2.208%, -0.878%]
Parameterization has too many items to render on hover (15).", "Arm 0_40

metric_1: -8.624% [-9.617%, -7.631%]
metric_2: -3.265% [-3.926%, -2.604%]
Parameterization has too many items to render on hover (15).", "Arm 0_41

metric_1: -5.157% [-6.184%, -4.129%]
metric_2: -1.741% [-2.399%, -1.084%]
Parameterization has too many items to render on hover (15).", "Arm 0_42

metric_1: -4.878% [-5.924%, -3.832%]
metric_2: -1.126% [-1.807%, -0.444%]
Parameterization has too many items to render on hover (15).", "Arm 0_43

metric_1: -3.655% [-4.706%, -2.605%]
metric_2: -1.381% [-2.048%, -0.714%]
Parameterization has too many items to render on hover (15).", "Arm 0_44

metric_1: 4.483% [3.314%, 5.652%]
metric_2: 7.652% [6.906%, 8.397%]
Parameterization has too many items to render on hover (15).", "Arm 0_45

metric_1: -3.06% [-4.117%, -2.004%]
metric_2: -2.202% [-2.874%, -1.529%]
Parameterization has too many items to render on hover (15).", "Arm 0_46

metric_1: -6.922% [-7.934%, -5.909%]
metric_2: -3.063% [-3.731%, -2.396%]
Parameterization has too many items to render on hover (15).", "Arm 0_47

metric_1: -3.478% [-4.539%, -2.418%]
metric_2: 4.013% [3.302%, 4.724%]
Parameterization has too many items to render on hover (15).", "Arm 0_48

metric_1: -4.546% [-5.591%, -3.501%]
metric_2: -1.169% [-1.827%, -0.511%]
Parameterization has too many items to render on hover (15).", "Arm 0_49

metric_1: -0.305% [-1.403%, 0.792%]
metric_2: 0.753% [0.071%, 1.434%]
Parameterization has too many items to render on hover (15).", "Arm 0_5

metric_1: -2.069% [-3.150%, -0.988%]
metric_2: -0.76% [-1.431%, -0.090%]
Parameterization has too many items to render on hover (15).", "Arm 0_50

metric_1: 0.367% [-0.736%, 1.470%]
metric_2: 2.231% [1.542%, 2.920%]
Parameterization has too many items to render on hover (15).", "Arm 0_51

metric_1: -4.597% [-5.631%, -3.562%]
metric_2: -1.371% [-2.056%, -0.687%]
Parameterization has too many items to render on hover (15).", "Arm 0_52

metric_1: -0.715% [-1.807%, 0.378%]
metric_2: 0.898% [0.220%, 1.576%]
Parameterization has too many items to render on hover (15).", "Arm 0_53

metric_1: -2.187% [-3.261%, -1.113%]
metric_2: -0.924% [-1.595%, -0.252%]
Parameterization has too many items to render on hover (15).", "Arm 0_54

metric_1: -3.62% [-4.663%, -2.577%]
metric_2: -1.266% [-1.928%, -0.603%]
Parameterization has too many items to render on hover (15).", "Arm 0_55

metric_1: -3.085% [-4.149%, -2.022%]
metric_2: -1.056% [-1.729%, -0.382%]
Parameterization has too many items to render on hover (15).", "Arm 0_56

metric_1: 0.065% [-1.043%, 1.174%]
metric_2: 1.104% [0.414%, 1.793%]
Parameterization has too many items to render on hover (15).", "Arm 0_57

metric_1: -8.045% [-9.053%, -7.036%]
metric_2: -3.231% [-3.897%, -2.565%]
Parameterization has too many items to render on hover (15).", "Arm 0_58

metric_1: -2.417% [-3.486%, -1.347%]
metric_2: -1.461% [-2.139%, -0.784%]
Parameterization has too many items to render on hover (15).", "Arm 0_59

metric_1: 2.155% [1.023%, 3.288%]
metric_2: 0.992% [0.310%, 1.673%]
Parameterization has too many items to render on hover (15).", "Arm 0_6

metric_1: -3.551% [-4.616%, -2.486%]
metric_2: -1.321% [-1.986%, -0.656%]
Parameterization has too many items to render on hover (15).", "Arm 0_60

metric_1: 3.0% [1.850%, 4.149%]
metric_2: 2.8% [2.112%, 3.488%]
Parameterization has too many items to render on hover (15).", "Arm 0_61

metric_1: 5.54% [4.346%, 6.735%]
metric_2: 10.809% [10.028%, 11.590%]
Parameterization has too many items to render on hover (15).", "Arm 0_62

metric_1: -13.137% [-14.095%, -12.178%]
metric_2: -3.839% [-4.509%, -3.169%]
Parameterization has too many items to render on hover (15).", "Arm 0_63

metric_1: -9.683% [-10.662%, -8.703%]
metric_2: -2.765% [-3.432%, -2.099%]
Parameterization has too many items to render on hover (15).", "Arm 0_7

metric_1: -4.089% [-5.138%, -3.041%]
metric_2: -1.576% [-2.248%, -0.903%]
Parameterization has too many items to render on hover (15).", "Arm 0_8

metric_1: -2.46% [-3.530%, -1.390%]
metric_2: -2.179% [-2.846%, -1.511%]
Parameterization has too many items to render on hover (15).", "Arm 0_9

metric_1: -7.98% [-8.985%, -6.976%]
metric_2: -2.79% [-3.451%, -2.128%]
Parameterization has too many items to render on hover (15).", "Arm status_quo

metric_1: -0.0% [-0.521%, 0.521%]
metric_2: -0.0% [-0.318%, 0.318%]
Parameterization has too many items to render on hover (15)." ], "type": "scatter", "visible": false, "x": [ -17.15914516790823, -11.573417823939199, -11.361067490605143, 3.037717284235046, -3.069542768219942, -2.864577054527247, -0.8171324453854826, -7.780377203320161, 2.475352747757105, -1.1963952539982303, -1.3598025263922875, -14.817429080033943, -3.2723491422755444, 0.08873184010870934, 0.5359430531455063, 1.8145980354381517, -6.471358153849177, 0.8877430970467538, 0.7286746192275023, 0.3510369131998831, -6.984895697391649, -7.505623786753492, 1.5767742612758522, -2.2143528351212733, -4.595608658658678, -19.18909654051259, 3.4292680540776312, 3.3710456099492796, -3.889937358175768, 0.6139708178717305, 0.5900679364090252, 0.8653940325415319, -1.2680473963845287, -0.1686822762476349, -2.893033151686222, -8.623721510464856, -5.156592129047242, -4.878241533976907, -3.655342272989576, 4.483189117418239, -3.0603072946347587, -6.921569399104486, -3.4781006474949763, -4.545929170059967, -0.305361571312272, -2.068706090724009, 0.36738131081906034, -4.596829363692114, -0.7145714514809897, -2.1870440120521244, -3.619802243175367, -3.0854247199357574, 0.06528116501171306, -8.04466137206524, -2.416515121000999, 2.1550821626518215, -3.550888972902983, 2.999501801038642, 5.540352816509654, -13.136826425368891, -9.682667832954975, -4.089456620979975, -2.4600469634137005, -7.980395623816304, -0.0 ], "y": [ -5.746731056831749, -3.560535696444575, -3.610268946914751, 2.5135502228088797, -0.5965513852397577, -1.0116808329390385, -0.8182560448021005, -2.4071834175723335, 2.5577489299929255, 0.348192434742279, -1.3114357729283859, -3.3382598457223502, -1.0068955594731426, -0.06867630083734555, 0.10424166292801242, -0.07210829452142886, -3.188401274811581, 1.960086317210278, 0.012674904654773453, 1.1203410785772772, 0.6155055435112974, -2.5504391581007666, 5.964668435375144, -0.4205314884998786, -2.726644960728147, -5.859418405279977, 3.1966830537300392, 3.950198735676712, 2.0568967995693455, 0.652872009905956, 1.286780707668062, 4.62085929335188, -0.7622630447726769, 0.123503526062618, -1.5431517157186252, -3.2650236570623457, -1.7414179188197445, -1.1257173899572386, -1.3809544654506094, 7.651708533863019, -2.2015816842718356, -3.063188080332386, 4.012625447421733, -1.168849751600452, 0.7526902410881314, -0.7604599707221315, 2.2308412374353317, -1.371441760784444, 0.8978911718791985, -0.9236769844543923, -1.2656161965636414, -1.0555488550285586, 1.1039153988973123, -3.2310973947137103, -1.4614566797717248, 0.9918050829129264, -1.3210922238627325, 2.8001328033355772, 10.809115959705165, -3.839017981482167, -2.765325873487098, -1.5757183832623392, -2.1787372563938, -2.789593215553577, -0.0 ] }, { "error_x": { "array": [ 0.9032151733520527, 0.9624260346524522, 0.9593909284698995, 1.1347609507759298, 1.0408257278018285, 1.0435860695228432, 1.0711864083714089, 1.0156140087849477, 1.128257833835402, 1.0788991832923005, 1.0706037001289912, 0.9284671425890606, 1.0523829338844282, 1.0903373892267718, 1.0861328996665252, 1.1145865581026473, 1.0063514862723812, 1.0994776629118859, 1.091675914931962, 1.0844593315915725, 1.0017853270032084, 0.9935701990690048, 1.1127508961856405, 1.0668110378139575, 1.0328601627012277, 0.8797268687204242, 1.140518932271723, 1.1417536403033617, 1.0764096893237975, 1.0965566556283748, 1.0984998396729218, 1.102540333829891, 1.0794770222090497, 1.085962611129493, 1.0433251902342626, 0.9820278721895025, 1.0166480934308502, 1.0343546221526094, 1.0382944853121021, 1.1551456071793769, 1.0464931893207308, 1.0023861997986674, 1.0516802404518037, 1.0346896327290815, 1.08327143184596, 1.0698309404523143, 1.0904297395971863, 1.0222860350150513, 1.0810704079796398, 1.0613893596292459, 1.0320103439592427, 1.0528002249874768, 1.0912064123686547, 0.9988930110467326, 1.0571937806819418, 1.1193739547640122, 1.0530800272030827, 1.1303999126786766, 1.180499014903316, 0.9513079288502773, 0.9719033134076178, 1.03791308728753, 1.0588852902469934, 0.9945314913677392, 0.5199803498226312 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "error_y": { "array": [ 0.6547387343317975, 0.6532033115837151, 0.6609466315731528, 0.6813554440847576, 0.6694705491295897, 0.6591303137747925, 0.6563086390178189, 0.6508039407989954, 0.6839216136881153, 0.6604406759568969, 0.6601787231575951, 0.6565366929811817, 0.6613093240694986, 0.6561140874958142, 0.6633706122047732, 0.6637799634962176, 0.6555405967295402, 0.6704566164608853, 0.66273573692496, 0.6685871803108022, 0.6628627361853555, 0.6572002148831203, 0.712446900341419, 0.6636937933944925, 0.6586242603445244, 0.6475500126287197, 0.6773760670047011, 0.6859237814996093, 0.6688072097037889, 0.6516465341587597, 0.6738024004862555, 0.7036162417194723, 0.6668939573219893, 0.6610547090506331, 0.6553518681013905, 0.6508051033775936, 0.64797757405811, 0.6713714106339492, 0.656723211631772, 0.7346090086077302, 0.6642524778308558, 0.6574536033119194, 0.7031723757695748, 0.6494774398002832, 0.6704885519899918, 0.6622056608350378, 0.6797639514603396, 0.6735895146091605, 0.6692892385552365, 0.6620882696925839, 0.6531942830594355, 0.6650013545172943, 0.6769982685390781, 0.6574731612170758, 0.6674033123702455, 0.6722373506713744, 0.6559893752039715, 0.6748097809173098, 0.7697444470145935, 0.6626605822603224, 0.6581382444319371, 0.663286850538724, 0.658955602798785, 0.6526018616070007, 0.3176861573292605 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "hoverinfo": "text", "marker": { "color": "rgba(128,177,211,1)" }, "mode": "markers", "name": "In-sample", "showlegend": true, "text": [ "Arm 0_0

metric_1: -17.022% [-17.925%, -16.119%]
metric_2: -5.683% [-6.338%, -5.028%]
Parameterization has too many items to render on hover (15).", "Arm 0_1

metric_1: -11.473% [-12.436%, -10.511%]
metric_2: -3.526% [-4.180%, -2.873%]
Parameterization has too many items to render on hover (15).", "Arm 0_10

metric_1: -11.26% [-12.220%, -10.301%]
metric_2: -3.535% [-4.196%, -2.874%]
Parameterization has too many items to render on hover (15).", "Arm 0_11

metric_1: 2.99% [1.855%, 4.125%]
metric_2: 2.542% [1.861%, 3.224%]
Parameterization has too many items to render on hover (15).", "Arm 0_12

metric_1: -3.205% [-4.246%, -2.164%]
metric_2: -0.629% [-1.298%, 0.041%]
Parameterization has too many items to render on hover (15).", "Arm 0_13

metric_1: -2.828% [-3.871%, -1.784%]
metric_2: -1.055% [-1.714%, -0.396%]
Parameterization has too many items to render on hover (15).", "Arm 0_14

metric_1: -0.807% [-1.878%, 0.264%]
metric_2: -0.832% [-1.488%, -0.176%]
Parameterization has too many items to render on hover (15).", "Arm 0_15

metric_1: -7.632% [-8.648%, -6.617%]
metric_2: -2.319% [-2.970%, -1.669%]
Parameterization has too many items to render on hover (15).", "Arm 0_16

metric_1: 2.493% [1.365%, 3.621%]
metric_2: 2.582% [1.898%, 3.266%]
Parameterization has too many items to render on hover (15).", "Arm 0_17

metric_1: -1.148% [-2.227%, -0.069%]
metric_2: 0.326% [-0.334%, 0.987%]
Parameterization has too many items to render on hover (15).", "Arm 0_18

metric_1: -1.417% [-2.488%, -0.346%]
metric_2: -1.321% [-1.981%, -0.661%]
Parameterization has too many items to render on hover (15).", "Arm 0_19

metric_1: -14.735% [-15.664%, -13.807%]
metric_2: -3.315% [-3.971%, -2.658%]
Parameterization has too many items to render on hover (15).", "Arm 0_2

metric_1: -3.211% [-4.263%, -2.158%]
metric_2: -0.992% [-1.653%, -0.331%]
Parameterization has too many items to render on hover (15).", "Arm 0_20

metric_1: 0.073% [-1.017%, 1.164%]
metric_2: -0.063% [-0.719%, 0.594%]
Parameterization has too many items to render on hover (15).", "Arm 0_21

metric_1: 0.528% [-0.558%, 1.614%]
metric_2: 0.066% [-0.597%, 0.730%]
Parameterization has too many items to render on hover (15).", "Arm 0_22

metric_1: 1.746% [0.632%, 2.861%]
metric_2: -0.055% [-0.719%, 0.608%]
Parameterization has too many items to render on hover (15).", "Arm 0_23

metric_1: -6.404% [-7.411%, -5.398%]
metric_2: -3.15% [-3.805%, -2.494%]
Parameterization has too many items to render on hover (15).", "Arm 0_24

metric_1: 0.836% [-0.264%, 1.935%]
metric_2: 1.893% [1.223%, 2.564%]
Parameterization has too many items to render on hover (15).", "Arm 0_25

metric_1: 0.699% [-0.393%, 1.790%]
metric_2: 0.024% [-0.639%, 0.686%]
Parameterization has too many items to render on hover (15).", "Arm 0_26

metric_1: 0.319% [-0.766%, 1.403%]
metric_2: 1.077% [0.409%, 1.746%]
Parameterization has too many items to render on hover (15).", "Arm 0_27

metric_1: -6.918% [-7.920%, -5.916%]
metric_2: 0.651% [-0.012%, 1.314%]
Parameterization has too many items to render on hover (15).", "Arm 0_28

metric_1: -7.463% [-8.457%, -6.469%]
metric_2: -2.49% [-3.147%, -1.832%]
Parameterization has too many items to render on hover (15).", "Arm 0_29

metric_1: 1.48% [0.367%, 2.592%]
metric_2: 5.858% [5.145%, 6.570%]
Parameterization has too many items to render on hover (15).", "Arm 0_3

metric_1: -2.25% [-3.317%, -1.183%]
metric_2: -0.448% [-1.112%, 0.215%]
Parameterization has too many items to render on hover (15).", "Arm 0_30

metric_1: -4.65% [-5.683%, -3.617%]
metric_2: -2.712% [-3.371%, -2.054%]
Parameterization has too many items to render on hover (15).", "Arm 0_31

metric_1: -19.06% [-19.939%, -18.180%]
metric_2: -5.792% [-6.440%, -5.145%]
Parameterization has too many items to render on hover (15).", "Arm 0_32

metric_1: 3.344% [2.204%, 4.485%]
metric_2: 3.158% [2.481%, 3.835%]
Parameterization has too many items to render on hover (15).", "Arm 0_33

metric_1: 3.298% [2.156%, 4.440%]
metric_2: 3.853% [3.167%, 4.539%]
Parameterization has too many items to render on hover (15).", "Arm 0_34

metric_1: -3.845% [-4.922%, -2.769%]
metric_2: 2.026% [1.357%, 2.695%]
Parameterization has too many items to render on hover (15).", "Arm 0_35

metric_1: 0.616% [-0.480%, 1.713%]
metric_2: 0.619% [-0.033%, 1.270%]
Parameterization has too many items to render on hover (15).", "Arm 0_36

metric_1: 0.606% [-0.493%, 1.704%]
metric_2: 1.328% [0.654%, 2.002%]
Parameterization has too many items to render on hover (15).", "Arm 0_37

metric_1: 0.801% [-0.301%, 1.904%]
metric_2: 4.539% [3.836%, 5.243%]
Parameterization has too many items to render on hover (15).", "Arm 0_38

metric_1: -1.23% [-2.310%, -0.151%]
metric_2: -0.74% [-1.407%, -0.074%]
Parameterization has too many items to render on hover (15).", "Arm 0_39

metric_1: -0.189% [-1.275%, 0.897%]
metric_2: 0.11% [-0.551%, 0.771%]
Parameterization has too many items to render on hover (15).", "Arm 0_4

metric_1: -2.877% [-3.920%, -1.834%]
metric_2: -1.538% [-2.193%, -0.883%]
Parameterization has too many items to render on hover (15).", "Arm 0_40

metric_1: -8.667% [-9.649%, -7.685%]
metric_2: -3.287% [-3.937%, -2.636%]
Parameterization has too many items to render on hover (15).", "Arm 0_41

metric_1: -5.112% [-6.129%, -4.096%]
metric_2: -1.776% [-2.424%, -1.128%]
Parameterization has too many items to render on hover (15).", "Arm 0_42

metric_1: -4.866% [-5.900%, -3.831%]
metric_2: -1.107% [-1.779%, -0.436%]
Parameterization has too many items to render on hover (15).", "Arm 0_43

metric_1: -3.621% [-4.659%, -2.582%]
metric_2: -1.374% [-2.031%, -0.717%]
Parameterization has too many items to render on hover (15).", "Arm 0_44

metric_1: 4.416% [3.261%, 5.571%]
metric_2: 7.531% [6.797%, 8.266%]
Parameterization has too many items to render on hover (15).", "Arm 0_45

metric_1: -3.094% [-4.140%, -2.047%]
metric_2: -2.19% [-2.854%, -1.526%]
Parameterization has too many items to render on hover (15).", "Arm 0_46

metric_1: -6.987% [-7.990%, -5.985%]
metric_2: -3.082% [-3.740%, -2.425%]
Parameterization has too many items to render on hover (15).", "Arm 0_47

metric_1: -3.456% [-4.507%, -2.404%]
metric_2: 3.964% [3.261%, 4.667%]
Parameterization has too many items to render on hover (15).", "Arm 0_48

metric_1: -4.487% [-5.521%, -3.452%]
metric_2: -1.178% [-1.827%, -0.528%]
Parameterization has too many items to render on hover (15).", "Arm 0_49

metric_1: -0.276% [-1.360%, 0.807%]
metric_2: 0.752% [0.081%, 1.422%]
Parameterization has too many items to render on hover (15).", "Arm 0_5

metric_1: -2.093% [-3.163%, -1.023%]
metric_2: -0.772% [-1.435%, -0.110%]
Parameterization has too many items to render on hover (15).", "Arm 0_50

metric_1: 0.34% [-0.750%, 1.431%]
metric_2: 2.197% [1.517%, 2.877%]
Parameterization has too many items to render on hover (15).", "Arm 0_51

metric_1: -4.611% [-5.633%, -3.589%]
metric_2: -1.337% [-2.011%, -0.664%]
Parameterization has too many items to render on hover (15).", "Arm 0_52

metric_1: -0.757% [-1.838%, 0.324%]
metric_2: 0.878% [0.208%, 1.547%]
Parameterization has too many items to render on hover (15).", "Arm 0_53

metric_1: -2.166% [-3.228%, -1.105%]
metric_2: -0.911% [-1.573%, -0.249%]
Parameterization has too many items to render on hover (15).", "Arm 0_54

metric_1: -3.611% [-4.643%, -2.579%]
metric_2: -1.267% [-1.920%, -0.613%]
Parameterization has too many items to render on hover (15).", "Arm 0_55

metric_1: -3.053% [-4.106%, -2.000%]
metric_2: -1.055% [-1.720%, -0.390%]
Parameterization has too many items to render on hover (15).", "Arm 0_56

metric_1: 0.115% [-0.976%, 1.206%]
metric_2: 1.138% [0.461%, 1.815%]
Parameterization has too many items to render on hover (15).", "Arm 0_57

metric_1: -8.012% [-9.011%, -7.013%]
metric_2: -3.221% [-3.878%, -2.563%]
Parameterization has too many items to render on hover (15).", "Arm 0_58

metric_1: -2.444% [-3.501%, -1.387%]
metric_2: -1.454% [-2.122%, -0.787%]
Parameterization has too many items to render on hover (15).", "Arm 0_59

metric_1: 2.105% [0.986%, 3.225%]
metric_2: 0.993% [0.320%, 1.665%]
Parameterization has too many items to render on hover (15).", "Arm 0_6

metric_1: -3.549% [-4.602%, -2.496%]
metric_2: -1.318% [-1.974%, -0.662%]
Parameterization has too many items to render on hover (15).", "Arm 0_60

metric_1: 2.966% [1.836%, 4.097%]
metric_2: 2.781% [2.106%, 3.455%]
Parameterization has too many items to render on hover (15).", "Arm 0_61

metric_1: 5.393% [4.212%, 6.573%]
metric_2: 10.553% [9.783%, 11.322%]
Parameterization has too many items to render on hover (15).", "Arm 0_62

metric_1: -13.026% [-13.978%, -12.075%]
metric_2: -3.787% [-4.450%, -3.125%]
Parameterization has too many items to render on hover (15).", "Arm 0_63

metric_1: -9.619% [-10.591%, -8.647%]
metric_2: -2.738% [-3.397%, -2.080%]
Parameterization has too many items to render on hover (15).", "Arm 0_7

metric_1: -4.118% [-5.156%, -3.080%]
metric_2: -1.61% [-2.273%, -0.947%]
Parameterization has too many items to render on hover (15).", "Arm 0_8

metric_1: -2.464% [-3.523%, -1.405%]
metric_2: -2.148% [-2.807%, -1.489%]
Parameterization has too many items to render on hover (15).", "Arm 0_9

metric_1: -7.957% [-8.951%, -6.962%]
metric_2: -2.794% [-3.447%, -2.142%]
Parameterization has too many items to render on hover (15).", "Arm status_quo

metric_1: -0.0% [-0.520%, 0.520%]
metric_2: -0.0% [-0.318%, 0.318%]
Parameterization has too many items to render on hover (15)." ], "type": "scatter", "visible": true, "x": [ -17.022045315104542, -11.473276363575263, -11.260309485449103, 2.9899364287480865, -3.2049806778237757, -2.827647808086468, -0.8072357684589017, -7.632201738791567, 2.4931910046363135, -1.1483177933819395, -1.4169105153422092, -14.735353419604191, -3.2108750242520334, 0.07332105300986574, 0.52794229128138, 1.7461298706241524, -6.404301715961119, 0.8357526716703064, 0.6988214644260046, 0.3185581570295636, -6.918118515914437, -7.462960459390675, 1.4797223030792654, -2.250057847895754, -4.649788400829341, -19.059723742205165, 3.3442377220374424, 3.2977920613814318, -3.845344844296581, 0.6160918095707203, 0.605873558955114, 0.8012482568804604, -1.2301194692314896, -0.18910424701708023, -2.8771572632426596, -8.666686434035148, -5.1123185935816275, -4.86569050603379, -3.620616821502621, 4.416040423353074, -3.093977759193243, -6.987479150965008, -3.45561072825807, -4.486522144334903, -0.27632651192689084, -2.092939160334957, 0.34040916740729205, -4.6111158309365, -0.7572230824447213, -2.1663524739549853, -3.611088388568022, -3.053027739941141, 0.11513485293184113, -8.011705772759294, -2.4442287384702572, 2.1052356784416553, -3.5492459592908614, 2.966172129108271, 5.392515165139946, -13.026404058841306, -9.619176461732291, -4.118044655666446, -2.4636655602657878, -7.956569241984238, -0.0 ], "y": [ -5.682890950120342, -3.5263840662392654, -3.5354295227440935, 2.542378866552539, -0.6285418092402999, -1.0553405972573464, -0.8320014955517723, -2.3193462375707914, 2.58163977526919, 0.32621268542555526, -1.3207797795363851, -3.314816695063182, -0.9921710458941605, -0.06250694414329654, 0.06640049636248754, -0.05540222390315327, -3.1495291448321576, 1.8934882315080965, 0.023529017577203423, 1.0773772731169438, 0.6512642256730126, -2.489527283561809, 5.857858819488534, -0.4484318638610448, -2.712182537468451, -5.792314875460171, 3.1580957792386144, 3.8533176800717563, 2.026262991103158, 0.6187928699607773, 1.3280865831055046, 4.5394243735749775, -0.7404858515521753, 0.10968492064137063, -1.5381048845570704, -3.2865434005114005, -1.7761680789956928, -1.1071508425287317, -1.3739381694523336, 7.5312245476719, -2.1901762525545276, -3.082214357601481, 3.963995297169013, -1.1779086966890597, 0.7516740725998268, -0.772463613395467, 2.196834434560228, -1.3372360602549658, 0.8775653888918572, -0.9111761852726025, -1.2666733248842346, -1.055168296030082, 1.1378418002527275, -3.2209479427630807, -1.4541461401402338, 0.9926717601692114, -1.3177840022000746, 2.7806338967179443, 10.552746740900881, -3.7873810524577847, -2.7384540934203034, -1.6100825481553491, -2.147955641504198, -2.7941662700346765, -0.0 ] } ], "layout": { "annotations": [ { "showarrow": false, "text": "Show CI", "x": 1.18, "xref": "paper", "y": 0.7, "yanchor": "middle", "yref": "paper" }, { "showarrow": false, "text": "Type", "x": 1.18, "xref": "paper", "y": 0.6, "yanchor": "middle", "yref": "paper" } ], "font": { "size": 10 }, "height": 600, "hovermode": "closest", "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "sequentialminus": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "title": { "text": "Objective Tradeoffs" }, "updatemenus": [ { "buttons": [ { "args": [ { "error_x.thickness": 2, "error_x.width": 4, "error_y.thickness": 2, "error_y.width": 4 } ], "label": "Yes", "method": "restyle" }, { "args": [ { "error_x.thickness": 0, "error_x.width": 0, "error_y.thickness": 0, "error_y.width": 0 } ], "label": "No", "method": "restyle" } ], "x": 1.25, "xanchor": "left", "y": 0.67, "yanchor": "middle" }, { "buttons": [ { "args": [ { "visible": [ false, true ] } ], "label": "Modeled", "method": "restyle" }, { "args": [ { "visible": [ true, false ] } ], "label": "Observed", "method": "restyle" } ], "x": 1.25, "xanchor": "left", "y": 0.57, "yanchor": "middle" } ], "width": 800, "xaxis": { "title": { "text": "metric_1 (%)" }, "zeroline": true, "zerolinecolor": "red" }, "yaxis": { "title": { "text": "metric_2 (%)" }, "zeroline": true, "zerolinecolor": "red" } } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "METRIC_X_AXIS = 'metric_1'\n", "METRIC_Y_AXIS = 'metric_2'\n", "\n", "render(plot_multiple_metrics(\n", " gp,\n", " metric_x=METRIC_X_AXIS,\n", " metric_y=METRIC_Y_AXIS,\n", "))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Candidate Generation\n", "\n", "With our fitted GPEI model, we can optimize EI (Expected Improvement) based on any optimization config.\n", "We can start with our initial optimization config, and aim to simply maximize the playback smoothness, without worrying about the constraint on quality. " ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/home/travis/build/facebook/Ax/ax/models/torch/utils.py:274: UserWarning:\n", "\n", "This overload of nonzero is deprecated:\n", "\tnonzero()\n", "Consider using one of the following signatures instead:\n", "\tnonzero(*, bool as_tuple) (Triggered internally at /pytorch/torch/csrc/utils/python_arg_parser.cpp:882.)\n", "\n" ] } ], "source": [ "unconstrained = gp.gen(\n", " n=3,\n", " optimization_config=OptimizationConfig(\n", " objective=Objective(objective_metric),\n", " )\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's plot the tradeoffs again, but with our new arms. " ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "linkText": "Export to plot.ly", "plotlyServerURL": "https://plot.ly", "showLink": false }, "data": [ { "error_x": { "array": [ 0.9105274690527578, 0.9725977092158347, 0.9694448615326413, 1.1526497268980418, 1.056129089525654, 1.0581150790455403, 1.0853729636494531, 1.025297982327309, 1.1509354346821015, 1.0946224534113276, 1.080994735603789, 0.9349800939958878, 1.0669782977880082, 1.1063382644746038, 1.109747945764228, 1.1312428838918263, 1.0151345627364643, 1.113745664637434, 1.1070645021428274, 1.098298041303658, 1.011825279703854, 1.004076551317031, 1.1247628081965457, 1.0803499998271806, 1.0451840940805415, 0.8854990204077908, 1.1551313262975833, 1.1582540625351196, 1.088250532740983, 1.1152606223383992, 1.1132227750465398, 1.1136443399827738, 1.0928257948335294, 1.098342295748459, 1.0562137210185467, 0.9929499124720941, 1.0275183054522712, 1.0461406507525532, 1.0507233488011212, 1.1693056777789836, 1.0564300559878494, 1.012746475490335, 1.060503126319525, 1.0453978622119475, 1.0972242260284542, 1.0810868080421863, 1.103084437746675, 1.0346109289273437, 1.0925727929561801, 1.0737167310801996, 1.042945064542918, 1.0636626539145406, 1.108765950358755, 1.0083792696638647, 1.069964804645196, 1.1324679783490406, 1.0646727177942992, 1.149093505549881, 1.19450723179394, 0.9584867213686696, 0.9797729070203455, 1.048287897726107, 1.0699714924908204, 1.0044799847095478, 0.5205646439608032 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "error_y": { "array": [ 0.6639175585103954, 0.6630929411298598, 0.6719874423233675, 0.6935680367786019, 0.6821107678411614, 0.6721157287254815, 0.6675598394585217, 0.6588395471546759, 0.7005079335820925, 0.6734693768172479, 0.6678033173208016, 0.6639242874696609, 0.6729502923953926, 0.6673739566339196, 0.6808476824923786, 0.675618453499239, 0.6630338514495195, 0.6814341339540937, 0.6732139305974362, 0.678993016475866, 0.6717812932415321, 0.6668482051137559, 0.7215454438634975, 0.6740882429740953, 0.6694676526234568, 0.6554367403862607, 0.6871145937679641, 0.6973627667199596, 0.677839082032618, 0.6656099293584739, 0.6853610253860684, 0.7127045342173167, 0.6774929784721044, 0.6697791158632792, 0.665276789680729, 0.6609617898538276, 0.6573670880847166, 0.6812573532252995, 0.6669304645971238, 0.7456805630690589, 0.6722619172607951, 0.6674666794827154, 0.71101954162863, 0.6581471114518421, 0.681784654518447, 0.6708890082180987, 0.6891795059009299, 0.6846723076532707, 0.6778079742730775, 0.6716150419025536, 0.6624185510214552, 0.6736751066980704, 0.6894763100114708, 0.6659339720000773, 0.6774968737338238, 0.6814011092512952, 0.6647910226422914, 0.6882834695714566, 0.7808801986391988, 0.6703813570659151, 0.666212972377905, 0.6724199160716084, 0.6676592507580481, 0.6614165175994707, 0.3181115760489757 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "hoverinfo": "text", "marker": { "color": "rgba(128,177,211,1)" }, "mode": "markers", "name": "In-sample", "showlegend": true, "text": [ "Arm 0_0

metric_1: -17.159% [-18.070%, -16.249%]
metric_2: -5.747% [-6.411%, -5.083%]
Parameterization has too many items to render on hover (15).", "Arm 0_1

metric_1: -11.573% [-12.546%, -10.601%]
metric_2: -3.561% [-4.224%, -2.897%]
Parameterization has too many items to render on hover (15).", "Arm 0_10

metric_1: -11.361% [-12.331%, -10.392%]
metric_2: -3.61% [-4.282%, -2.938%]
Parameterization has too many items to render on hover (15).", "Arm 0_11

metric_1: 3.038% [1.885%, 4.190%]
metric_2: 2.514% [1.820%, 3.207%]
Parameterization has too many items to render on hover (15).", "Arm 0_12

metric_1: -3.07% [-4.126%, -2.013%]
metric_2: -0.597% [-1.279%, 0.086%]
Parameterization has too many items to render on hover (15).", "Arm 0_13

metric_1: -2.865% [-3.923%, -1.806%]
metric_2: -1.012% [-1.684%, -0.340%]
Parameterization has too many items to render on hover (15).", "Arm 0_14

metric_1: -0.817% [-1.903%, 0.268%]
metric_2: -0.818% [-1.486%, -0.151%]
Parameterization has too many items to render on hover (15).", "Arm 0_15

metric_1: -7.78% [-8.806%, -6.755%]
metric_2: -2.407% [-3.066%, -1.748%]
Parameterization has too many items to render on hover (15).", "Arm 0_16

metric_1: 2.475% [1.324%, 3.626%]
metric_2: 2.558% [1.857%, 3.258%]
Parameterization has too many items to render on hover (15).", "Arm 0_17

metric_1: -1.196% [-2.291%, -0.102%]
metric_2: 0.348% [-0.325%, 1.022%]
Parameterization has too many items to render on hover (15).", "Arm 0_18

metric_1: -1.36% [-2.441%, -0.279%]
metric_2: -1.311% [-1.979%, -0.644%]
Parameterization has too many items to render on hover (15).", "Arm 0_19

metric_1: -14.817% [-15.752%, -13.882%]
metric_2: -3.338% [-4.002%, -2.674%]
Parameterization has too many items to render on hover (15).", "Arm 0_2

metric_1: -3.272% [-4.339%, -2.205%]
metric_2: -1.007% [-1.680%, -0.334%]
Parameterization has too many items to render on hover (15).", "Arm 0_20

metric_1: 0.089% [-1.018%, 1.195%]
metric_2: -0.069% [-0.736%, 0.599%]
Parameterization has too many items to render on hover (15).", "Arm 0_21

metric_1: 0.536% [-0.574%, 1.646%]
metric_2: 0.104% [-0.577%, 0.785%]
Parameterization has too many items to render on hover (15).", "Arm 0_22

metric_1: 1.815% [0.683%, 2.946%]
metric_2: -0.072% [-0.748%, 0.604%]
Parameterization has too many items to render on hover (15).", "Arm 0_23

metric_1: -6.471% [-7.486%, -5.456%]
metric_2: -3.188% [-3.851%, -2.525%]
Parameterization has too many items to render on hover (15).", "Arm 0_24

metric_1: 0.888% [-0.226%, 2.001%]
metric_2: 1.96% [1.279%, 2.642%]
Parameterization has too many items to render on hover (15).", "Arm 0_25

metric_1: 0.729% [-0.378%, 1.836%]
metric_2: 0.013% [-0.661%, 0.686%]
Parameterization has too many items to render on hover (15).", "Arm 0_26

metric_1: 0.351% [-0.747%, 1.449%]
metric_2: 1.12% [0.441%, 1.799%]
Parameterization has too many items to render on hover (15).", "Arm 0_27

metric_1: -6.985% [-7.997%, -5.973%]
metric_2: 0.616% [-0.056%, 1.287%]
Parameterization has too many items to render on hover (15).", "Arm 0_28

metric_1: -7.506% [-8.510%, -6.502%]
metric_2: -2.55% [-3.217%, -1.884%]
Parameterization has too many items to render on hover (15).", "Arm 0_29

metric_1: 1.577% [0.452%, 2.702%]
metric_2: 5.965% [5.243%, 6.686%]
Parameterization has too many items to render on hover (15).", "Arm 0_3

metric_1: -2.214% [-3.295%, -1.134%]
metric_2: -0.421% [-1.095%, 0.254%]
Parameterization has too many items to render on hover (15).", "Arm 0_30

metric_1: -4.596% [-5.641%, -3.550%]
metric_2: -2.727% [-3.396%, -2.057%]
Parameterization has too many items to render on hover (15).", "Arm 0_31

metric_1: -19.189% [-20.075%, -18.304%]
metric_2: -5.859% [-6.515%, -5.204%]
Parameterization has too many items to render on hover (15).", "Arm 0_32

metric_1: 3.429% [2.274%, 4.584%]
metric_2: 3.197% [2.510%, 3.884%]
Parameterization has too many items to render on hover (15).", "Arm 0_33

metric_1: 3.371% [2.213%, 4.529%]
metric_2: 3.95% [3.253%, 4.648%]
Parameterization has too many items to render on hover (15).", "Arm 0_34

metric_1: -3.89% [-4.978%, -2.802%]
metric_2: 2.057% [1.379%, 2.735%]
Parameterization has too many items to render on hover (15).", "Arm 0_35

metric_1: 0.614% [-0.501%, 1.729%]
metric_2: 0.653% [-0.013%, 1.318%]
Parameterization has too many items to render on hover (15).", "Arm 0_36

metric_1: 0.59% [-0.523%, 1.703%]
metric_2: 1.287% [0.601%, 1.972%]
Parameterization has too many items to render on hover (15).", "Arm 0_37

metric_1: 0.865% [-0.248%, 1.979%]
metric_2: 4.621% [3.908%, 5.334%]
Parameterization has too many items to render on hover (15).", "Arm 0_38

metric_1: -1.268% [-2.361%, -0.175%]
metric_2: -0.762% [-1.440%, -0.085%]
Parameterization has too many items to render on hover (15).", "Arm 0_39

metric_1: -0.169% [-1.267%, 0.930%]
metric_2: 0.124% [-0.546%, 0.793%]
Parameterization has too many items to render on hover (15).", "Arm 0_4

metric_1: -2.893% [-3.949%, -1.837%]
metric_2: -1.543% [-2.208%, -0.878%]
Parameterization has too many items to render on hover (15).", "Arm 0_40

metric_1: -8.624% [-9.617%, -7.631%]
metric_2: -3.265% [-3.926%, -2.604%]
Parameterization has too many items to render on hover (15).", "Arm 0_41

metric_1: -5.157% [-6.184%, -4.129%]
metric_2: -1.741% [-2.399%, -1.084%]
Parameterization has too many items to render on hover (15).", "Arm 0_42

metric_1: -4.878% [-5.924%, -3.832%]
metric_2: -1.126% [-1.807%, -0.444%]
Parameterization has too many items to render on hover (15).", "Arm 0_43

metric_1: -3.655% [-4.706%, -2.605%]
metric_2: -1.381% [-2.048%, -0.714%]
Parameterization has too many items to render on hover (15).", "Arm 0_44

metric_1: 4.483% [3.314%, 5.652%]
metric_2: 7.652% [6.906%, 8.397%]
Parameterization has too many items to render on hover (15).", "Arm 0_45

metric_1: -3.06% [-4.117%, -2.004%]
metric_2: -2.202% [-2.874%, -1.529%]
Parameterization has too many items to render on hover (15).", "Arm 0_46

metric_1: -6.922% [-7.934%, -5.909%]
metric_2: -3.063% [-3.731%, -2.396%]
Parameterization has too many items to render on hover (15).", "Arm 0_47

metric_1: -3.478% [-4.539%, -2.418%]
metric_2: 4.013% [3.302%, 4.724%]
Parameterization has too many items to render on hover (15).", "Arm 0_48

metric_1: -4.546% [-5.591%, -3.501%]
metric_2: -1.169% [-1.827%, -0.511%]
Parameterization has too many items to render on hover (15).", "Arm 0_49

metric_1: -0.305% [-1.403%, 0.792%]
metric_2: 0.753% [0.071%, 1.434%]
Parameterization has too many items to render on hover (15).", "Arm 0_5

metric_1: -2.069% [-3.150%, -0.988%]
metric_2: -0.76% [-1.431%, -0.090%]
Parameterization has too many items to render on hover (15).", "Arm 0_50

metric_1: 0.367% [-0.736%, 1.470%]
metric_2: 2.231% [1.542%, 2.920%]
Parameterization has too many items to render on hover (15).", "Arm 0_51

metric_1: -4.597% [-5.631%, -3.562%]
metric_2: -1.371% [-2.056%, -0.687%]
Parameterization has too many items to render on hover (15).", "Arm 0_52

metric_1: -0.715% [-1.807%, 0.378%]
metric_2: 0.898% [0.220%, 1.576%]
Parameterization has too many items to render on hover (15).", "Arm 0_53

metric_1: -2.187% [-3.261%, -1.113%]
metric_2: -0.924% [-1.595%, -0.252%]
Parameterization has too many items to render on hover (15).", "Arm 0_54

metric_1: -3.62% [-4.663%, -2.577%]
metric_2: -1.266% [-1.928%, -0.603%]
Parameterization has too many items to render on hover (15).", "Arm 0_55

metric_1: -3.085% [-4.149%, -2.022%]
metric_2: -1.056% [-1.729%, -0.382%]
Parameterization has too many items to render on hover (15).", "Arm 0_56

metric_1: 0.065% [-1.043%, 1.174%]
metric_2: 1.104% [0.414%, 1.793%]
Parameterization has too many items to render on hover (15).", "Arm 0_57

metric_1: -8.045% [-9.053%, -7.036%]
metric_2: -3.231% [-3.897%, -2.565%]
Parameterization has too many items to render on hover (15).", "Arm 0_58

metric_1: -2.417% [-3.486%, -1.347%]
metric_2: -1.461% [-2.139%, -0.784%]
Parameterization has too many items to render on hover (15).", "Arm 0_59

metric_1: 2.155% [1.023%, 3.288%]
metric_2: 0.992% [0.310%, 1.673%]
Parameterization has too many items to render on hover (15).", "Arm 0_6

metric_1: -3.551% [-4.616%, -2.486%]
metric_2: -1.321% [-1.986%, -0.656%]
Parameterization has too many items to render on hover (15).", "Arm 0_60

metric_1: 3.0% [1.850%, 4.149%]
metric_2: 2.8% [2.112%, 3.488%]
Parameterization has too many items to render on hover (15).", "Arm 0_61

metric_1: 5.54% [4.346%, 6.735%]
metric_2: 10.809% [10.028%, 11.590%]
Parameterization has too many items to render on hover (15).", "Arm 0_62

metric_1: -13.137% [-14.095%, -12.178%]
metric_2: -3.839% [-4.509%, -3.169%]
Parameterization has too many items to render on hover (15).", "Arm 0_63

metric_1: -9.683% [-10.662%, -8.703%]
metric_2: -2.765% [-3.432%, -2.099%]
Parameterization has too many items to render on hover (15).", "Arm 0_7

metric_1: -4.089% [-5.138%, -3.041%]
metric_2: -1.576% [-2.248%, -0.903%]
Parameterization has too many items to render on hover (15).", "Arm 0_8

metric_1: -2.46% [-3.530%, -1.390%]
metric_2: -2.179% [-2.846%, -1.511%]
Parameterization has too many items to render on hover (15).", "Arm 0_9

metric_1: -7.98% [-8.985%, -6.976%]
metric_2: -2.79% [-3.451%, -2.128%]
Parameterization has too many items to render on hover (15).", "Arm status_quo

metric_1: -0.0% [-0.521%, 0.521%]
metric_2: -0.0% [-0.318%, 0.318%]
Parameterization has too many items to render on hover (15)." ], "type": "scatter", "visible": false, "x": [ -17.15914516790823, -11.573417823939199, -11.361067490605143, 3.037717284235046, -3.069542768219942, -2.864577054527247, -0.8171324453854826, -7.780377203320161, 2.475352747757105, -1.1963952539982303, -1.3598025263922875, -14.817429080033943, -3.2723491422755444, 0.08873184010870934, 0.5359430531455063, 1.8145980354381517, -6.471358153849177, 0.8877430970467538, 0.7286746192275023, 0.3510369131998831, -6.984895697391649, -7.505623786753492, 1.5767742612758522, -2.2143528351212733, -4.595608658658678, -19.18909654051259, 3.4292680540776312, 3.3710456099492796, -3.889937358175768, 0.6139708178717305, 0.5900679364090252, 0.8653940325415319, -1.2680473963845287, -0.1686822762476349, -2.893033151686222, -8.623721510464856, -5.156592129047242, -4.878241533976907, -3.655342272989576, 4.483189117418239, -3.0603072946347587, -6.921569399104486, -3.4781006474949763, -4.545929170059967, -0.305361571312272, -2.068706090724009, 0.36738131081906034, -4.596829363692114, -0.7145714514809897, -2.1870440120521244, -3.619802243175367, -3.0854247199357574, 0.06528116501171306, -8.04466137206524, -2.416515121000999, 2.1550821626518215, -3.550888972902983, 2.999501801038642, 5.540352816509654, -13.136826425368891, -9.682667832954975, -4.089456620979975, -2.4600469634137005, -7.980395623816304, -0.0 ], "y": [ -5.746731056831749, -3.560535696444575, -3.610268946914751, 2.5135502228088797, -0.5965513852397577, -1.0116808329390385, -0.8182560448021005, -2.4071834175723335, 2.5577489299929255, 0.348192434742279, -1.3114357729283859, -3.3382598457223502, -1.0068955594731426, -0.06867630083734555, 0.10424166292801242, -0.07210829452142886, -3.188401274811581, 1.960086317210278, 0.012674904654773453, 1.1203410785772772, 0.6155055435112974, -2.5504391581007666, 5.964668435375144, -0.4205314884998786, -2.726644960728147, -5.859418405279977, 3.1966830537300392, 3.950198735676712, 2.0568967995693455, 0.652872009905956, 1.286780707668062, 4.62085929335188, -0.7622630447726769, 0.123503526062618, -1.5431517157186252, -3.2650236570623457, -1.7414179188197445, -1.1257173899572386, -1.3809544654506094, 7.651708533863019, -2.2015816842718356, -3.063188080332386, 4.012625447421733, -1.168849751600452, 0.7526902410881314, -0.7604599707221315, 2.2308412374353317, -1.371441760784444, 0.8978911718791985, -0.9236769844543923, -1.2656161965636414, -1.0555488550285586, 1.1039153988973123, -3.2310973947137103, -1.4614566797717248, 0.9918050829129264, -1.3210922238627325, 2.8001328033355772, 10.809115959705165, -3.839017981482167, -2.765325873487098, -1.5757183832623392, -2.1787372563938, -2.789593215553577, -0.0 ] }, { "error_x": { "array": [ 0.9032151733520527, 0.9624260346524522, 0.9593909284698995, 1.1347609507759298, 1.0408257278018285, 1.0435860695228432, 1.0711864083714089, 1.0156140087849477, 1.128257833835402, 1.0788991832923005, 1.0706037001289912, 0.9284671425890606, 1.0523829338844282, 1.0903373892267718, 1.0861328996665252, 1.1145865581026473, 1.0063514862723812, 1.0994776629118859, 1.091675914931962, 1.0844593315915725, 1.0017853270032084, 0.9935701990690048, 1.1127508961856405, 1.0668110378139575, 1.0328601627012277, 0.8797268687204242, 1.140518932271723, 1.1417536403033617, 1.0764096893237975, 1.0965566556283748, 1.0984998396729218, 1.102540333829891, 1.0794770222090497, 1.085962611129493, 1.0433251902342626, 0.9820278721895025, 1.0166480934308502, 1.0343546221526094, 1.0382944853121021, 1.1551456071793769, 1.0464931893207308, 1.0023861997986674, 1.0516802404518037, 1.0346896327290815, 1.08327143184596, 1.0698309404523143, 1.0904297395971863, 1.0222860350150513, 1.0810704079796398, 1.0613893596292459, 1.0320103439592427, 1.0528002249874768, 1.0912064123686547, 0.9988930110467326, 1.0571937806819418, 1.1193739547640122, 1.0530800272030827, 1.1303999126786766, 1.180499014903316, 0.9513079288502773, 0.9719033134076178, 1.03791308728753, 1.0588852902469934, 0.9945314913677392, 0.5199803498226312 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "error_y": { "array": [ 0.6547387343317975, 0.6532033115837151, 0.6609466315731528, 0.6813554440847576, 0.6694705491295897, 0.6591303137747925, 0.6563086390178189, 0.6508039407989954, 0.6839216136881153, 0.6604406759568969, 0.6601787231575951, 0.6565366929811817, 0.6613093240694986, 0.6561140874958142, 0.6633706122047732, 0.6637799634962176, 0.6555405967295402, 0.6704566164608853, 0.66273573692496, 0.6685871803108022, 0.6628627361853555, 0.6572002148831203, 0.712446900341419, 0.6636937933944925, 0.6586242603445244, 0.6475500126287197, 0.6773760670047011, 0.6859237814996093, 0.6688072097037889, 0.6516465341587597, 0.6738024004862555, 0.7036162417194723, 0.6668939573219893, 0.6610547090506331, 0.6553518681013905, 0.6508051033775936, 0.64797757405811, 0.6713714106339492, 0.656723211631772, 0.7346090086077302, 0.6642524778308558, 0.6574536033119194, 0.7031723757695748, 0.6494774398002832, 0.6704885519899918, 0.6622056608350378, 0.6797639514603396, 0.6735895146091605, 0.6692892385552365, 0.6620882696925839, 0.6531942830594355, 0.6650013545172943, 0.6769982685390781, 0.6574731612170758, 0.6674033123702455, 0.6722373506713744, 0.6559893752039715, 0.6748097809173098, 0.7697444470145935, 0.6626605822603224, 0.6581382444319371, 0.663286850538724, 0.658955602798785, 0.6526018616070007, 0.3176861573292605 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "hoverinfo": "text", "marker": { "color": "rgba(128,177,211,1)" }, "mode": "markers", "name": "In-sample", "showlegend": true, "text": [ "Arm 0_0

metric_1: -17.022% [-17.925%, -16.119%]
metric_2: -5.683% [-6.338%, -5.028%]
Parameterization has too many items to render on hover (15).", "Arm 0_1

metric_1: -11.473% [-12.436%, -10.511%]
metric_2: -3.526% [-4.180%, -2.873%]
Parameterization has too many items to render on hover (15).", "Arm 0_10

metric_1: -11.26% [-12.220%, -10.301%]
metric_2: -3.535% [-4.196%, -2.874%]
Parameterization has too many items to render on hover (15).", "Arm 0_11

metric_1: 2.99% [1.855%, 4.125%]
metric_2: 2.542% [1.861%, 3.224%]
Parameterization has too many items to render on hover (15).", "Arm 0_12

metric_1: -3.205% [-4.246%, -2.164%]
metric_2: -0.629% [-1.298%, 0.041%]
Parameterization has too many items to render on hover (15).", "Arm 0_13

metric_1: -2.828% [-3.871%, -1.784%]
metric_2: -1.055% [-1.714%, -0.396%]
Parameterization has too many items to render on hover (15).", "Arm 0_14

metric_1: -0.807% [-1.878%, 0.264%]
metric_2: -0.832% [-1.488%, -0.176%]
Parameterization has too many items to render on hover (15).", "Arm 0_15

metric_1: -7.632% [-8.648%, -6.617%]
metric_2: -2.319% [-2.970%, -1.669%]
Parameterization has too many items to render on hover (15).", "Arm 0_16

metric_1: 2.493% [1.365%, 3.621%]
metric_2: 2.582% [1.898%, 3.266%]
Parameterization has too many items to render on hover (15).", "Arm 0_17

metric_1: -1.148% [-2.227%, -0.069%]
metric_2: 0.326% [-0.334%, 0.987%]
Parameterization has too many items to render on hover (15).", "Arm 0_18

metric_1: -1.417% [-2.488%, -0.346%]
metric_2: -1.321% [-1.981%, -0.661%]
Parameterization has too many items to render on hover (15).", "Arm 0_19

metric_1: -14.735% [-15.664%, -13.807%]
metric_2: -3.315% [-3.971%, -2.658%]
Parameterization has too many items to render on hover (15).", "Arm 0_2

metric_1: -3.211% [-4.263%, -2.158%]
metric_2: -0.992% [-1.653%, -0.331%]
Parameterization has too many items to render on hover (15).", "Arm 0_20

metric_1: 0.073% [-1.017%, 1.164%]
metric_2: -0.063% [-0.719%, 0.594%]
Parameterization has too many items to render on hover (15).", "Arm 0_21

metric_1: 0.528% [-0.558%, 1.614%]
metric_2: 0.066% [-0.597%, 0.730%]
Parameterization has too many items to render on hover (15).", "Arm 0_22

metric_1: 1.746% [0.632%, 2.861%]
metric_2: -0.055% [-0.719%, 0.608%]
Parameterization has too many items to render on hover (15).", "Arm 0_23

metric_1: -6.404% [-7.411%, -5.398%]
metric_2: -3.15% [-3.805%, -2.494%]
Parameterization has too many items to render on hover (15).", "Arm 0_24

metric_1: 0.836% [-0.264%, 1.935%]
metric_2: 1.893% [1.223%, 2.564%]
Parameterization has too many items to render on hover (15).", "Arm 0_25

metric_1: 0.699% [-0.393%, 1.790%]
metric_2: 0.024% [-0.639%, 0.686%]
Parameterization has too many items to render on hover (15).", "Arm 0_26

metric_1: 0.319% [-0.766%, 1.403%]
metric_2: 1.077% [0.409%, 1.746%]
Parameterization has too many items to render on hover (15).", "Arm 0_27

metric_1: -6.918% [-7.920%, -5.916%]
metric_2: 0.651% [-0.012%, 1.314%]
Parameterization has too many items to render on hover (15).", "Arm 0_28

metric_1: -7.463% [-8.457%, -6.469%]
metric_2: -2.49% [-3.147%, -1.832%]
Parameterization has too many items to render on hover (15).", "Arm 0_29

metric_1: 1.48% [0.367%, 2.592%]
metric_2: 5.858% [5.145%, 6.570%]
Parameterization has too many items to render on hover (15).", "Arm 0_3

metric_1: -2.25% [-3.317%, -1.183%]
metric_2: -0.448% [-1.112%, 0.215%]
Parameterization has too many items to render on hover (15).", "Arm 0_30

metric_1: -4.65% [-5.683%, -3.617%]
metric_2: -2.712% [-3.371%, -2.054%]
Parameterization has too many items to render on hover (15).", "Arm 0_31

metric_1: -19.06% [-19.939%, -18.180%]
metric_2: -5.792% [-6.440%, -5.145%]
Parameterization has too many items to render on hover (15).", "Arm 0_32

metric_1: 3.344% [2.204%, 4.485%]
metric_2: 3.158% [2.481%, 3.835%]
Parameterization has too many items to render on hover (15).", "Arm 0_33

metric_1: 3.298% [2.156%, 4.440%]
metric_2: 3.853% [3.167%, 4.539%]
Parameterization has too many items to render on hover (15).", "Arm 0_34

metric_1: -3.845% [-4.922%, -2.769%]
metric_2: 2.026% [1.357%, 2.695%]
Parameterization has too many items to render on hover (15).", "Arm 0_35

metric_1: 0.616% [-0.480%, 1.713%]
metric_2: 0.619% [-0.033%, 1.270%]
Parameterization has too many items to render on hover (15).", "Arm 0_36

metric_1: 0.606% [-0.493%, 1.704%]
metric_2: 1.328% [0.654%, 2.002%]
Parameterization has too many items to render on hover (15).", "Arm 0_37

metric_1: 0.801% [-0.301%, 1.904%]
metric_2: 4.539% [3.836%, 5.243%]
Parameterization has too many items to render on hover (15).", "Arm 0_38

metric_1: -1.23% [-2.310%, -0.151%]
metric_2: -0.74% [-1.407%, -0.074%]
Parameterization has too many items to render on hover (15).", "Arm 0_39

metric_1: -0.189% [-1.275%, 0.897%]
metric_2: 0.11% [-0.551%, 0.771%]
Parameterization has too many items to render on hover (15).", "Arm 0_4

metric_1: -2.877% [-3.920%, -1.834%]
metric_2: -1.538% [-2.193%, -0.883%]
Parameterization has too many items to render on hover (15).", "Arm 0_40

metric_1: -8.667% [-9.649%, -7.685%]
metric_2: -3.287% [-3.937%, -2.636%]
Parameterization has too many items to render on hover (15).", "Arm 0_41

metric_1: -5.112% [-6.129%, -4.096%]
metric_2: -1.776% [-2.424%, -1.128%]
Parameterization has too many items to render on hover (15).", "Arm 0_42

metric_1: -4.866% [-5.900%, -3.831%]
metric_2: -1.107% [-1.779%, -0.436%]
Parameterization has too many items to render on hover (15).", "Arm 0_43

metric_1: -3.621% [-4.659%, -2.582%]
metric_2: -1.374% [-2.031%, -0.717%]
Parameterization has too many items to render on hover (15).", "Arm 0_44

metric_1: 4.416% [3.261%, 5.571%]
metric_2: 7.531% [6.797%, 8.266%]
Parameterization has too many items to render on hover (15).", "Arm 0_45

metric_1: -3.094% [-4.140%, -2.047%]
metric_2: -2.19% [-2.854%, -1.526%]
Parameterization has too many items to render on hover (15).", "Arm 0_46

metric_1: -6.987% [-7.990%, -5.985%]
metric_2: -3.082% [-3.740%, -2.425%]
Parameterization has too many items to render on hover (15).", "Arm 0_47

metric_1: -3.456% [-4.507%, -2.404%]
metric_2: 3.964% [3.261%, 4.667%]
Parameterization has too many items to render on hover (15).", "Arm 0_48

metric_1: -4.487% [-5.521%, -3.452%]
metric_2: -1.178% [-1.827%, -0.528%]
Parameterization has too many items to render on hover (15).", "Arm 0_49

metric_1: -0.276% [-1.360%, 0.807%]
metric_2: 0.752% [0.081%, 1.422%]
Parameterization has too many items to render on hover (15).", "Arm 0_5

metric_1: -2.093% [-3.163%, -1.023%]
metric_2: -0.772% [-1.435%, -0.110%]
Parameterization has too many items to render on hover (15).", "Arm 0_50

metric_1: 0.34% [-0.750%, 1.431%]
metric_2: 2.197% [1.517%, 2.877%]
Parameterization has too many items to render on hover (15).", "Arm 0_51

metric_1: -4.611% [-5.633%, -3.589%]
metric_2: -1.337% [-2.011%, -0.664%]
Parameterization has too many items to render on hover (15).", "Arm 0_52

metric_1: -0.757% [-1.838%, 0.324%]
metric_2: 0.878% [0.208%, 1.547%]
Parameterization has too many items to render on hover (15).", "Arm 0_53

metric_1: -2.166% [-3.228%, -1.105%]
metric_2: -0.911% [-1.573%, -0.249%]
Parameterization has too many items to render on hover (15).", "Arm 0_54

metric_1: -3.611% [-4.643%, -2.579%]
metric_2: -1.267% [-1.920%, -0.613%]
Parameterization has too many items to render on hover (15).", "Arm 0_55

metric_1: -3.053% [-4.106%, -2.000%]
metric_2: -1.055% [-1.720%, -0.390%]
Parameterization has too many items to render on hover (15).", "Arm 0_56

metric_1: 0.115% [-0.976%, 1.206%]
metric_2: 1.138% [0.461%, 1.815%]
Parameterization has too many items to render on hover (15).", "Arm 0_57

metric_1: -8.012% [-9.011%, -7.013%]
metric_2: -3.221% [-3.878%, -2.563%]
Parameterization has too many items to render on hover (15).", "Arm 0_58

metric_1: -2.444% [-3.501%, -1.387%]
metric_2: -1.454% [-2.122%, -0.787%]
Parameterization has too many items to render on hover (15).", "Arm 0_59

metric_1: 2.105% [0.986%, 3.225%]
metric_2: 0.993% [0.320%, 1.665%]
Parameterization has too many items to render on hover (15).", "Arm 0_6

metric_1: -3.549% [-4.602%, -2.496%]
metric_2: -1.318% [-1.974%, -0.662%]
Parameterization has too many items to render on hover (15).", "Arm 0_60

metric_1: 2.966% [1.836%, 4.097%]
metric_2: 2.781% [2.106%, 3.455%]
Parameterization has too many items to render on hover (15).", "Arm 0_61

metric_1: 5.393% [4.212%, 6.573%]
metric_2: 10.553% [9.783%, 11.322%]
Parameterization has too many items to render on hover (15).", "Arm 0_62

metric_1: -13.026% [-13.978%, -12.075%]
metric_2: -3.787% [-4.450%, -3.125%]
Parameterization has too many items to render on hover (15).", "Arm 0_63

metric_1: -9.619% [-10.591%, -8.647%]
metric_2: -2.738% [-3.397%, -2.080%]
Parameterization has too many items to render on hover (15).", "Arm 0_7

metric_1: -4.118% [-5.156%, -3.080%]
metric_2: -1.61% [-2.273%, -0.947%]
Parameterization has too many items to render on hover (15).", "Arm 0_8

metric_1: -2.464% [-3.523%, -1.405%]
metric_2: -2.148% [-2.807%, -1.489%]
Parameterization has too many items to render on hover (15).", "Arm 0_9

metric_1: -7.957% [-8.951%, -6.962%]
metric_2: -2.794% [-3.447%, -2.142%]
Parameterization has too many items to render on hover (15).", "Arm status_quo

metric_1: -0.0% [-0.520%, 0.520%]
metric_2: -0.0% [-0.318%, 0.318%]
Parameterization has too many items to render on hover (15)." ], "type": "scatter", "visible": true, "x": [ -17.022045315104542, -11.473276363575263, -11.260309485449103, 2.9899364287480865, -3.2049806778237757, -2.827647808086468, -0.8072357684589017, -7.632201738791567, 2.4931910046363135, -1.1483177933819395, -1.4169105153422092, -14.735353419604191, -3.2108750242520334, 0.07332105300986574, 0.52794229128138, 1.7461298706241524, -6.404301715961119, 0.8357526716703064, 0.6988214644260046, 0.3185581570295636, -6.918118515914437, -7.462960459390675, 1.4797223030792654, -2.250057847895754, -4.649788400829341, -19.059723742205165, 3.3442377220374424, 3.2977920613814318, -3.845344844296581, 0.6160918095707203, 0.605873558955114, 0.8012482568804604, -1.2301194692314896, -0.18910424701708023, -2.8771572632426596, -8.666686434035148, -5.1123185935816275, -4.86569050603379, -3.620616821502621, 4.416040423353074, -3.093977759193243, -6.987479150965008, -3.45561072825807, -4.486522144334903, -0.27632651192689084, -2.092939160334957, 0.34040916740729205, -4.6111158309365, -0.7572230824447213, -2.1663524739549853, -3.611088388568022, -3.053027739941141, 0.11513485293184113, -8.011705772759294, -2.4442287384702572, 2.1052356784416553, -3.5492459592908614, 2.966172129108271, 5.392515165139946, -13.026404058841306, -9.619176461732291, -4.118044655666446, -2.4636655602657878, -7.956569241984238, -0.0 ], "y": [ -5.682890950120342, -3.5263840662392654, -3.5354295227440935, 2.542378866552539, -0.6285418092402999, -1.0553405972573464, -0.8320014955517723, -2.3193462375707914, 2.58163977526919, 0.32621268542555526, -1.3207797795363851, -3.314816695063182, -0.9921710458941605, -0.06250694414329654, 0.06640049636248754, -0.05540222390315327, -3.1495291448321576, 1.8934882315080965, 0.023529017577203423, 1.0773772731169438, 0.6512642256730126, -2.489527283561809, 5.857858819488534, -0.4484318638610448, -2.712182537468451, -5.792314875460171, 3.1580957792386144, 3.8533176800717563, 2.026262991103158, 0.6187928699607773, 1.3280865831055046, 4.5394243735749775, -0.7404858515521753, 0.10968492064137063, -1.5381048845570704, -3.2865434005114005, -1.7761680789956928, -1.1071508425287317, -1.3739381694523336, 7.5312245476719, -2.1901762525545276, -3.082214357601481, 3.963995297169013, -1.1779086966890597, 0.7516740725998268, -0.772463613395467, 2.196834434560228, -1.3372360602549658, 0.8775653888918572, -0.9111761852726025, -1.2666733248842346, -1.055168296030082, 1.1378418002527275, -3.2209479427630807, -1.4541461401402338, 0.9926717601692114, -1.3177840022000746, 2.7806338967179443, 10.552746740900881, -3.7873810524577847, -2.7384540934203034, -1.6100825481553491, -2.147955641504198, -2.7941662700346765, -0.0 ] }, { "error_x": { "array": [ 4.83875056447029, 5.102041394990379, 4.662392753925473 ], "color": "rgba(251,128,114,0.4)", "type": "data" }, "error_y": { "array": [ 2.808469142463111, 2.8179402082703007, 2.752087033496351 ], "color": "rgba(251,128,114,0.4)", "type": "data" }, "hoverinfo": "text", "marker": { "color": "rgba(251,128,114,1)" }, "mode": "markers", "name": "unconstrained", "showlegend": true, "text": [ "Arm b849

metric_1: 6.105% [1.266%, 10.944%]
metric_2: 8.281% [5.473%, 11.090%]
Parameterization has too many items to render on hover (15).", "Arm 3ad4

metric_1: 5.457% [0.355%, 10.559%]
metric_2: 6.539% [3.721%, 9.357%]
Parameterization has too many items to render on hover (15).", "Arm 998f

metric_1: 5.544% [0.881%, 10.206%]
metric_2: 8.915% [6.163%, 11.667%]
Parameterization has too many items to render on hover (15)." ], "type": "scatter", "visible": true, "x": [ 6.104854772592679, 5.4570750726576485, 5.543724025368393 ], "y": [ 8.281103305736018, 6.539035341740647, 8.915317158366165 ] } ], "layout": { "annotations": [ { "showarrow": false, "text": "Show CI", "x": 1.18, "xref": "paper", "y": 0.7, "yanchor": "middle", "yref": "paper" }, { "showarrow": false, "text": "Type", "x": 1.18, "xref": "paper", "y": 0.6, "yanchor": "middle", "yref": "paper" } ], "font": { "size": 10 }, "height": 600, "hovermode": "closest", "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "sequentialminus": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "title": { "text": "Objective Tradeoffs" }, "updatemenus": [ { "buttons": [ { "args": [ { "error_x.thickness": 2, "error_x.width": 4, "error_y.thickness": 2, "error_y.width": 4 } ], "label": "Yes", "method": "restyle" }, { "args": [ { "error_x.thickness": 0, "error_x.width": 0, "error_y.thickness": 0, "error_y.width": 0 } ], "label": "No", "method": "restyle" } ], "x": 1.25, "xanchor": "left", "y": 0.67, "yanchor": "middle" }, { "buttons": [ { "args": [ { "visible": [ false, true, true ] } ], "label": "Modeled", "method": "restyle" }, { "args": [ { "visible": [ true, false, false ] } ], "label": "Observed", "method": "restyle" } ], "x": 1.25, "xanchor": "left", "y": 0.57, "yanchor": "middle" } ], "width": 800, "xaxis": { "title": { "text": "metric_1 (%)" }, "zeroline": true, "zerolinecolor": "red" }, "yaxis": { "title": { "text": "metric_2 (%)" }, "zeroline": true, "zerolinecolor": "red" } } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "render(plot_multiple_metrics(\n", " gp,\n", " metric_x=METRIC_X_AXIS,\n", " metric_y=METRIC_Y_AXIS,\n", " generator_runs_dict={\n", " 'unconstrained': unconstrained,\n", " }\n", "))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Change Objectives" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With our unconstrained optimization, we generate some candidates which are pretty promising with respect to our objective! However, there is a clear regression in our constraint metric, above our initial 5% desired constraint. Let's add that constraint back in. " ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "constraint_5 = OutcomeConstraint(metric=constraint_metric, op=ComparisonOp.LEQ, bound=5)\n", "constraint_5_results = gp.gen(\n", " n=3, \n", " optimization_config=OptimizationConfig(\n", " objective=Objective(objective_metric),\n", " outcome_constraints=[constraint_5]\n", " )\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This yields a *GeneratorRun*, which contains points according to our specified optimization config, along with metadata about how the points were generated. Let's plot the tradeoffs in these new points. " ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "linkText": "Export to plot.ly", "plotlyServerURL": "https://plot.ly", "showLink": false }, "data": [ { "error_x": { "array": [ 0.9105274690527578, 0.9725977092158347, 0.9694448615326413, 1.1526497268980418, 1.056129089525654, 1.0581150790455403, 1.0853729636494531, 1.025297982327309, 1.1509354346821015, 1.0946224534113276, 1.080994735603789, 0.9349800939958878, 1.0669782977880082, 1.1063382644746038, 1.109747945764228, 1.1312428838918263, 1.0151345627364643, 1.113745664637434, 1.1070645021428274, 1.098298041303658, 1.011825279703854, 1.004076551317031, 1.1247628081965457, 1.0803499998271806, 1.0451840940805415, 0.8854990204077908, 1.1551313262975833, 1.1582540625351196, 1.088250532740983, 1.1152606223383992, 1.1132227750465398, 1.1136443399827738, 1.0928257948335294, 1.098342295748459, 1.0562137210185467, 0.9929499124720941, 1.0275183054522712, 1.0461406507525532, 1.0507233488011212, 1.1693056777789836, 1.0564300559878494, 1.012746475490335, 1.060503126319525, 1.0453978622119475, 1.0972242260284542, 1.0810868080421863, 1.103084437746675, 1.0346109289273437, 1.0925727929561801, 1.0737167310801996, 1.042945064542918, 1.0636626539145406, 1.108765950358755, 1.0083792696638647, 1.069964804645196, 1.1324679783490406, 1.0646727177942992, 1.149093505549881, 1.19450723179394, 0.9584867213686696, 0.9797729070203455, 1.048287897726107, 1.0699714924908204, 1.0044799847095478, 0.5205646439608032 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "error_y": { "array": [ 0.6639175585103954, 0.6630929411298598, 0.6719874423233675, 0.6935680367786019, 0.6821107678411614, 0.6721157287254815, 0.6675598394585217, 0.6588395471546759, 0.7005079335820925, 0.6734693768172479, 0.6678033173208016, 0.6639242874696609, 0.6729502923953926, 0.6673739566339196, 0.6808476824923786, 0.675618453499239, 0.6630338514495195, 0.6814341339540937, 0.6732139305974362, 0.678993016475866, 0.6717812932415321, 0.6668482051137559, 0.7215454438634975, 0.6740882429740953, 0.6694676526234568, 0.6554367403862607, 0.6871145937679641, 0.6973627667199596, 0.677839082032618, 0.6656099293584739, 0.6853610253860684, 0.7127045342173167, 0.6774929784721044, 0.6697791158632792, 0.665276789680729, 0.6609617898538276, 0.6573670880847166, 0.6812573532252995, 0.6669304645971238, 0.7456805630690589, 0.6722619172607951, 0.6674666794827154, 0.71101954162863, 0.6581471114518421, 0.681784654518447, 0.6708890082180987, 0.6891795059009299, 0.6846723076532707, 0.6778079742730775, 0.6716150419025536, 0.6624185510214552, 0.6736751066980704, 0.6894763100114708, 0.6659339720000773, 0.6774968737338238, 0.6814011092512952, 0.6647910226422914, 0.6882834695714566, 0.7808801986391988, 0.6703813570659151, 0.666212972377905, 0.6724199160716084, 0.6676592507580481, 0.6614165175994707, 0.3181115760489757 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "hoverinfo": "text", "marker": { "color": "rgba(128,177,211,1)" }, "mode": "markers", "name": "In-sample", "showlegend": true, "text": [ "Arm 0_0

metric_1: -17.159% [-18.070%, -16.249%]
metric_2: -5.747% [-6.411%, -5.083%]
Parameterization has too many items to render on hover (15).", "Arm 0_1

metric_1: -11.573% [-12.546%, -10.601%]
metric_2: -3.561% [-4.224%, -2.897%]
Parameterization has too many items to render on hover (15).", "Arm 0_10

metric_1: -11.361% [-12.331%, -10.392%]
metric_2: -3.61% [-4.282%, -2.938%]
Parameterization has too many items to render on hover (15).", "Arm 0_11

metric_1: 3.038% [1.885%, 4.190%]
metric_2: 2.514% [1.820%, 3.207%]
Parameterization has too many items to render on hover (15).", "Arm 0_12

metric_1: -3.07% [-4.126%, -2.013%]
metric_2: -0.597% [-1.279%, 0.086%]
Parameterization has too many items to render on hover (15).", "Arm 0_13

metric_1: -2.865% [-3.923%, -1.806%]
metric_2: -1.012% [-1.684%, -0.340%]
Parameterization has too many items to render on hover (15).", "Arm 0_14

metric_1: -0.817% [-1.903%, 0.268%]
metric_2: -0.818% [-1.486%, -0.151%]
Parameterization has too many items to render on hover (15).", "Arm 0_15

metric_1: -7.78% [-8.806%, -6.755%]
metric_2: -2.407% [-3.066%, -1.748%]
Parameterization has too many items to render on hover (15).", "Arm 0_16

metric_1: 2.475% [1.324%, 3.626%]
metric_2: 2.558% [1.857%, 3.258%]
Parameterization has too many items to render on hover (15).", "Arm 0_17

metric_1: -1.196% [-2.291%, -0.102%]
metric_2: 0.348% [-0.325%, 1.022%]
Parameterization has too many items to render on hover (15).", "Arm 0_18

metric_1: -1.36% [-2.441%, -0.279%]
metric_2: -1.311% [-1.979%, -0.644%]
Parameterization has too many items to render on hover (15).", "Arm 0_19

metric_1: -14.817% [-15.752%, -13.882%]
metric_2: -3.338% [-4.002%, -2.674%]
Parameterization has too many items to render on hover (15).", "Arm 0_2

metric_1: -3.272% [-4.339%, -2.205%]
metric_2: -1.007% [-1.680%, -0.334%]
Parameterization has too many items to render on hover (15).", "Arm 0_20

metric_1: 0.089% [-1.018%, 1.195%]
metric_2: -0.069% [-0.736%, 0.599%]
Parameterization has too many items to render on hover (15).", "Arm 0_21

metric_1: 0.536% [-0.574%, 1.646%]
metric_2: 0.104% [-0.577%, 0.785%]
Parameterization has too many items to render on hover (15).", "Arm 0_22

metric_1: 1.815% [0.683%, 2.946%]
metric_2: -0.072% [-0.748%, 0.604%]
Parameterization has too many items to render on hover (15).", "Arm 0_23

metric_1: -6.471% [-7.486%, -5.456%]
metric_2: -3.188% [-3.851%, -2.525%]
Parameterization has too many items to render on hover (15).", "Arm 0_24

metric_1: 0.888% [-0.226%, 2.001%]
metric_2: 1.96% [1.279%, 2.642%]
Parameterization has too many items to render on hover (15).", "Arm 0_25

metric_1: 0.729% [-0.378%, 1.836%]
metric_2: 0.013% [-0.661%, 0.686%]
Parameterization has too many items to render on hover (15).", "Arm 0_26

metric_1: 0.351% [-0.747%, 1.449%]
metric_2: 1.12% [0.441%, 1.799%]
Parameterization has too many items to render on hover (15).", "Arm 0_27

metric_1: -6.985% [-7.997%, -5.973%]
metric_2: 0.616% [-0.056%, 1.287%]
Parameterization has too many items to render on hover (15).", "Arm 0_28

metric_1: -7.506% [-8.510%, -6.502%]
metric_2: -2.55% [-3.217%, -1.884%]
Parameterization has too many items to render on hover (15).", "Arm 0_29

metric_1: 1.577% [0.452%, 2.702%]
metric_2: 5.965% [5.243%, 6.686%]
Parameterization has too many items to render on hover (15).", "Arm 0_3

metric_1: -2.214% [-3.295%, -1.134%]
metric_2: -0.421% [-1.095%, 0.254%]
Parameterization has too many items to render on hover (15).", "Arm 0_30

metric_1: -4.596% [-5.641%, -3.550%]
metric_2: -2.727% [-3.396%, -2.057%]
Parameterization has too many items to render on hover (15).", "Arm 0_31

metric_1: -19.189% [-20.075%, -18.304%]
metric_2: -5.859% [-6.515%, -5.204%]
Parameterization has too many items to render on hover (15).", "Arm 0_32

metric_1: 3.429% [2.274%, 4.584%]
metric_2: 3.197% [2.510%, 3.884%]
Parameterization has too many items to render on hover (15).", "Arm 0_33

metric_1: 3.371% [2.213%, 4.529%]
metric_2: 3.95% [3.253%, 4.648%]
Parameterization has too many items to render on hover (15).", "Arm 0_34

metric_1: -3.89% [-4.978%, -2.802%]
metric_2: 2.057% [1.379%, 2.735%]
Parameterization has too many items to render on hover (15).", "Arm 0_35

metric_1: 0.614% [-0.501%, 1.729%]
metric_2: 0.653% [-0.013%, 1.318%]
Parameterization has too many items to render on hover (15).", "Arm 0_36

metric_1: 0.59% [-0.523%, 1.703%]
metric_2: 1.287% [0.601%, 1.972%]
Parameterization has too many items to render on hover (15).", "Arm 0_37

metric_1: 0.865% [-0.248%, 1.979%]
metric_2: 4.621% [3.908%, 5.334%]
Parameterization has too many items to render on hover (15).", "Arm 0_38

metric_1: -1.268% [-2.361%, -0.175%]
metric_2: -0.762% [-1.440%, -0.085%]
Parameterization has too many items to render on hover (15).", "Arm 0_39

metric_1: -0.169% [-1.267%, 0.930%]
metric_2: 0.124% [-0.546%, 0.793%]
Parameterization has too many items to render on hover (15).", "Arm 0_4

metric_1: -2.893% [-3.949%, -1.837%]
metric_2: -1.543% [-2.208%, -0.878%]
Parameterization has too many items to render on hover (15).", "Arm 0_40

metric_1: -8.624% [-9.617%, -7.631%]
metric_2: -3.265% [-3.926%, -2.604%]
Parameterization has too many items to render on hover (15).", "Arm 0_41

metric_1: -5.157% [-6.184%, -4.129%]
metric_2: -1.741% [-2.399%, -1.084%]
Parameterization has too many items to render on hover (15).", "Arm 0_42

metric_1: -4.878% [-5.924%, -3.832%]
metric_2: -1.126% [-1.807%, -0.444%]
Parameterization has too many items to render on hover (15).", "Arm 0_43

metric_1: -3.655% [-4.706%, -2.605%]
metric_2: -1.381% [-2.048%, -0.714%]
Parameterization has too many items to render on hover (15).", "Arm 0_44

metric_1: 4.483% [3.314%, 5.652%]
metric_2: 7.652% [6.906%, 8.397%]
Parameterization has too many items to render on hover (15).", "Arm 0_45

metric_1: -3.06% [-4.117%, -2.004%]
metric_2: -2.202% [-2.874%, -1.529%]
Parameterization has too many items to render on hover (15).", "Arm 0_46

metric_1: -6.922% [-7.934%, -5.909%]
metric_2: -3.063% [-3.731%, -2.396%]
Parameterization has too many items to render on hover (15).", "Arm 0_47

metric_1: -3.478% [-4.539%, -2.418%]
metric_2: 4.013% [3.302%, 4.724%]
Parameterization has too many items to render on hover (15).", "Arm 0_48

metric_1: -4.546% [-5.591%, -3.501%]
metric_2: -1.169% [-1.827%, -0.511%]
Parameterization has too many items to render on hover (15).", "Arm 0_49

metric_1: -0.305% [-1.403%, 0.792%]
metric_2: 0.753% [0.071%, 1.434%]
Parameterization has too many items to render on hover (15).", "Arm 0_5

metric_1: -2.069% [-3.150%, -0.988%]
metric_2: -0.76% [-1.431%, -0.090%]
Parameterization has too many items to render on hover (15).", "Arm 0_50

metric_1: 0.367% [-0.736%, 1.470%]
metric_2: 2.231% [1.542%, 2.920%]
Parameterization has too many items to render on hover (15).", "Arm 0_51

metric_1: -4.597% [-5.631%, -3.562%]
metric_2: -1.371% [-2.056%, -0.687%]
Parameterization has too many items to render on hover (15).", "Arm 0_52

metric_1: -0.715% [-1.807%, 0.378%]
metric_2: 0.898% [0.220%, 1.576%]
Parameterization has too many items to render on hover (15).", "Arm 0_53

metric_1: -2.187% [-3.261%, -1.113%]
metric_2: -0.924% [-1.595%, -0.252%]
Parameterization has too many items to render on hover (15).", "Arm 0_54

metric_1: -3.62% [-4.663%, -2.577%]
metric_2: -1.266% [-1.928%, -0.603%]
Parameterization has too many items to render on hover (15).", "Arm 0_55

metric_1: -3.085% [-4.149%, -2.022%]
metric_2: -1.056% [-1.729%, -0.382%]
Parameterization has too many items to render on hover (15).", "Arm 0_56

metric_1: 0.065% [-1.043%, 1.174%]
metric_2: 1.104% [0.414%, 1.793%]
Parameterization has too many items to render on hover (15).", "Arm 0_57

metric_1: -8.045% [-9.053%, -7.036%]
metric_2: -3.231% [-3.897%, -2.565%]
Parameterization has too many items to render on hover (15).", "Arm 0_58

metric_1: -2.417% [-3.486%, -1.347%]
metric_2: -1.461% [-2.139%, -0.784%]
Parameterization has too many items to render on hover (15).", "Arm 0_59

metric_1: 2.155% [1.023%, 3.288%]
metric_2: 0.992% [0.310%, 1.673%]
Parameterization has too many items to render on hover (15).", "Arm 0_6

metric_1: -3.551% [-4.616%, -2.486%]
metric_2: -1.321% [-1.986%, -0.656%]
Parameterization has too many items to render on hover (15).", "Arm 0_60

metric_1: 3.0% [1.850%, 4.149%]
metric_2: 2.8% [2.112%, 3.488%]
Parameterization has too many items to render on hover (15).", "Arm 0_61

metric_1: 5.54% [4.346%, 6.735%]
metric_2: 10.809% [10.028%, 11.590%]
Parameterization has too many items to render on hover (15).", "Arm 0_62

metric_1: -13.137% [-14.095%, -12.178%]
metric_2: -3.839% [-4.509%, -3.169%]
Parameterization has too many items to render on hover (15).", "Arm 0_63

metric_1: -9.683% [-10.662%, -8.703%]
metric_2: -2.765% [-3.432%, -2.099%]
Parameterization has too many items to render on hover (15).", "Arm 0_7

metric_1: -4.089% [-5.138%, -3.041%]
metric_2: -1.576% [-2.248%, -0.903%]
Parameterization has too many items to render on hover (15).", "Arm 0_8

metric_1: -2.46% [-3.530%, -1.390%]
metric_2: -2.179% [-2.846%, -1.511%]
Parameterization has too many items to render on hover (15).", "Arm 0_9

metric_1: -7.98% [-8.985%, -6.976%]
metric_2: -2.79% [-3.451%, -2.128%]
Parameterization has too many items to render on hover (15).", "Arm status_quo

metric_1: -0.0% [-0.521%, 0.521%]
metric_2: -0.0% [-0.318%, 0.318%]
Parameterization has too many items to render on hover (15)." ], "type": "scatter", "visible": false, "x": [ -17.15914516790823, -11.573417823939199, -11.361067490605143, 3.037717284235046, -3.069542768219942, -2.864577054527247, -0.8171324453854826, -7.780377203320161, 2.475352747757105, -1.1963952539982303, -1.3598025263922875, -14.817429080033943, -3.2723491422755444, 0.08873184010870934, 0.5359430531455063, 1.8145980354381517, -6.471358153849177, 0.8877430970467538, 0.7286746192275023, 0.3510369131998831, -6.984895697391649, -7.505623786753492, 1.5767742612758522, -2.2143528351212733, -4.595608658658678, -19.18909654051259, 3.4292680540776312, 3.3710456099492796, -3.889937358175768, 0.6139708178717305, 0.5900679364090252, 0.8653940325415319, -1.2680473963845287, -0.1686822762476349, -2.893033151686222, -8.623721510464856, -5.156592129047242, -4.878241533976907, -3.655342272989576, 4.483189117418239, -3.0603072946347587, -6.921569399104486, -3.4781006474949763, -4.545929170059967, -0.305361571312272, -2.068706090724009, 0.36738131081906034, -4.596829363692114, -0.7145714514809897, -2.1870440120521244, -3.619802243175367, -3.0854247199357574, 0.06528116501171306, -8.04466137206524, -2.416515121000999, 2.1550821626518215, -3.550888972902983, 2.999501801038642, 5.540352816509654, -13.136826425368891, -9.682667832954975, -4.089456620979975, -2.4600469634137005, -7.980395623816304, -0.0 ], "y": [ -5.746731056831749, -3.560535696444575, -3.610268946914751, 2.5135502228088797, -0.5965513852397577, -1.0116808329390385, -0.8182560448021005, -2.4071834175723335, 2.5577489299929255, 0.348192434742279, -1.3114357729283859, -3.3382598457223502, -1.0068955594731426, -0.06867630083734555, 0.10424166292801242, -0.07210829452142886, -3.188401274811581, 1.960086317210278, 0.012674904654773453, 1.1203410785772772, 0.6155055435112974, -2.5504391581007666, 5.964668435375144, -0.4205314884998786, -2.726644960728147, -5.859418405279977, 3.1966830537300392, 3.950198735676712, 2.0568967995693455, 0.652872009905956, 1.286780707668062, 4.62085929335188, -0.7622630447726769, 0.123503526062618, -1.5431517157186252, -3.2650236570623457, -1.7414179188197445, -1.1257173899572386, -1.3809544654506094, 7.651708533863019, -2.2015816842718356, -3.063188080332386, 4.012625447421733, -1.168849751600452, 0.7526902410881314, -0.7604599707221315, 2.2308412374353317, -1.371441760784444, 0.8978911718791985, -0.9236769844543923, -1.2656161965636414, -1.0555488550285586, 1.1039153988973123, -3.2310973947137103, -1.4614566797717248, 0.9918050829129264, -1.3210922238627325, 2.8001328033355772, 10.809115959705165, -3.839017981482167, -2.765325873487098, -1.5757183832623392, -2.1787372563938, -2.789593215553577, -0.0 ] }, { "error_x": { "array": [ 0.9032151733520527, 0.9624260346524522, 0.9593909284698995, 1.1347609507759298, 1.0408257278018285, 1.0435860695228432, 1.0711864083714089, 1.0156140087849477, 1.128257833835402, 1.0788991832923005, 1.0706037001289912, 0.9284671425890606, 1.0523829338844282, 1.0903373892267718, 1.0861328996665252, 1.1145865581026473, 1.0063514862723812, 1.0994776629118859, 1.091675914931962, 1.0844593315915725, 1.0017853270032084, 0.9935701990690048, 1.1127508961856405, 1.0668110378139575, 1.0328601627012277, 0.8797268687204242, 1.140518932271723, 1.1417536403033617, 1.0764096893237975, 1.0965566556283748, 1.0984998396729218, 1.102540333829891, 1.0794770222090497, 1.085962611129493, 1.0433251902342626, 0.9820278721895025, 1.0166480934308502, 1.0343546221526094, 1.0382944853121021, 1.1551456071793769, 1.0464931893207308, 1.0023861997986674, 1.0516802404518037, 1.0346896327290815, 1.08327143184596, 1.0698309404523143, 1.0904297395971863, 1.0222860350150513, 1.0810704079796398, 1.0613893596292459, 1.0320103439592427, 1.0528002249874768, 1.0912064123686547, 0.9988930110467326, 1.0571937806819418, 1.1193739547640122, 1.0530800272030827, 1.1303999126786766, 1.180499014903316, 0.9513079288502773, 0.9719033134076178, 1.03791308728753, 1.0588852902469934, 0.9945314913677392, 0.5199803498226312 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "error_y": { "array": [ 0.6547387343317975, 0.6532033115837151, 0.6609466315731528, 0.6813554440847576, 0.6694705491295897, 0.6591303137747925, 0.6563086390178189, 0.6508039407989954, 0.6839216136881153, 0.6604406759568969, 0.6601787231575951, 0.6565366929811817, 0.6613093240694986, 0.6561140874958142, 0.6633706122047732, 0.6637799634962176, 0.6555405967295402, 0.6704566164608853, 0.66273573692496, 0.6685871803108022, 0.6628627361853555, 0.6572002148831203, 0.712446900341419, 0.6636937933944925, 0.6586242603445244, 0.6475500126287197, 0.6773760670047011, 0.6859237814996093, 0.6688072097037889, 0.6516465341587597, 0.6738024004862555, 0.7036162417194723, 0.6668939573219893, 0.6610547090506331, 0.6553518681013905, 0.6508051033775936, 0.64797757405811, 0.6713714106339492, 0.656723211631772, 0.7346090086077302, 0.6642524778308558, 0.6574536033119194, 0.7031723757695748, 0.6494774398002832, 0.6704885519899918, 0.6622056608350378, 0.6797639514603396, 0.6735895146091605, 0.6692892385552365, 0.6620882696925839, 0.6531942830594355, 0.6650013545172943, 0.6769982685390781, 0.6574731612170758, 0.6674033123702455, 0.6722373506713744, 0.6559893752039715, 0.6748097809173098, 0.7697444470145935, 0.6626605822603224, 0.6581382444319371, 0.663286850538724, 0.658955602798785, 0.6526018616070007, 0.3176861573292605 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "hoverinfo": "text", "marker": { "color": "rgba(128,177,211,1)" }, "mode": "markers", "name": "In-sample", "showlegend": true, "text": [ "Arm 0_0

metric_1: -17.022% [-17.925%, -16.119%]
metric_2: -5.683% [-6.338%, -5.028%]
Parameterization has too many items to render on hover (15).", "Arm 0_1

metric_1: -11.473% [-12.436%, -10.511%]
metric_2: -3.526% [-4.180%, -2.873%]
Parameterization has too many items to render on hover (15).", "Arm 0_10

metric_1: -11.26% [-12.220%, -10.301%]
metric_2: -3.535% [-4.196%, -2.874%]
Parameterization has too many items to render on hover (15).", "Arm 0_11

metric_1: 2.99% [1.855%, 4.125%]
metric_2: 2.542% [1.861%, 3.224%]
Parameterization has too many items to render on hover (15).", "Arm 0_12

metric_1: -3.205% [-4.246%, -2.164%]
metric_2: -0.629% [-1.298%, 0.041%]
Parameterization has too many items to render on hover (15).", "Arm 0_13

metric_1: -2.828% [-3.871%, -1.784%]
metric_2: -1.055% [-1.714%, -0.396%]
Parameterization has too many items to render on hover (15).", "Arm 0_14

metric_1: -0.807% [-1.878%, 0.264%]
metric_2: -0.832% [-1.488%, -0.176%]
Parameterization has too many items to render on hover (15).", "Arm 0_15

metric_1: -7.632% [-8.648%, -6.617%]
metric_2: -2.319% [-2.970%, -1.669%]
Parameterization has too many items to render on hover (15).", "Arm 0_16

metric_1: 2.493% [1.365%, 3.621%]
metric_2: 2.582% [1.898%, 3.266%]
Parameterization has too many items to render on hover (15).", "Arm 0_17

metric_1: -1.148% [-2.227%, -0.069%]
metric_2: 0.326% [-0.334%, 0.987%]
Parameterization has too many items to render on hover (15).", "Arm 0_18

metric_1: -1.417% [-2.488%, -0.346%]
metric_2: -1.321% [-1.981%, -0.661%]
Parameterization has too many items to render on hover (15).", "Arm 0_19

metric_1: -14.735% [-15.664%, -13.807%]
metric_2: -3.315% [-3.971%, -2.658%]
Parameterization has too many items to render on hover (15).", "Arm 0_2

metric_1: -3.211% [-4.263%, -2.158%]
metric_2: -0.992% [-1.653%, -0.331%]
Parameterization has too many items to render on hover (15).", "Arm 0_20

metric_1: 0.073% [-1.017%, 1.164%]
metric_2: -0.063% [-0.719%, 0.594%]
Parameterization has too many items to render on hover (15).", "Arm 0_21

metric_1: 0.528% [-0.558%, 1.614%]
metric_2: 0.066% [-0.597%, 0.730%]
Parameterization has too many items to render on hover (15).", "Arm 0_22

metric_1: 1.746% [0.632%, 2.861%]
metric_2: -0.055% [-0.719%, 0.608%]
Parameterization has too many items to render on hover (15).", "Arm 0_23

metric_1: -6.404% [-7.411%, -5.398%]
metric_2: -3.15% [-3.805%, -2.494%]
Parameterization has too many items to render on hover (15).", "Arm 0_24

metric_1: 0.836% [-0.264%, 1.935%]
metric_2: 1.893% [1.223%, 2.564%]
Parameterization has too many items to render on hover (15).", "Arm 0_25

metric_1: 0.699% [-0.393%, 1.790%]
metric_2: 0.024% [-0.639%, 0.686%]
Parameterization has too many items to render on hover (15).", "Arm 0_26

metric_1: 0.319% [-0.766%, 1.403%]
metric_2: 1.077% [0.409%, 1.746%]
Parameterization has too many items to render on hover (15).", "Arm 0_27

metric_1: -6.918% [-7.920%, -5.916%]
metric_2: 0.651% [-0.012%, 1.314%]
Parameterization has too many items to render on hover (15).", "Arm 0_28

metric_1: -7.463% [-8.457%, -6.469%]
metric_2: -2.49% [-3.147%, -1.832%]
Parameterization has too many items to render on hover (15).", "Arm 0_29

metric_1: 1.48% [0.367%, 2.592%]
metric_2: 5.858% [5.145%, 6.570%]
Parameterization has too many items to render on hover (15).", "Arm 0_3

metric_1: -2.25% [-3.317%, -1.183%]
metric_2: -0.448% [-1.112%, 0.215%]
Parameterization has too many items to render on hover (15).", "Arm 0_30

metric_1: -4.65% [-5.683%, -3.617%]
metric_2: -2.712% [-3.371%, -2.054%]
Parameterization has too many items to render on hover (15).", "Arm 0_31

metric_1: -19.06% [-19.939%, -18.180%]
metric_2: -5.792% [-6.440%, -5.145%]
Parameterization has too many items to render on hover (15).", "Arm 0_32

metric_1: 3.344% [2.204%, 4.485%]
metric_2: 3.158% [2.481%, 3.835%]
Parameterization has too many items to render on hover (15).", "Arm 0_33

metric_1: 3.298% [2.156%, 4.440%]
metric_2: 3.853% [3.167%, 4.539%]
Parameterization has too many items to render on hover (15).", "Arm 0_34

metric_1: -3.845% [-4.922%, -2.769%]
metric_2: 2.026% [1.357%, 2.695%]
Parameterization has too many items to render on hover (15).", "Arm 0_35

metric_1: 0.616% [-0.480%, 1.713%]
metric_2: 0.619% [-0.033%, 1.270%]
Parameterization has too many items to render on hover (15).", "Arm 0_36

metric_1: 0.606% [-0.493%, 1.704%]
metric_2: 1.328% [0.654%, 2.002%]
Parameterization has too many items to render on hover (15).", "Arm 0_37

metric_1: 0.801% [-0.301%, 1.904%]
metric_2: 4.539% [3.836%, 5.243%]
Parameterization has too many items to render on hover (15).", "Arm 0_38

metric_1: -1.23% [-2.310%, -0.151%]
metric_2: -0.74% [-1.407%, -0.074%]
Parameterization has too many items to render on hover (15).", "Arm 0_39

metric_1: -0.189% [-1.275%, 0.897%]
metric_2: 0.11% [-0.551%, 0.771%]
Parameterization has too many items to render on hover (15).", "Arm 0_4

metric_1: -2.877% [-3.920%, -1.834%]
metric_2: -1.538% [-2.193%, -0.883%]
Parameterization has too many items to render on hover (15).", "Arm 0_40

metric_1: -8.667% [-9.649%, -7.685%]
metric_2: -3.287% [-3.937%, -2.636%]
Parameterization has too many items to render on hover (15).", "Arm 0_41

metric_1: -5.112% [-6.129%, -4.096%]
metric_2: -1.776% [-2.424%, -1.128%]
Parameterization has too many items to render on hover (15).", "Arm 0_42

metric_1: -4.866% [-5.900%, -3.831%]
metric_2: -1.107% [-1.779%, -0.436%]
Parameterization has too many items to render on hover (15).", "Arm 0_43

metric_1: -3.621% [-4.659%, -2.582%]
metric_2: -1.374% [-2.031%, -0.717%]
Parameterization has too many items to render on hover (15).", "Arm 0_44

metric_1: 4.416% [3.261%, 5.571%]
metric_2: 7.531% [6.797%, 8.266%]
Parameterization has too many items to render on hover (15).", "Arm 0_45

metric_1: -3.094% [-4.140%, -2.047%]
metric_2: -2.19% [-2.854%, -1.526%]
Parameterization has too many items to render on hover (15).", "Arm 0_46

metric_1: -6.987% [-7.990%, -5.985%]
metric_2: -3.082% [-3.740%, -2.425%]
Parameterization has too many items to render on hover (15).", "Arm 0_47

metric_1: -3.456% [-4.507%, -2.404%]
metric_2: 3.964% [3.261%, 4.667%]
Parameterization has too many items to render on hover (15).", "Arm 0_48

metric_1: -4.487% [-5.521%, -3.452%]
metric_2: -1.178% [-1.827%, -0.528%]
Parameterization has too many items to render on hover (15).", "Arm 0_49

metric_1: -0.276% [-1.360%, 0.807%]
metric_2: 0.752% [0.081%, 1.422%]
Parameterization has too many items to render on hover (15).", "Arm 0_5

metric_1: -2.093% [-3.163%, -1.023%]
metric_2: -0.772% [-1.435%, -0.110%]
Parameterization has too many items to render on hover (15).", "Arm 0_50

metric_1: 0.34% [-0.750%, 1.431%]
metric_2: 2.197% [1.517%, 2.877%]
Parameterization has too many items to render on hover (15).", "Arm 0_51

metric_1: -4.611% [-5.633%, -3.589%]
metric_2: -1.337% [-2.011%, -0.664%]
Parameterization has too many items to render on hover (15).", "Arm 0_52

metric_1: -0.757% [-1.838%, 0.324%]
metric_2: 0.878% [0.208%, 1.547%]
Parameterization has too many items to render on hover (15).", "Arm 0_53

metric_1: -2.166% [-3.228%, -1.105%]
metric_2: -0.911% [-1.573%, -0.249%]
Parameterization has too many items to render on hover (15).", "Arm 0_54

metric_1: -3.611% [-4.643%, -2.579%]
metric_2: -1.267% [-1.920%, -0.613%]
Parameterization has too many items to render on hover (15).", "Arm 0_55

metric_1: -3.053% [-4.106%, -2.000%]
metric_2: -1.055% [-1.720%, -0.390%]
Parameterization has too many items to render on hover (15).", "Arm 0_56

metric_1: 0.115% [-0.976%, 1.206%]
metric_2: 1.138% [0.461%, 1.815%]
Parameterization has too many items to render on hover (15).", "Arm 0_57

metric_1: -8.012% [-9.011%, -7.013%]
metric_2: -3.221% [-3.878%, -2.563%]
Parameterization has too many items to render on hover (15).", "Arm 0_58

metric_1: -2.444% [-3.501%, -1.387%]
metric_2: -1.454% [-2.122%, -0.787%]
Parameterization has too many items to render on hover (15).", "Arm 0_59

metric_1: 2.105% [0.986%, 3.225%]
metric_2: 0.993% [0.320%, 1.665%]
Parameterization has too many items to render on hover (15).", "Arm 0_6

metric_1: -3.549% [-4.602%, -2.496%]
metric_2: -1.318% [-1.974%, -0.662%]
Parameterization has too many items to render on hover (15).", "Arm 0_60

metric_1: 2.966% [1.836%, 4.097%]
metric_2: 2.781% [2.106%, 3.455%]
Parameterization has too many items to render on hover (15).", "Arm 0_61

metric_1: 5.393% [4.212%, 6.573%]
metric_2: 10.553% [9.783%, 11.322%]
Parameterization has too many items to render on hover (15).", "Arm 0_62

metric_1: -13.026% [-13.978%, -12.075%]
metric_2: -3.787% [-4.450%, -3.125%]
Parameterization has too many items to render on hover (15).", "Arm 0_63

metric_1: -9.619% [-10.591%, -8.647%]
metric_2: -2.738% [-3.397%, -2.080%]
Parameterization has too many items to render on hover (15).", "Arm 0_7

metric_1: -4.118% [-5.156%, -3.080%]
metric_2: -1.61% [-2.273%, -0.947%]
Parameterization has too many items to render on hover (15).", "Arm 0_8

metric_1: -2.464% [-3.523%, -1.405%]
metric_2: -2.148% [-2.807%, -1.489%]
Parameterization has too many items to render on hover (15).", "Arm 0_9

metric_1: -7.957% [-8.951%, -6.962%]
metric_2: -2.794% [-3.447%, -2.142%]
Parameterization has too many items to render on hover (15).", "Arm status_quo

metric_1: -0.0% [-0.520%, 0.520%]
metric_2: -0.0% [-0.318%, 0.318%]
Parameterization has too many items to render on hover (15)." ], "type": "scatter", "visible": true, "x": [ -17.022045315104542, -11.473276363575263, -11.260309485449103, 2.9899364287480865, -3.2049806778237757, -2.827647808086468, -0.8072357684589017, -7.632201738791567, 2.4931910046363135, -1.1483177933819395, -1.4169105153422092, -14.735353419604191, -3.2108750242520334, 0.07332105300986574, 0.52794229128138, 1.7461298706241524, -6.404301715961119, 0.8357526716703064, 0.6988214644260046, 0.3185581570295636, -6.918118515914437, -7.462960459390675, 1.4797223030792654, -2.250057847895754, -4.649788400829341, -19.059723742205165, 3.3442377220374424, 3.2977920613814318, -3.845344844296581, 0.6160918095707203, 0.605873558955114, 0.8012482568804604, -1.2301194692314896, -0.18910424701708023, -2.8771572632426596, -8.666686434035148, -5.1123185935816275, -4.86569050603379, -3.620616821502621, 4.416040423353074, -3.093977759193243, -6.987479150965008, -3.45561072825807, -4.486522144334903, -0.27632651192689084, -2.092939160334957, 0.34040916740729205, -4.6111158309365, -0.7572230824447213, -2.1663524739549853, -3.611088388568022, -3.053027739941141, 0.11513485293184113, -8.011705772759294, -2.4442287384702572, 2.1052356784416553, -3.5492459592908614, 2.966172129108271, 5.392515165139946, -13.026404058841306, -9.619176461732291, -4.118044655666446, -2.4636655602657878, -7.956569241984238, -0.0 ], "y": [ -5.682890950120342, -3.5263840662392654, -3.5354295227440935, 2.542378866552539, -0.6285418092402999, -1.0553405972573464, -0.8320014955517723, -2.3193462375707914, 2.58163977526919, 0.32621268542555526, -1.3207797795363851, -3.314816695063182, -0.9921710458941605, -0.06250694414329654, 0.06640049636248754, -0.05540222390315327, -3.1495291448321576, 1.8934882315080965, 0.023529017577203423, 1.0773772731169438, 0.6512642256730126, -2.489527283561809, 5.857858819488534, -0.4484318638610448, -2.712182537468451, -5.792314875460171, 3.1580957792386144, 3.8533176800717563, 2.026262991103158, 0.6187928699607773, 1.3280865831055046, 4.5394243735749775, -0.7404858515521753, 0.10968492064137063, -1.5381048845570704, -3.2865434005114005, -1.7761680789956928, -1.1071508425287317, -1.3739381694523336, 7.5312245476719, -2.1901762525545276, -3.082214357601481, 3.963995297169013, -1.1779086966890597, 0.7516740725998268, -0.772463613395467, 2.196834434560228, -1.3372360602549658, 0.8775653888918572, -0.9111761852726025, -1.2666733248842346, -1.055168296030082, 1.1378418002527275, -3.2209479427630807, -1.4541461401402338, 0.9926717601692114, -1.3177840022000746, 2.7806338967179443, 10.552746740900881, -3.7873810524577847, -2.7384540934203034, -1.6100825481553491, -2.147955641504198, -2.7941662700346765, -0.0 ] }, { "error_x": { "array": [ 4.620799942815001, 4.341764333801678, 4.67115577126623 ], "color": "rgba(251,128,114,0.4)", "type": "data" }, "error_y": { "array": [ 2.5437250774011937, 2.2532861566721074, 2.6503979821172843 ], "color": "rgba(251,128,114,0.4)", "type": "data" }, "hoverinfo": "text", "marker": { "color": "rgba(251,128,114,1)" }, "mode": "markers", "name": "constraint_5", "showlegend": true, "text": [ "Arm 2663

metric_1: 5.026% [0.405%, 9.647%]
metric_2: 3.761% [1.217%, 6.305%]
Parameterization has too many items to render on hover (15).", "Arm 5174

metric_1: 4.459% [0.117%, 8.800%]
metric_2: 3.719% [1.466%, 5.972%]
Parameterization has too many items to render on hover (15).", "Arm 7d93

metric_1: 4.418% [-0.254%, 9.089%]
metric_2: 3.722% [1.072%, 6.372%]
Parameterization has too many items to render on hover (15)." ], "type": "scatter", "visible": true, "x": [ 5.02607550500538, 4.458715026100761, 4.4175213815962335 ], "y": [ 3.7610174232852804, 3.7188506476744347, 3.7220056829809645 ] } ], "layout": { "annotations": [ { "showarrow": false, "text": "Show CI", "x": 1.18, "xref": "paper", "y": 0.7, "yanchor": "middle", "yref": "paper" }, { "showarrow": false, "text": "Type", "x": 1.18, "xref": "paper", "y": 0.6, "yanchor": "middle", "yref": "paper" } ], "font": { "size": 10 }, "height": 600, "hovermode": "closest", "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "sequentialminus": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "title": { "text": "Objective Tradeoffs" }, "updatemenus": [ { "buttons": [ { "args": [ { "error_x.thickness": 2, "error_x.width": 4, "error_y.thickness": 2, "error_y.width": 4 } ], "label": "Yes", "method": "restyle" }, { "args": [ { "error_x.thickness": 0, "error_x.width": 0, "error_y.thickness": 0, "error_y.width": 0 } ], "label": "No", "method": "restyle" } ], "x": 1.25, "xanchor": "left", "y": 0.67, "yanchor": "middle" }, { "buttons": [ { "args": [ { "visible": [ false, true, true ] } ], "label": "Modeled", "method": "restyle" }, { "args": [ { "visible": [ true, false, false ] } ], "label": "Observed", "method": "restyle" } ], "x": 1.25, "xanchor": "left", "y": 0.57, "yanchor": "middle" } ], "width": 800, "xaxis": { "title": { "text": "metric_1 (%)" }, "zeroline": true, "zerolinecolor": "red" }, "yaxis": { "title": { "text": "metric_2 (%)" }, "zeroline": true, "zerolinecolor": "red" } } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from ax.plot.scatter import plot_multiple_metrics\n", "render(plot_multiple_metrics(\n", " gp,\n", " metric_x=METRIC_X_AXIS,\n", " metric_y=METRIC_Y_AXIS,\n", " generator_runs_dict={\n", " 'constraint_5': constraint_5_results\n", " }\n", "))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It is important to note that the treatment of constraints in GP EI is probabilistic. The acquisition function weights our objective by the probability that each constraint is feasible. Thus, we may allow points with a very small probability of violating the constraint to be generated, as long as the chance of the points increasing our objective is high enough. \n", "\n", "You can see above that the point estimate for each point is significantly below a 5% increase in the constraint metric, but that there is uncertainty in our prediction, and the tail probabilities do include probabilities of small regressions beyond 5%. " ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "collapsed": true }, "outputs": [], "source": [ "constraint_1 = OutcomeConstraint(metric=constraint_metric, op=ComparisonOp.LEQ, bound=1)\n", "constraint_1_results = gp.gen(\n", " n=3, \n", " optimization_config=OptimizationConfig(\n", " objective=Objective(objective_metric),\n", " outcome_constraints=[constraint_1],\n", " )\n", ")" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "scrolled": true }, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "linkText": "Export to plot.ly", "plotlyServerURL": "https://plot.ly", "showLink": false }, "data": [ { "error_x": { "array": [ 0.9105274690527578, 0.9725977092158347, 0.9694448615326413, 1.1526497268980418, 1.056129089525654, 1.0581150790455403, 1.0853729636494531, 1.025297982327309, 1.1509354346821015, 1.0946224534113276, 1.080994735603789, 0.9349800939958878, 1.0669782977880082, 1.1063382644746038, 1.109747945764228, 1.1312428838918263, 1.0151345627364643, 1.113745664637434, 1.1070645021428274, 1.098298041303658, 1.011825279703854, 1.004076551317031, 1.1247628081965457, 1.0803499998271806, 1.0451840940805415, 0.8854990204077908, 1.1551313262975833, 1.1582540625351196, 1.088250532740983, 1.1152606223383992, 1.1132227750465398, 1.1136443399827738, 1.0928257948335294, 1.098342295748459, 1.0562137210185467, 0.9929499124720941, 1.0275183054522712, 1.0461406507525532, 1.0507233488011212, 1.1693056777789836, 1.0564300559878494, 1.012746475490335, 1.060503126319525, 1.0453978622119475, 1.0972242260284542, 1.0810868080421863, 1.103084437746675, 1.0346109289273437, 1.0925727929561801, 1.0737167310801996, 1.042945064542918, 1.0636626539145406, 1.108765950358755, 1.0083792696638647, 1.069964804645196, 1.1324679783490406, 1.0646727177942992, 1.149093505549881, 1.19450723179394, 0.9584867213686696, 0.9797729070203455, 1.048287897726107, 1.0699714924908204, 1.0044799847095478, 0.5205646439608032 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "error_y": { "array": [ 0.6639175585103954, 0.6630929411298598, 0.6719874423233675, 0.6935680367786019, 0.6821107678411614, 0.6721157287254815, 0.6675598394585217, 0.6588395471546759, 0.7005079335820925, 0.6734693768172479, 0.6678033173208016, 0.6639242874696609, 0.6729502923953926, 0.6673739566339196, 0.6808476824923786, 0.675618453499239, 0.6630338514495195, 0.6814341339540937, 0.6732139305974362, 0.678993016475866, 0.6717812932415321, 0.6668482051137559, 0.7215454438634975, 0.6740882429740953, 0.6694676526234568, 0.6554367403862607, 0.6871145937679641, 0.6973627667199596, 0.677839082032618, 0.6656099293584739, 0.6853610253860684, 0.7127045342173167, 0.6774929784721044, 0.6697791158632792, 0.665276789680729, 0.6609617898538276, 0.6573670880847166, 0.6812573532252995, 0.6669304645971238, 0.7456805630690589, 0.6722619172607951, 0.6674666794827154, 0.71101954162863, 0.6581471114518421, 0.681784654518447, 0.6708890082180987, 0.6891795059009299, 0.6846723076532707, 0.6778079742730775, 0.6716150419025536, 0.6624185510214552, 0.6736751066980704, 0.6894763100114708, 0.6659339720000773, 0.6774968737338238, 0.6814011092512952, 0.6647910226422914, 0.6882834695714566, 0.7808801986391988, 0.6703813570659151, 0.666212972377905, 0.6724199160716084, 0.6676592507580481, 0.6614165175994707, 0.3181115760489757 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "hoverinfo": "text", "marker": { "color": "rgba(128,177,211,1)" }, "mode": "markers", "name": "In-sample", "showlegend": true, "text": [ "Arm 0_0

metric_1: -17.159% [-18.070%, -16.249%]
metric_2: -5.747% [-6.411%, -5.083%]
Parameterization has too many items to render on hover (15).", "Arm 0_1

metric_1: -11.573% [-12.546%, -10.601%]
metric_2: -3.561% [-4.224%, -2.897%]
Parameterization has too many items to render on hover (15).", "Arm 0_10

metric_1: -11.361% [-12.331%, -10.392%]
metric_2: -3.61% [-4.282%, -2.938%]
Parameterization has too many items to render on hover (15).", "Arm 0_11

metric_1: 3.038% [1.885%, 4.190%]
metric_2: 2.514% [1.820%, 3.207%]
Parameterization has too many items to render on hover (15).", "Arm 0_12

metric_1: -3.07% [-4.126%, -2.013%]
metric_2: -0.597% [-1.279%, 0.086%]
Parameterization has too many items to render on hover (15).", "Arm 0_13

metric_1: -2.865% [-3.923%, -1.806%]
metric_2: -1.012% [-1.684%, -0.340%]
Parameterization has too many items to render on hover (15).", "Arm 0_14

metric_1: -0.817% [-1.903%, 0.268%]
metric_2: -0.818% [-1.486%, -0.151%]
Parameterization has too many items to render on hover (15).", "Arm 0_15

metric_1: -7.78% [-8.806%, -6.755%]
metric_2: -2.407% [-3.066%, -1.748%]
Parameterization has too many items to render on hover (15).", "Arm 0_16

metric_1: 2.475% [1.324%, 3.626%]
metric_2: 2.558% [1.857%, 3.258%]
Parameterization has too many items to render on hover (15).", "Arm 0_17

metric_1: -1.196% [-2.291%, -0.102%]
metric_2: 0.348% [-0.325%, 1.022%]
Parameterization has too many items to render on hover (15).", "Arm 0_18

metric_1: -1.36% [-2.441%, -0.279%]
metric_2: -1.311% [-1.979%, -0.644%]
Parameterization has too many items to render on hover (15).", "Arm 0_19

metric_1: -14.817% [-15.752%, -13.882%]
metric_2: -3.338% [-4.002%, -2.674%]
Parameterization has too many items to render on hover (15).", "Arm 0_2

metric_1: -3.272% [-4.339%, -2.205%]
metric_2: -1.007% [-1.680%, -0.334%]
Parameterization has too many items to render on hover (15).", "Arm 0_20

metric_1: 0.089% [-1.018%, 1.195%]
metric_2: -0.069% [-0.736%, 0.599%]
Parameterization has too many items to render on hover (15).", "Arm 0_21

metric_1: 0.536% [-0.574%, 1.646%]
metric_2: 0.104% [-0.577%, 0.785%]
Parameterization has too many items to render on hover (15).", "Arm 0_22

metric_1: 1.815% [0.683%, 2.946%]
metric_2: -0.072% [-0.748%, 0.604%]
Parameterization has too many items to render on hover (15).", "Arm 0_23

metric_1: -6.471% [-7.486%, -5.456%]
metric_2: -3.188% [-3.851%, -2.525%]
Parameterization has too many items to render on hover (15).", "Arm 0_24

metric_1: 0.888% [-0.226%, 2.001%]
metric_2: 1.96% [1.279%, 2.642%]
Parameterization has too many items to render on hover (15).", "Arm 0_25

metric_1: 0.729% [-0.378%, 1.836%]
metric_2: 0.013% [-0.661%, 0.686%]
Parameterization has too many items to render on hover (15).", "Arm 0_26

metric_1: 0.351% [-0.747%, 1.449%]
metric_2: 1.12% [0.441%, 1.799%]
Parameterization has too many items to render on hover (15).", "Arm 0_27

metric_1: -6.985% [-7.997%, -5.973%]
metric_2: 0.616% [-0.056%, 1.287%]
Parameterization has too many items to render on hover (15).", "Arm 0_28

metric_1: -7.506% [-8.510%, -6.502%]
metric_2: -2.55% [-3.217%, -1.884%]
Parameterization has too many items to render on hover (15).", "Arm 0_29

metric_1: 1.577% [0.452%, 2.702%]
metric_2: 5.965% [5.243%, 6.686%]
Parameterization has too many items to render on hover (15).", "Arm 0_3

metric_1: -2.214% [-3.295%, -1.134%]
metric_2: -0.421% [-1.095%, 0.254%]
Parameterization has too many items to render on hover (15).", "Arm 0_30

metric_1: -4.596% [-5.641%, -3.550%]
metric_2: -2.727% [-3.396%, -2.057%]
Parameterization has too many items to render on hover (15).", "Arm 0_31

metric_1: -19.189% [-20.075%, -18.304%]
metric_2: -5.859% [-6.515%, -5.204%]
Parameterization has too many items to render on hover (15).", "Arm 0_32

metric_1: 3.429% [2.274%, 4.584%]
metric_2: 3.197% [2.510%, 3.884%]
Parameterization has too many items to render on hover (15).", "Arm 0_33

metric_1: 3.371% [2.213%, 4.529%]
metric_2: 3.95% [3.253%, 4.648%]
Parameterization has too many items to render on hover (15).", "Arm 0_34

metric_1: -3.89% [-4.978%, -2.802%]
metric_2: 2.057% [1.379%, 2.735%]
Parameterization has too many items to render on hover (15).", "Arm 0_35

metric_1: 0.614% [-0.501%, 1.729%]
metric_2: 0.653% [-0.013%, 1.318%]
Parameterization has too many items to render on hover (15).", "Arm 0_36

metric_1: 0.59% [-0.523%, 1.703%]
metric_2: 1.287% [0.601%, 1.972%]
Parameterization has too many items to render on hover (15).", "Arm 0_37

metric_1: 0.865% [-0.248%, 1.979%]
metric_2: 4.621% [3.908%, 5.334%]
Parameterization has too many items to render on hover (15).", "Arm 0_38

metric_1: -1.268% [-2.361%, -0.175%]
metric_2: -0.762% [-1.440%, -0.085%]
Parameterization has too many items to render on hover (15).", "Arm 0_39

metric_1: -0.169% [-1.267%, 0.930%]
metric_2: 0.124% [-0.546%, 0.793%]
Parameterization has too many items to render on hover (15).", "Arm 0_4

metric_1: -2.893% [-3.949%, -1.837%]
metric_2: -1.543% [-2.208%, -0.878%]
Parameterization has too many items to render on hover (15).", "Arm 0_40

metric_1: -8.624% [-9.617%, -7.631%]
metric_2: -3.265% [-3.926%, -2.604%]
Parameterization has too many items to render on hover (15).", "Arm 0_41

metric_1: -5.157% [-6.184%, -4.129%]
metric_2: -1.741% [-2.399%, -1.084%]
Parameterization has too many items to render on hover (15).", "Arm 0_42

metric_1: -4.878% [-5.924%, -3.832%]
metric_2: -1.126% [-1.807%, -0.444%]
Parameterization has too many items to render on hover (15).", "Arm 0_43

metric_1: -3.655% [-4.706%, -2.605%]
metric_2: -1.381% [-2.048%, -0.714%]
Parameterization has too many items to render on hover (15).", "Arm 0_44

metric_1: 4.483% [3.314%, 5.652%]
metric_2: 7.652% [6.906%, 8.397%]
Parameterization has too many items to render on hover (15).", "Arm 0_45

metric_1: -3.06% [-4.117%, -2.004%]
metric_2: -2.202% [-2.874%, -1.529%]
Parameterization has too many items to render on hover (15).", "Arm 0_46

metric_1: -6.922% [-7.934%, -5.909%]
metric_2: -3.063% [-3.731%, -2.396%]
Parameterization has too many items to render on hover (15).", "Arm 0_47

metric_1: -3.478% [-4.539%, -2.418%]
metric_2: 4.013% [3.302%, 4.724%]
Parameterization has too many items to render on hover (15).", "Arm 0_48

metric_1: -4.546% [-5.591%, -3.501%]
metric_2: -1.169% [-1.827%, -0.511%]
Parameterization has too many items to render on hover (15).", "Arm 0_49

metric_1: -0.305% [-1.403%, 0.792%]
metric_2: 0.753% [0.071%, 1.434%]
Parameterization has too many items to render on hover (15).", "Arm 0_5

metric_1: -2.069% [-3.150%, -0.988%]
metric_2: -0.76% [-1.431%, -0.090%]
Parameterization has too many items to render on hover (15).", "Arm 0_50

metric_1: 0.367% [-0.736%, 1.470%]
metric_2: 2.231% [1.542%, 2.920%]
Parameterization has too many items to render on hover (15).", "Arm 0_51

metric_1: -4.597% [-5.631%, -3.562%]
metric_2: -1.371% [-2.056%, -0.687%]
Parameterization has too many items to render on hover (15).", "Arm 0_52

metric_1: -0.715% [-1.807%, 0.378%]
metric_2: 0.898% [0.220%, 1.576%]
Parameterization has too many items to render on hover (15).", "Arm 0_53

metric_1: -2.187% [-3.261%, -1.113%]
metric_2: -0.924% [-1.595%, -0.252%]
Parameterization has too many items to render on hover (15).", "Arm 0_54

metric_1: -3.62% [-4.663%, -2.577%]
metric_2: -1.266% [-1.928%, -0.603%]
Parameterization has too many items to render on hover (15).", "Arm 0_55

metric_1: -3.085% [-4.149%, -2.022%]
metric_2: -1.056% [-1.729%, -0.382%]
Parameterization has too many items to render on hover (15).", "Arm 0_56

metric_1: 0.065% [-1.043%, 1.174%]
metric_2: 1.104% [0.414%, 1.793%]
Parameterization has too many items to render on hover (15).", "Arm 0_57

metric_1: -8.045% [-9.053%, -7.036%]
metric_2: -3.231% [-3.897%, -2.565%]
Parameterization has too many items to render on hover (15).", "Arm 0_58

metric_1: -2.417% [-3.486%, -1.347%]
metric_2: -1.461% [-2.139%, -0.784%]
Parameterization has too many items to render on hover (15).", "Arm 0_59

metric_1: 2.155% [1.023%, 3.288%]
metric_2: 0.992% [0.310%, 1.673%]
Parameterization has too many items to render on hover (15).", "Arm 0_6

metric_1: -3.551% [-4.616%, -2.486%]
metric_2: -1.321% [-1.986%, -0.656%]
Parameterization has too many items to render on hover (15).", "Arm 0_60

metric_1: 3.0% [1.850%, 4.149%]
metric_2: 2.8% [2.112%, 3.488%]
Parameterization has too many items to render on hover (15).", "Arm 0_61

metric_1: 5.54% [4.346%, 6.735%]
metric_2: 10.809% [10.028%, 11.590%]
Parameterization has too many items to render on hover (15).", "Arm 0_62

metric_1: -13.137% [-14.095%, -12.178%]
metric_2: -3.839% [-4.509%, -3.169%]
Parameterization has too many items to render on hover (15).", "Arm 0_63

metric_1: -9.683% [-10.662%, -8.703%]
metric_2: -2.765% [-3.432%, -2.099%]
Parameterization has too many items to render on hover (15).", "Arm 0_7

metric_1: -4.089% [-5.138%, -3.041%]
metric_2: -1.576% [-2.248%, -0.903%]
Parameterization has too many items to render on hover (15).", "Arm 0_8

metric_1: -2.46% [-3.530%, -1.390%]
metric_2: -2.179% [-2.846%, -1.511%]
Parameterization has too many items to render on hover (15).", "Arm 0_9

metric_1: -7.98% [-8.985%, -6.976%]
metric_2: -2.79% [-3.451%, -2.128%]
Parameterization has too many items to render on hover (15).", "Arm status_quo

metric_1: -0.0% [-0.521%, 0.521%]
metric_2: -0.0% [-0.318%, 0.318%]
Parameterization has too many items to render on hover (15)." ], "type": "scatter", "visible": false, "x": [ -17.15914516790823, -11.573417823939199, -11.361067490605143, 3.037717284235046, -3.069542768219942, -2.864577054527247, -0.8171324453854826, -7.780377203320161, 2.475352747757105, -1.1963952539982303, -1.3598025263922875, -14.817429080033943, -3.2723491422755444, 0.08873184010870934, 0.5359430531455063, 1.8145980354381517, -6.471358153849177, 0.8877430970467538, 0.7286746192275023, 0.3510369131998831, -6.984895697391649, -7.505623786753492, 1.5767742612758522, -2.2143528351212733, -4.595608658658678, -19.18909654051259, 3.4292680540776312, 3.3710456099492796, -3.889937358175768, 0.6139708178717305, 0.5900679364090252, 0.8653940325415319, -1.2680473963845287, -0.1686822762476349, -2.893033151686222, -8.623721510464856, -5.156592129047242, -4.878241533976907, -3.655342272989576, 4.483189117418239, -3.0603072946347587, -6.921569399104486, -3.4781006474949763, -4.545929170059967, -0.305361571312272, -2.068706090724009, 0.36738131081906034, -4.596829363692114, -0.7145714514809897, -2.1870440120521244, -3.619802243175367, -3.0854247199357574, 0.06528116501171306, -8.04466137206524, -2.416515121000999, 2.1550821626518215, -3.550888972902983, 2.999501801038642, 5.540352816509654, -13.136826425368891, -9.682667832954975, -4.089456620979975, -2.4600469634137005, -7.980395623816304, -0.0 ], "y": [ -5.746731056831749, -3.560535696444575, -3.610268946914751, 2.5135502228088797, -0.5965513852397577, -1.0116808329390385, -0.8182560448021005, -2.4071834175723335, 2.5577489299929255, 0.348192434742279, -1.3114357729283859, -3.3382598457223502, -1.0068955594731426, -0.06867630083734555, 0.10424166292801242, -0.07210829452142886, -3.188401274811581, 1.960086317210278, 0.012674904654773453, 1.1203410785772772, 0.6155055435112974, -2.5504391581007666, 5.964668435375144, -0.4205314884998786, -2.726644960728147, -5.859418405279977, 3.1966830537300392, 3.950198735676712, 2.0568967995693455, 0.652872009905956, 1.286780707668062, 4.62085929335188, -0.7622630447726769, 0.123503526062618, -1.5431517157186252, -3.2650236570623457, -1.7414179188197445, -1.1257173899572386, -1.3809544654506094, 7.651708533863019, -2.2015816842718356, -3.063188080332386, 4.012625447421733, -1.168849751600452, 0.7526902410881314, -0.7604599707221315, 2.2308412374353317, -1.371441760784444, 0.8978911718791985, -0.9236769844543923, -1.2656161965636414, -1.0555488550285586, 1.1039153988973123, -3.2310973947137103, -1.4614566797717248, 0.9918050829129264, -1.3210922238627325, 2.8001328033355772, 10.809115959705165, -3.839017981482167, -2.765325873487098, -1.5757183832623392, -2.1787372563938, -2.789593215553577, -0.0 ] }, { "error_x": { "array": [ 0.9032151733520527, 0.9624260346524522, 0.9593909284698995, 1.1347609507759298, 1.0408257278018285, 1.0435860695228432, 1.0711864083714089, 1.0156140087849477, 1.128257833835402, 1.0788991832923005, 1.0706037001289912, 0.9284671425890606, 1.0523829338844282, 1.0903373892267718, 1.0861328996665252, 1.1145865581026473, 1.0063514862723812, 1.0994776629118859, 1.091675914931962, 1.0844593315915725, 1.0017853270032084, 0.9935701990690048, 1.1127508961856405, 1.0668110378139575, 1.0328601627012277, 0.8797268687204242, 1.140518932271723, 1.1417536403033617, 1.0764096893237975, 1.0965566556283748, 1.0984998396729218, 1.102540333829891, 1.0794770222090497, 1.085962611129493, 1.0433251902342626, 0.9820278721895025, 1.0166480934308502, 1.0343546221526094, 1.0382944853121021, 1.1551456071793769, 1.0464931893207308, 1.0023861997986674, 1.0516802404518037, 1.0346896327290815, 1.08327143184596, 1.0698309404523143, 1.0904297395971863, 1.0222860350150513, 1.0810704079796398, 1.0613893596292459, 1.0320103439592427, 1.0528002249874768, 1.0912064123686547, 0.9988930110467326, 1.0571937806819418, 1.1193739547640122, 1.0530800272030827, 1.1303999126786766, 1.180499014903316, 0.9513079288502773, 0.9719033134076178, 1.03791308728753, 1.0588852902469934, 0.9945314913677392, 0.5199803498226312 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "error_y": { "array": [ 0.6547387343317975, 0.6532033115837151, 0.6609466315731528, 0.6813554440847576, 0.6694705491295897, 0.6591303137747925, 0.6563086390178189, 0.6508039407989954, 0.6839216136881153, 0.6604406759568969, 0.6601787231575951, 0.6565366929811817, 0.6613093240694986, 0.6561140874958142, 0.6633706122047732, 0.6637799634962176, 0.6555405967295402, 0.6704566164608853, 0.66273573692496, 0.6685871803108022, 0.6628627361853555, 0.6572002148831203, 0.712446900341419, 0.6636937933944925, 0.6586242603445244, 0.6475500126287197, 0.6773760670047011, 0.6859237814996093, 0.6688072097037889, 0.6516465341587597, 0.6738024004862555, 0.7036162417194723, 0.6668939573219893, 0.6610547090506331, 0.6553518681013905, 0.6508051033775936, 0.64797757405811, 0.6713714106339492, 0.656723211631772, 0.7346090086077302, 0.6642524778308558, 0.6574536033119194, 0.7031723757695748, 0.6494774398002832, 0.6704885519899918, 0.6622056608350378, 0.6797639514603396, 0.6735895146091605, 0.6692892385552365, 0.6620882696925839, 0.6531942830594355, 0.6650013545172943, 0.6769982685390781, 0.6574731612170758, 0.6674033123702455, 0.6722373506713744, 0.6559893752039715, 0.6748097809173098, 0.7697444470145935, 0.6626605822603224, 0.6581382444319371, 0.663286850538724, 0.658955602798785, 0.6526018616070007, 0.3176861573292605 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "hoverinfo": "text", "marker": { "color": "rgba(128,177,211,1)" }, "mode": "markers", "name": "In-sample", "showlegend": true, "text": [ "Arm 0_0

metric_1: -17.022% [-17.925%, -16.119%]
metric_2: -5.683% [-6.338%, -5.028%]
Parameterization has too many items to render on hover (15).", "Arm 0_1

metric_1: -11.473% [-12.436%, -10.511%]
metric_2: -3.526% [-4.180%, -2.873%]
Parameterization has too many items to render on hover (15).", "Arm 0_10

metric_1: -11.26% [-12.220%, -10.301%]
metric_2: -3.535% [-4.196%, -2.874%]
Parameterization has too many items to render on hover (15).", "Arm 0_11

metric_1: 2.99% [1.855%, 4.125%]
metric_2: 2.542% [1.861%, 3.224%]
Parameterization has too many items to render on hover (15).", "Arm 0_12

metric_1: -3.205% [-4.246%, -2.164%]
metric_2: -0.629% [-1.298%, 0.041%]
Parameterization has too many items to render on hover (15).", "Arm 0_13

metric_1: -2.828% [-3.871%, -1.784%]
metric_2: -1.055% [-1.714%, -0.396%]
Parameterization has too many items to render on hover (15).", "Arm 0_14

metric_1: -0.807% [-1.878%, 0.264%]
metric_2: -0.832% [-1.488%, -0.176%]
Parameterization has too many items to render on hover (15).", "Arm 0_15

metric_1: -7.632% [-8.648%, -6.617%]
metric_2: -2.319% [-2.970%, -1.669%]
Parameterization has too many items to render on hover (15).", "Arm 0_16

metric_1: 2.493% [1.365%, 3.621%]
metric_2: 2.582% [1.898%, 3.266%]
Parameterization has too many items to render on hover (15).", "Arm 0_17

metric_1: -1.148% [-2.227%, -0.069%]
metric_2: 0.326% [-0.334%, 0.987%]
Parameterization has too many items to render on hover (15).", "Arm 0_18

metric_1: -1.417% [-2.488%, -0.346%]
metric_2: -1.321% [-1.981%, -0.661%]
Parameterization has too many items to render on hover (15).", "Arm 0_19

metric_1: -14.735% [-15.664%, -13.807%]
metric_2: -3.315% [-3.971%, -2.658%]
Parameterization has too many items to render on hover (15).", "Arm 0_2

metric_1: -3.211% [-4.263%, -2.158%]
metric_2: -0.992% [-1.653%, -0.331%]
Parameterization has too many items to render on hover (15).", "Arm 0_20

metric_1: 0.073% [-1.017%, 1.164%]
metric_2: -0.063% [-0.719%, 0.594%]
Parameterization has too many items to render on hover (15).", "Arm 0_21

metric_1: 0.528% [-0.558%, 1.614%]
metric_2: 0.066% [-0.597%, 0.730%]
Parameterization has too many items to render on hover (15).", "Arm 0_22

metric_1: 1.746% [0.632%, 2.861%]
metric_2: -0.055% [-0.719%, 0.608%]
Parameterization has too many items to render on hover (15).", "Arm 0_23

metric_1: -6.404% [-7.411%, -5.398%]
metric_2: -3.15% [-3.805%, -2.494%]
Parameterization has too many items to render on hover (15).", "Arm 0_24

metric_1: 0.836% [-0.264%, 1.935%]
metric_2: 1.893% [1.223%, 2.564%]
Parameterization has too many items to render on hover (15).", "Arm 0_25

metric_1: 0.699% [-0.393%, 1.790%]
metric_2: 0.024% [-0.639%, 0.686%]
Parameterization has too many items to render on hover (15).", "Arm 0_26

metric_1: 0.319% [-0.766%, 1.403%]
metric_2: 1.077% [0.409%, 1.746%]
Parameterization has too many items to render on hover (15).", "Arm 0_27

metric_1: -6.918% [-7.920%, -5.916%]
metric_2: 0.651% [-0.012%, 1.314%]
Parameterization has too many items to render on hover (15).", "Arm 0_28

metric_1: -7.463% [-8.457%, -6.469%]
metric_2: -2.49% [-3.147%, -1.832%]
Parameterization has too many items to render on hover (15).", "Arm 0_29

metric_1: 1.48% [0.367%, 2.592%]
metric_2: 5.858% [5.145%, 6.570%]
Parameterization has too many items to render on hover (15).", "Arm 0_3

metric_1: -2.25% [-3.317%, -1.183%]
metric_2: -0.448% [-1.112%, 0.215%]
Parameterization has too many items to render on hover (15).", "Arm 0_30

metric_1: -4.65% [-5.683%, -3.617%]
metric_2: -2.712% [-3.371%, -2.054%]
Parameterization has too many items to render on hover (15).", "Arm 0_31

metric_1: -19.06% [-19.939%, -18.180%]
metric_2: -5.792% [-6.440%, -5.145%]
Parameterization has too many items to render on hover (15).", "Arm 0_32

metric_1: 3.344% [2.204%, 4.485%]
metric_2: 3.158% [2.481%, 3.835%]
Parameterization has too many items to render on hover (15).", "Arm 0_33

metric_1: 3.298% [2.156%, 4.440%]
metric_2: 3.853% [3.167%, 4.539%]
Parameterization has too many items to render on hover (15).", "Arm 0_34

metric_1: -3.845% [-4.922%, -2.769%]
metric_2: 2.026% [1.357%, 2.695%]
Parameterization has too many items to render on hover (15).", "Arm 0_35

metric_1: 0.616% [-0.480%, 1.713%]
metric_2: 0.619% [-0.033%, 1.270%]
Parameterization has too many items to render on hover (15).", "Arm 0_36

metric_1: 0.606% [-0.493%, 1.704%]
metric_2: 1.328% [0.654%, 2.002%]
Parameterization has too many items to render on hover (15).", "Arm 0_37

metric_1: 0.801% [-0.301%, 1.904%]
metric_2: 4.539% [3.836%, 5.243%]
Parameterization has too many items to render on hover (15).", "Arm 0_38

metric_1: -1.23% [-2.310%, -0.151%]
metric_2: -0.74% [-1.407%, -0.074%]
Parameterization has too many items to render on hover (15).", "Arm 0_39

metric_1: -0.189% [-1.275%, 0.897%]
metric_2: 0.11% [-0.551%, 0.771%]
Parameterization has too many items to render on hover (15).", "Arm 0_4

metric_1: -2.877% [-3.920%, -1.834%]
metric_2: -1.538% [-2.193%, -0.883%]
Parameterization has too many items to render on hover (15).", "Arm 0_40

metric_1: -8.667% [-9.649%, -7.685%]
metric_2: -3.287% [-3.937%, -2.636%]
Parameterization has too many items to render on hover (15).", "Arm 0_41

metric_1: -5.112% [-6.129%, -4.096%]
metric_2: -1.776% [-2.424%, -1.128%]
Parameterization has too many items to render on hover (15).", "Arm 0_42

metric_1: -4.866% [-5.900%, -3.831%]
metric_2: -1.107% [-1.779%, -0.436%]
Parameterization has too many items to render on hover (15).", "Arm 0_43

metric_1: -3.621% [-4.659%, -2.582%]
metric_2: -1.374% [-2.031%, -0.717%]
Parameterization has too many items to render on hover (15).", "Arm 0_44

metric_1: 4.416% [3.261%, 5.571%]
metric_2: 7.531% [6.797%, 8.266%]
Parameterization has too many items to render on hover (15).", "Arm 0_45

metric_1: -3.094% [-4.140%, -2.047%]
metric_2: -2.19% [-2.854%, -1.526%]
Parameterization has too many items to render on hover (15).", "Arm 0_46

metric_1: -6.987% [-7.990%, -5.985%]
metric_2: -3.082% [-3.740%, -2.425%]
Parameterization has too many items to render on hover (15).", "Arm 0_47

metric_1: -3.456% [-4.507%, -2.404%]
metric_2: 3.964% [3.261%, 4.667%]
Parameterization has too many items to render on hover (15).", "Arm 0_48

metric_1: -4.487% [-5.521%, -3.452%]
metric_2: -1.178% [-1.827%, -0.528%]
Parameterization has too many items to render on hover (15).", "Arm 0_49

metric_1: -0.276% [-1.360%, 0.807%]
metric_2: 0.752% [0.081%, 1.422%]
Parameterization has too many items to render on hover (15).", "Arm 0_5

metric_1: -2.093% [-3.163%, -1.023%]
metric_2: -0.772% [-1.435%, -0.110%]
Parameterization has too many items to render on hover (15).", "Arm 0_50

metric_1: 0.34% [-0.750%, 1.431%]
metric_2: 2.197% [1.517%, 2.877%]
Parameterization has too many items to render on hover (15).", "Arm 0_51

metric_1: -4.611% [-5.633%, -3.589%]
metric_2: -1.337% [-2.011%, -0.664%]
Parameterization has too many items to render on hover (15).", "Arm 0_52

metric_1: -0.757% [-1.838%, 0.324%]
metric_2: 0.878% [0.208%, 1.547%]
Parameterization has too many items to render on hover (15).", "Arm 0_53

metric_1: -2.166% [-3.228%, -1.105%]
metric_2: -0.911% [-1.573%, -0.249%]
Parameterization has too many items to render on hover (15).", "Arm 0_54

metric_1: -3.611% [-4.643%, -2.579%]
metric_2: -1.267% [-1.920%, -0.613%]
Parameterization has too many items to render on hover (15).", "Arm 0_55

metric_1: -3.053% [-4.106%, -2.000%]
metric_2: -1.055% [-1.720%, -0.390%]
Parameterization has too many items to render on hover (15).", "Arm 0_56

metric_1: 0.115% [-0.976%, 1.206%]
metric_2: 1.138% [0.461%, 1.815%]
Parameterization has too many items to render on hover (15).", "Arm 0_57

metric_1: -8.012% [-9.011%, -7.013%]
metric_2: -3.221% [-3.878%, -2.563%]
Parameterization has too many items to render on hover (15).", "Arm 0_58

metric_1: -2.444% [-3.501%, -1.387%]
metric_2: -1.454% [-2.122%, -0.787%]
Parameterization has too many items to render on hover (15).", "Arm 0_59

metric_1: 2.105% [0.986%, 3.225%]
metric_2: 0.993% [0.320%, 1.665%]
Parameterization has too many items to render on hover (15).", "Arm 0_6

metric_1: -3.549% [-4.602%, -2.496%]
metric_2: -1.318% [-1.974%, -0.662%]
Parameterization has too many items to render on hover (15).", "Arm 0_60

metric_1: 2.966% [1.836%, 4.097%]
metric_2: 2.781% [2.106%, 3.455%]
Parameterization has too many items to render on hover (15).", "Arm 0_61

metric_1: 5.393% [4.212%, 6.573%]
metric_2: 10.553% [9.783%, 11.322%]
Parameterization has too many items to render on hover (15).", "Arm 0_62

metric_1: -13.026% [-13.978%, -12.075%]
metric_2: -3.787% [-4.450%, -3.125%]
Parameterization has too many items to render on hover (15).", "Arm 0_63

metric_1: -9.619% [-10.591%, -8.647%]
metric_2: -2.738% [-3.397%, -2.080%]
Parameterization has too many items to render on hover (15).", "Arm 0_7

metric_1: -4.118% [-5.156%, -3.080%]
metric_2: -1.61% [-2.273%, -0.947%]
Parameterization has too many items to render on hover (15).", "Arm 0_8

metric_1: -2.464% [-3.523%, -1.405%]
metric_2: -2.148% [-2.807%, -1.489%]
Parameterization has too many items to render on hover (15).", "Arm 0_9

metric_1: -7.957% [-8.951%, -6.962%]
metric_2: -2.794% [-3.447%, -2.142%]
Parameterization has too many items to render on hover (15).", "Arm status_quo

metric_1: -0.0% [-0.520%, 0.520%]
metric_2: -0.0% [-0.318%, 0.318%]
Parameterization has too many items to render on hover (15)." ], "type": "scatter", "visible": true, "x": [ -17.022045315104542, -11.473276363575263, -11.260309485449103, 2.9899364287480865, -3.2049806778237757, -2.827647808086468, -0.8072357684589017, -7.632201738791567, 2.4931910046363135, -1.1483177933819395, -1.4169105153422092, -14.735353419604191, -3.2108750242520334, 0.07332105300986574, 0.52794229128138, 1.7461298706241524, -6.404301715961119, 0.8357526716703064, 0.6988214644260046, 0.3185581570295636, -6.918118515914437, -7.462960459390675, 1.4797223030792654, -2.250057847895754, -4.649788400829341, -19.059723742205165, 3.3442377220374424, 3.2977920613814318, -3.845344844296581, 0.6160918095707203, 0.605873558955114, 0.8012482568804604, -1.2301194692314896, -0.18910424701708023, -2.8771572632426596, -8.666686434035148, -5.1123185935816275, -4.86569050603379, -3.620616821502621, 4.416040423353074, -3.093977759193243, -6.987479150965008, -3.45561072825807, -4.486522144334903, -0.27632651192689084, -2.092939160334957, 0.34040916740729205, -4.6111158309365, -0.7572230824447213, -2.1663524739549853, -3.611088388568022, -3.053027739941141, 0.11513485293184113, -8.011705772759294, -2.4442287384702572, 2.1052356784416553, -3.5492459592908614, 2.966172129108271, 5.392515165139946, -13.026404058841306, -9.619176461732291, -4.118044655666446, -2.4636655602657878, -7.956569241984238, -0.0 ], "y": [ -5.682890950120342, -3.5263840662392654, -3.5354295227440935, 2.542378866552539, -0.6285418092402999, -1.0553405972573464, -0.8320014955517723, -2.3193462375707914, 2.58163977526919, 0.32621268542555526, -1.3207797795363851, -3.314816695063182, -0.9921710458941605, -0.06250694414329654, 0.06640049636248754, -0.05540222390315327, -3.1495291448321576, 1.8934882315080965, 0.023529017577203423, 1.0773772731169438, 0.6512642256730126, -2.489527283561809, 5.857858819488534, -0.4484318638610448, -2.712182537468451, -5.792314875460171, 3.1580957792386144, 3.8533176800717563, 2.026262991103158, 0.6187928699607773, 1.3280865831055046, 4.5394243735749775, -0.7404858515521753, 0.10968492064137063, -1.5381048845570704, -3.2865434005114005, -1.7761680789956928, -1.1071508425287317, -1.3739381694523336, 7.5312245476719, -2.1901762525545276, -3.082214357601481, 3.963995297169013, -1.1779086966890597, 0.7516740725998268, -0.772463613395467, 2.196834434560228, -1.3372360602549658, 0.8775653888918572, -0.9111761852726025, -1.2666733248842346, -1.055168296030082, 1.1378418002527275, -3.2209479427630807, -1.4541461401402338, 0.9926717601692114, -1.3177840022000746, 2.7806338967179443, 10.552746740900881, -3.7873810524577847, -2.7384540934203034, -1.6100825481553491, -2.147955641504198, -2.7941662700346765, -0.0 ] }, { "error_x": { "array": [ 4.660267831501327, 4.887460782453055, 5.635425857830732 ], "color": "rgba(251,128,114,0.4)", "type": "data" }, "error_y": { "array": [ 2.5193699490248327, 2.8125842237332273, 3.1483376877014773 ], "color": "rgba(251,128,114,0.4)", "type": "data" }, "hoverinfo": "text", "marker": { "color": "rgba(251,128,114,1)" }, "mode": "markers", "name": "constraint_1", "showlegend": true, "text": [ "Arm b7d5

metric_1: 2.692% [-1.968%, 7.353%]
metric_2: 0.431% [-2.089%, 2.950%]
Parameterization has too many items to render on hover (15).", "Arm e3b1

metric_1: 2.395% [-2.493%, 7.282%]
metric_2: 0.867% [-1.946%, 3.679%]
Parameterization has too many items to render on hover (15).", "Arm 140e

metric_1: 1.953% [-3.682%, 7.589%]
metric_2: 0.672% [-2.476%, 3.820%]
Parameterization has too many items to render on hover (15)." ], "type": "scatter", "visible": true, "x": [ 2.692240778165111, 2.3947768825258926, 1.9532508957716572 ], "y": [ 0.4308677854504356, 0.866514006381011, 0.6719251619346646 ] } ], "layout": { "annotations": [ { "showarrow": false, "text": "Show CI", "x": 1.18, "xref": "paper", "y": 0.7, "yanchor": "middle", "yref": "paper" }, { "showarrow": false, "text": "Type", "x": 1.18, "xref": "paper", "y": 0.6, "yanchor": "middle", "yref": "paper" } ], "font": { "size": 10 }, "height": 600, "hovermode": "closest", "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "sequentialminus": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "title": { "text": "Objective Tradeoffs" }, "updatemenus": [ { "buttons": [ { "args": [ { "error_x.thickness": 2, "error_x.width": 4, "error_y.thickness": 2, "error_y.width": 4 } ], "label": "Yes", "method": "restyle" }, { "args": [ { "error_x.thickness": 0, "error_x.width": 0, "error_y.thickness": 0, "error_y.width": 0 } ], "label": "No", "method": "restyle" } ], "x": 1.25, "xanchor": "left", "y": 0.67, "yanchor": "middle" }, { "buttons": [ { "args": [ { "visible": [ false, true, true ] } ], "label": "Modeled", "method": "restyle" }, { "args": [ { "visible": [ true, false, false ] } ], "label": "Observed", "method": "restyle" } ], "x": 1.25, "xanchor": "left", "y": 0.57, "yanchor": "middle" } ], "width": 800, "xaxis": { "title": { "text": "metric_1 (%)" }, "zeroline": true, "zerolinecolor": "red" }, "yaxis": { "title": { "text": "metric_2 (%)" }, "zeroline": true, "zerolinecolor": "red" } } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "render(plot_multiple_metrics(\n", " gp,\n", " metric_x=METRIC_X_AXIS,\n", " metric_y=METRIC_Y_AXIS,\n", " generator_runs_dict={\n", " \"constraint_1\": constraint_1_results,\n", " }\n", "))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, let's view all three sets of candidates together. " ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "linkText": "Export to plot.ly", "plotlyServerURL": "https://plot.ly", "showLink": false }, "data": [ { "error_x": { "array": [ 0.9105274690527578, 0.9725977092158347, 0.9694448615326413, 1.1526497268980418, 1.056129089525654, 1.0581150790455403, 1.0853729636494531, 1.025297982327309, 1.1509354346821015, 1.0946224534113276, 1.080994735603789, 0.9349800939958878, 1.0669782977880082, 1.1063382644746038, 1.109747945764228, 1.1312428838918263, 1.0151345627364643, 1.113745664637434, 1.1070645021428274, 1.098298041303658, 1.011825279703854, 1.004076551317031, 1.1247628081965457, 1.0803499998271806, 1.0451840940805415, 0.8854990204077908, 1.1551313262975833, 1.1582540625351196, 1.088250532740983, 1.1152606223383992, 1.1132227750465398, 1.1136443399827738, 1.0928257948335294, 1.098342295748459, 1.0562137210185467, 0.9929499124720941, 1.0275183054522712, 1.0461406507525532, 1.0507233488011212, 1.1693056777789836, 1.0564300559878494, 1.012746475490335, 1.060503126319525, 1.0453978622119475, 1.0972242260284542, 1.0810868080421863, 1.103084437746675, 1.0346109289273437, 1.0925727929561801, 1.0737167310801996, 1.042945064542918, 1.0636626539145406, 1.108765950358755, 1.0083792696638647, 1.069964804645196, 1.1324679783490406, 1.0646727177942992, 1.149093505549881, 1.19450723179394, 0.9584867213686696, 0.9797729070203455, 1.048287897726107, 1.0699714924908204, 1.0044799847095478, 0.5205646439608032 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "error_y": { "array": [ 0.6639175585103954, 0.6630929411298598, 0.6719874423233675, 0.6935680367786019, 0.6821107678411614, 0.6721157287254815, 0.6675598394585217, 0.6588395471546759, 0.7005079335820925, 0.6734693768172479, 0.6678033173208016, 0.6639242874696609, 0.6729502923953926, 0.6673739566339196, 0.6808476824923786, 0.675618453499239, 0.6630338514495195, 0.6814341339540937, 0.6732139305974362, 0.678993016475866, 0.6717812932415321, 0.6668482051137559, 0.7215454438634975, 0.6740882429740953, 0.6694676526234568, 0.6554367403862607, 0.6871145937679641, 0.6973627667199596, 0.677839082032618, 0.6656099293584739, 0.6853610253860684, 0.7127045342173167, 0.6774929784721044, 0.6697791158632792, 0.665276789680729, 0.6609617898538276, 0.6573670880847166, 0.6812573532252995, 0.6669304645971238, 0.7456805630690589, 0.6722619172607951, 0.6674666794827154, 0.71101954162863, 0.6581471114518421, 0.681784654518447, 0.6708890082180987, 0.6891795059009299, 0.6846723076532707, 0.6778079742730775, 0.6716150419025536, 0.6624185510214552, 0.6736751066980704, 0.6894763100114708, 0.6659339720000773, 0.6774968737338238, 0.6814011092512952, 0.6647910226422914, 0.6882834695714566, 0.7808801986391988, 0.6703813570659151, 0.666212972377905, 0.6724199160716084, 0.6676592507580481, 0.6614165175994707, 0.3181115760489757 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "hoverinfo": "text", "marker": { "color": "rgba(128,177,211,1)" }, "mode": "markers", "name": "In-sample", "showlegend": true, "text": [ "Arm 0_0

metric_1: -17.159% [-18.070%, -16.249%]
metric_2: -5.747% [-6.411%, -5.083%]
Parameterization has too many items to render on hover (15).", "Arm 0_1

metric_1: -11.573% [-12.546%, -10.601%]
metric_2: -3.561% [-4.224%, -2.897%]
Parameterization has too many items to render on hover (15).", "Arm 0_10

metric_1: -11.361% [-12.331%, -10.392%]
metric_2: -3.61% [-4.282%, -2.938%]
Parameterization has too many items to render on hover (15).", "Arm 0_11

metric_1: 3.038% [1.885%, 4.190%]
metric_2: 2.514% [1.820%, 3.207%]
Parameterization has too many items to render on hover (15).", "Arm 0_12

metric_1: -3.07% [-4.126%, -2.013%]
metric_2: -0.597% [-1.279%, 0.086%]
Parameterization has too many items to render on hover (15).", "Arm 0_13

metric_1: -2.865% [-3.923%, -1.806%]
metric_2: -1.012% [-1.684%, -0.340%]
Parameterization has too many items to render on hover (15).", "Arm 0_14

metric_1: -0.817% [-1.903%, 0.268%]
metric_2: -0.818% [-1.486%, -0.151%]
Parameterization has too many items to render on hover (15).", "Arm 0_15

metric_1: -7.78% [-8.806%, -6.755%]
metric_2: -2.407% [-3.066%, -1.748%]
Parameterization has too many items to render on hover (15).", "Arm 0_16

metric_1: 2.475% [1.324%, 3.626%]
metric_2: 2.558% [1.857%, 3.258%]
Parameterization has too many items to render on hover (15).", "Arm 0_17

metric_1: -1.196% [-2.291%, -0.102%]
metric_2: 0.348% [-0.325%, 1.022%]
Parameterization has too many items to render on hover (15).", "Arm 0_18

metric_1: -1.36% [-2.441%, -0.279%]
metric_2: -1.311% [-1.979%, -0.644%]
Parameterization has too many items to render on hover (15).", "Arm 0_19

metric_1: -14.817% [-15.752%, -13.882%]
metric_2: -3.338% [-4.002%, -2.674%]
Parameterization has too many items to render on hover (15).", "Arm 0_2

metric_1: -3.272% [-4.339%, -2.205%]
metric_2: -1.007% [-1.680%, -0.334%]
Parameterization has too many items to render on hover (15).", "Arm 0_20

metric_1: 0.089% [-1.018%, 1.195%]
metric_2: -0.069% [-0.736%, 0.599%]
Parameterization has too many items to render on hover (15).", "Arm 0_21

metric_1: 0.536% [-0.574%, 1.646%]
metric_2: 0.104% [-0.577%, 0.785%]
Parameterization has too many items to render on hover (15).", "Arm 0_22

metric_1: 1.815% [0.683%, 2.946%]
metric_2: -0.072% [-0.748%, 0.604%]
Parameterization has too many items to render on hover (15).", "Arm 0_23

metric_1: -6.471% [-7.486%, -5.456%]
metric_2: -3.188% [-3.851%, -2.525%]
Parameterization has too many items to render on hover (15).", "Arm 0_24

metric_1: 0.888% [-0.226%, 2.001%]
metric_2: 1.96% [1.279%, 2.642%]
Parameterization has too many items to render on hover (15).", "Arm 0_25

metric_1: 0.729% [-0.378%, 1.836%]
metric_2: 0.013% [-0.661%, 0.686%]
Parameterization has too many items to render on hover (15).", "Arm 0_26

metric_1: 0.351% [-0.747%, 1.449%]
metric_2: 1.12% [0.441%, 1.799%]
Parameterization has too many items to render on hover (15).", "Arm 0_27

metric_1: -6.985% [-7.997%, -5.973%]
metric_2: 0.616% [-0.056%, 1.287%]
Parameterization has too many items to render on hover (15).", "Arm 0_28

metric_1: -7.506% [-8.510%, -6.502%]
metric_2: -2.55% [-3.217%, -1.884%]
Parameterization has too many items to render on hover (15).", "Arm 0_29

metric_1: 1.577% [0.452%, 2.702%]
metric_2: 5.965% [5.243%, 6.686%]
Parameterization has too many items to render on hover (15).", "Arm 0_3

metric_1: -2.214% [-3.295%, -1.134%]
metric_2: -0.421% [-1.095%, 0.254%]
Parameterization has too many items to render on hover (15).", "Arm 0_30

metric_1: -4.596% [-5.641%, -3.550%]
metric_2: -2.727% [-3.396%, -2.057%]
Parameterization has too many items to render on hover (15).", "Arm 0_31

metric_1: -19.189% [-20.075%, -18.304%]
metric_2: -5.859% [-6.515%, -5.204%]
Parameterization has too many items to render on hover (15).", "Arm 0_32

metric_1: 3.429% [2.274%, 4.584%]
metric_2: 3.197% [2.510%, 3.884%]
Parameterization has too many items to render on hover (15).", "Arm 0_33

metric_1: 3.371% [2.213%, 4.529%]
metric_2: 3.95% [3.253%, 4.648%]
Parameterization has too many items to render on hover (15).", "Arm 0_34

metric_1: -3.89% [-4.978%, -2.802%]
metric_2: 2.057% [1.379%, 2.735%]
Parameterization has too many items to render on hover (15).", "Arm 0_35

metric_1: 0.614% [-0.501%, 1.729%]
metric_2: 0.653% [-0.013%, 1.318%]
Parameterization has too many items to render on hover (15).", "Arm 0_36

metric_1: 0.59% [-0.523%, 1.703%]
metric_2: 1.287% [0.601%, 1.972%]
Parameterization has too many items to render on hover (15).", "Arm 0_37

metric_1: 0.865% [-0.248%, 1.979%]
metric_2: 4.621% [3.908%, 5.334%]
Parameterization has too many items to render on hover (15).", "Arm 0_38

metric_1: -1.268% [-2.361%, -0.175%]
metric_2: -0.762% [-1.440%, -0.085%]
Parameterization has too many items to render on hover (15).", "Arm 0_39

metric_1: -0.169% [-1.267%, 0.930%]
metric_2: 0.124% [-0.546%, 0.793%]
Parameterization has too many items to render on hover (15).", "Arm 0_4

metric_1: -2.893% [-3.949%, -1.837%]
metric_2: -1.543% [-2.208%, -0.878%]
Parameterization has too many items to render on hover (15).", "Arm 0_40

metric_1: -8.624% [-9.617%, -7.631%]
metric_2: -3.265% [-3.926%, -2.604%]
Parameterization has too many items to render on hover (15).", "Arm 0_41

metric_1: -5.157% [-6.184%, -4.129%]
metric_2: -1.741% [-2.399%, -1.084%]
Parameterization has too many items to render on hover (15).", "Arm 0_42

metric_1: -4.878% [-5.924%, -3.832%]
metric_2: -1.126% [-1.807%, -0.444%]
Parameterization has too many items to render on hover (15).", "Arm 0_43

metric_1: -3.655% [-4.706%, -2.605%]
metric_2: -1.381% [-2.048%, -0.714%]
Parameterization has too many items to render on hover (15).", "Arm 0_44

metric_1: 4.483% [3.314%, 5.652%]
metric_2: 7.652% [6.906%, 8.397%]
Parameterization has too many items to render on hover (15).", "Arm 0_45

metric_1: -3.06% [-4.117%, -2.004%]
metric_2: -2.202% [-2.874%, -1.529%]
Parameterization has too many items to render on hover (15).", "Arm 0_46

metric_1: -6.922% [-7.934%, -5.909%]
metric_2: -3.063% [-3.731%, -2.396%]
Parameterization has too many items to render on hover (15).", "Arm 0_47

metric_1: -3.478% [-4.539%, -2.418%]
metric_2: 4.013% [3.302%, 4.724%]
Parameterization has too many items to render on hover (15).", "Arm 0_48

metric_1: -4.546% [-5.591%, -3.501%]
metric_2: -1.169% [-1.827%, -0.511%]
Parameterization has too many items to render on hover (15).", "Arm 0_49

metric_1: -0.305% [-1.403%, 0.792%]
metric_2: 0.753% [0.071%, 1.434%]
Parameterization has too many items to render on hover (15).", "Arm 0_5

metric_1: -2.069% [-3.150%, -0.988%]
metric_2: -0.76% [-1.431%, -0.090%]
Parameterization has too many items to render on hover (15).", "Arm 0_50

metric_1: 0.367% [-0.736%, 1.470%]
metric_2: 2.231% [1.542%, 2.920%]
Parameterization has too many items to render on hover (15).", "Arm 0_51

metric_1: -4.597% [-5.631%, -3.562%]
metric_2: -1.371% [-2.056%, -0.687%]
Parameterization has too many items to render on hover (15).", "Arm 0_52

metric_1: -0.715% [-1.807%, 0.378%]
metric_2: 0.898% [0.220%, 1.576%]
Parameterization has too many items to render on hover (15).", "Arm 0_53

metric_1: -2.187% [-3.261%, -1.113%]
metric_2: -0.924% [-1.595%, -0.252%]
Parameterization has too many items to render on hover (15).", "Arm 0_54

metric_1: -3.62% [-4.663%, -2.577%]
metric_2: -1.266% [-1.928%, -0.603%]
Parameterization has too many items to render on hover (15).", "Arm 0_55

metric_1: -3.085% [-4.149%, -2.022%]
metric_2: -1.056% [-1.729%, -0.382%]
Parameterization has too many items to render on hover (15).", "Arm 0_56

metric_1: 0.065% [-1.043%, 1.174%]
metric_2: 1.104% [0.414%, 1.793%]
Parameterization has too many items to render on hover (15).", "Arm 0_57

metric_1: -8.045% [-9.053%, -7.036%]
metric_2: -3.231% [-3.897%, -2.565%]
Parameterization has too many items to render on hover (15).", "Arm 0_58

metric_1: -2.417% [-3.486%, -1.347%]
metric_2: -1.461% [-2.139%, -0.784%]
Parameterization has too many items to render on hover (15).", "Arm 0_59

metric_1: 2.155% [1.023%, 3.288%]
metric_2: 0.992% [0.310%, 1.673%]
Parameterization has too many items to render on hover (15).", "Arm 0_6

metric_1: -3.551% [-4.616%, -2.486%]
metric_2: -1.321% [-1.986%, -0.656%]
Parameterization has too many items to render on hover (15).", "Arm 0_60

metric_1: 3.0% [1.850%, 4.149%]
metric_2: 2.8% [2.112%, 3.488%]
Parameterization has too many items to render on hover (15).", "Arm 0_61

metric_1: 5.54% [4.346%, 6.735%]
metric_2: 10.809% [10.028%, 11.590%]
Parameterization has too many items to render on hover (15).", "Arm 0_62

metric_1: -13.137% [-14.095%, -12.178%]
metric_2: -3.839% [-4.509%, -3.169%]
Parameterization has too many items to render on hover (15).", "Arm 0_63

metric_1: -9.683% [-10.662%, -8.703%]
metric_2: -2.765% [-3.432%, -2.099%]
Parameterization has too many items to render on hover (15).", "Arm 0_7

metric_1: -4.089% [-5.138%, -3.041%]
metric_2: -1.576% [-2.248%, -0.903%]
Parameterization has too many items to render on hover (15).", "Arm 0_8

metric_1: -2.46% [-3.530%, -1.390%]
metric_2: -2.179% [-2.846%, -1.511%]
Parameterization has too many items to render on hover (15).", "Arm 0_9

metric_1: -7.98% [-8.985%, -6.976%]
metric_2: -2.79% [-3.451%, -2.128%]
Parameterization has too many items to render on hover (15).", "Arm status_quo

metric_1: -0.0% [-0.521%, 0.521%]
metric_2: -0.0% [-0.318%, 0.318%]
Parameterization has too many items to render on hover (15)." ], "type": "scatter", "visible": false, "x": [ -17.15914516790823, -11.573417823939199, -11.361067490605143, 3.037717284235046, -3.069542768219942, -2.864577054527247, -0.8171324453854826, -7.780377203320161, 2.475352747757105, -1.1963952539982303, -1.3598025263922875, -14.817429080033943, -3.2723491422755444, 0.08873184010870934, 0.5359430531455063, 1.8145980354381517, -6.471358153849177, 0.8877430970467538, 0.7286746192275023, 0.3510369131998831, -6.984895697391649, -7.505623786753492, 1.5767742612758522, -2.2143528351212733, -4.595608658658678, -19.18909654051259, 3.4292680540776312, 3.3710456099492796, -3.889937358175768, 0.6139708178717305, 0.5900679364090252, 0.8653940325415319, -1.2680473963845287, -0.1686822762476349, -2.893033151686222, -8.623721510464856, -5.156592129047242, -4.878241533976907, -3.655342272989576, 4.483189117418239, -3.0603072946347587, -6.921569399104486, -3.4781006474949763, -4.545929170059967, -0.305361571312272, -2.068706090724009, 0.36738131081906034, -4.596829363692114, -0.7145714514809897, -2.1870440120521244, -3.619802243175367, -3.0854247199357574, 0.06528116501171306, -8.04466137206524, -2.416515121000999, 2.1550821626518215, -3.550888972902983, 2.999501801038642, 5.540352816509654, -13.136826425368891, -9.682667832954975, -4.089456620979975, -2.4600469634137005, -7.980395623816304, -0.0 ], "y": [ -5.746731056831749, -3.560535696444575, -3.610268946914751, 2.5135502228088797, -0.5965513852397577, -1.0116808329390385, -0.8182560448021005, -2.4071834175723335, 2.5577489299929255, 0.348192434742279, -1.3114357729283859, -3.3382598457223502, -1.0068955594731426, -0.06867630083734555, 0.10424166292801242, -0.07210829452142886, -3.188401274811581, 1.960086317210278, 0.012674904654773453, 1.1203410785772772, 0.6155055435112974, -2.5504391581007666, 5.964668435375144, -0.4205314884998786, -2.726644960728147, -5.859418405279977, 3.1966830537300392, 3.950198735676712, 2.0568967995693455, 0.652872009905956, 1.286780707668062, 4.62085929335188, -0.7622630447726769, 0.123503526062618, -1.5431517157186252, -3.2650236570623457, -1.7414179188197445, -1.1257173899572386, -1.3809544654506094, 7.651708533863019, -2.2015816842718356, -3.063188080332386, 4.012625447421733, -1.168849751600452, 0.7526902410881314, -0.7604599707221315, 2.2308412374353317, -1.371441760784444, 0.8978911718791985, -0.9236769844543923, -1.2656161965636414, -1.0555488550285586, 1.1039153988973123, -3.2310973947137103, -1.4614566797717248, 0.9918050829129264, -1.3210922238627325, 2.8001328033355772, 10.809115959705165, -3.839017981482167, -2.765325873487098, -1.5757183832623392, -2.1787372563938, -2.789593215553577, -0.0 ] }, { "error_x": { "array": [ 0.9032151733520527, 0.9624260346524522, 0.9593909284698995, 1.1347609507759298, 1.0408257278018285, 1.0435860695228432, 1.0711864083714089, 1.0156140087849477, 1.128257833835402, 1.0788991832923005, 1.0706037001289912, 0.9284671425890606, 1.0523829338844282, 1.0903373892267718, 1.0861328996665252, 1.1145865581026473, 1.0063514862723812, 1.0994776629118859, 1.091675914931962, 1.0844593315915725, 1.0017853270032084, 0.9935701990690048, 1.1127508961856405, 1.0668110378139575, 1.0328601627012277, 0.8797268687204242, 1.140518932271723, 1.1417536403033617, 1.0764096893237975, 1.0965566556283748, 1.0984998396729218, 1.102540333829891, 1.0794770222090497, 1.085962611129493, 1.0433251902342626, 0.9820278721895025, 1.0166480934308502, 1.0343546221526094, 1.0382944853121021, 1.1551456071793769, 1.0464931893207308, 1.0023861997986674, 1.0516802404518037, 1.0346896327290815, 1.08327143184596, 1.0698309404523143, 1.0904297395971863, 1.0222860350150513, 1.0810704079796398, 1.0613893596292459, 1.0320103439592427, 1.0528002249874768, 1.0912064123686547, 0.9988930110467326, 1.0571937806819418, 1.1193739547640122, 1.0530800272030827, 1.1303999126786766, 1.180499014903316, 0.9513079288502773, 0.9719033134076178, 1.03791308728753, 1.0588852902469934, 0.9945314913677392, 0.5199803498226312 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "error_y": { "array": [ 0.6547387343317975, 0.6532033115837151, 0.6609466315731528, 0.6813554440847576, 0.6694705491295897, 0.6591303137747925, 0.6563086390178189, 0.6508039407989954, 0.6839216136881153, 0.6604406759568969, 0.6601787231575951, 0.6565366929811817, 0.6613093240694986, 0.6561140874958142, 0.6633706122047732, 0.6637799634962176, 0.6555405967295402, 0.6704566164608853, 0.66273573692496, 0.6685871803108022, 0.6628627361853555, 0.6572002148831203, 0.712446900341419, 0.6636937933944925, 0.6586242603445244, 0.6475500126287197, 0.6773760670047011, 0.6859237814996093, 0.6688072097037889, 0.6516465341587597, 0.6738024004862555, 0.7036162417194723, 0.6668939573219893, 0.6610547090506331, 0.6553518681013905, 0.6508051033775936, 0.64797757405811, 0.6713714106339492, 0.656723211631772, 0.7346090086077302, 0.6642524778308558, 0.6574536033119194, 0.7031723757695748, 0.6494774398002832, 0.6704885519899918, 0.6622056608350378, 0.6797639514603396, 0.6735895146091605, 0.6692892385552365, 0.6620882696925839, 0.6531942830594355, 0.6650013545172943, 0.6769982685390781, 0.6574731612170758, 0.6674033123702455, 0.6722373506713744, 0.6559893752039715, 0.6748097809173098, 0.7697444470145935, 0.6626605822603224, 0.6581382444319371, 0.663286850538724, 0.658955602798785, 0.6526018616070007, 0.3176861573292605 ], "color": "rgba(128,177,211,0.4)", "type": "data" }, "hoverinfo": "text", "marker": { "color": "rgba(128,177,211,1)" }, "mode": "markers", "name": "In-sample", "showlegend": true, "text": [ "Arm 0_0

metric_1: -17.022% [-17.925%, -16.119%]
metric_2: -5.683% [-6.338%, -5.028%]
Parameterization has too many items to render on hover (15).", "Arm 0_1

metric_1: -11.473% [-12.436%, -10.511%]
metric_2: -3.526% [-4.180%, -2.873%]
Parameterization has too many items to render on hover (15).", "Arm 0_10

metric_1: -11.26% [-12.220%, -10.301%]
metric_2: -3.535% [-4.196%, -2.874%]
Parameterization has too many items to render on hover (15).", "Arm 0_11

metric_1: 2.99% [1.855%, 4.125%]
metric_2: 2.542% [1.861%, 3.224%]
Parameterization has too many items to render on hover (15).", "Arm 0_12

metric_1: -3.205% [-4.246%, -2.164%]
metric_2: -0.629% [-1.298%, 0.041%]
Parameterization has too many items to render on hover (15).", "Arm 0_13

metric_1: -2.828% [-3.871%, -1.784%]
metric_2: -1.055% [-1.714%, -0.396%]
Parameterization has too many items to render on hover (15).", "Arm 0_14

metric_1: -0.807% [-1.878%, 0.264%]
metric_2: -0.832% [-1.488%, -0.176%]
Parameterization has too many items to render on hover (15).", "Arm 0_15

metric_1: -7.632% [-8.648%, -6.617%]
metric_2: -2.319% [-2.970%, -1.669%]
Parameterization has too many items to render on hover (15).", "Arm 0_16

metric_1: 2.493% [1.365%, 3.621%]
metric_2: 2.582% [1.898%, 3.266%]
Parameterization has too many items to render on hover (15).", "Arm 0_17

metric_1: -1.148% [-2.227%, -0.069%]
metric_2: 0.326% [-0.334%, 0.987%]
Parameterization has too many items to render on hover (15).", "Arm 0_18

metric_1: -1.417% [-2.488%, -0.346%]
metric_2: -1.321% [-1.981%, -0.661%]
Parameterization has too many items to render on hover (15).", "Arm 0_19

metric_1: -14.735% [-15.664%, -13.807%]
metric_2: -3.315% [-3.971%, -2.658%]
Parameterization has too many items to render on hover (15).", "Arm 0_2

metric_1: -3.211% [-4.263%, -2.158%]
metric_2: -0.992% [-1.653%, -0.331%]
Parameterization has too many items to render on hover (15).", "Arm 0_20

metric_1: 0.073% [-1.017%, 1.164%]
metric_2: -0.063% [-0.719%, 0.594%]
Parameterization has too many items to render on hover (15).", "Arm 0_21

metric_1: 0.528% [-0.558%, 1.614%]
metric_2: 0.066% [-0.597%, 0.730%]
Parameterization has too many items to render on hover (15).", "Arm 0_22

metric_1: 1.746% [0.632%, 2.861%]
metric_2: -0.055% [-0.719%, 0.608%]
Parameterization has too many items to render on hover (15).", "Arm 0_23

metric_1: -6.404% [-7.411%, -5.398%]
metric_2: -3.15% [-3.805%, -2.494%]
Parameterization has too many items to render on hover (15).", "Arm 0_24

metric_1: 0.836% [-0.264%, 1.935%]
metric_2: 1.893% [1.223%, 2.564%]
Parameterization has too many items to render on hover (15).", "Arm 0_25

metric_1: 0.699% [-0.393%, 1.790%]
metric_2: 0.024% [-0.639%, 0.686%]
Parameterization has too many items to render on hover (15).", "Arm 0_26

metric_1: 0.319% [-0.766%, 1.403%]
metric_2: 1.077% [0.409%, 1.746%]
Parameterization has too many items to render on hover (15).", "Arm 0_27

metric_1: -6.918% [-7.920%, -5.916%]
metric_2: 0.651% [-0.012%, 1.314%]
Parameterization has too many items to render on hover (15).", "Arm 0_28

metric_1: -7.463% [-8.457%, -6.469%]
metric_2: -2.49% [-3.147%, -1.832%]
Parameterization has too many items to render on hover (15).", "Arm 0_29

metric_1: 1.48% [0.367%, 2.592%]
metric_2: 5.858% [5.145%, 6.570%]
Parameterization has too many items to render on hover (15).", "Arm 0_3

metric_1: -2.25% [-3.317%, -1.183%]
metric_2: -0.448% [-1.112%, 0.215%]
Parameterization has too many items to render on hover (15).", "Arm 0_30

metric_1: -4.65% [-5.683%, -3.617%]
metric_2: -2.712% [-3.371%, -2.054%]
Parameterization has too many items to render on hover (15).", "Arm 0_31

metric_1: -19.06% [-19.939%, -18.180%]
metric_2: -5.792% [-6.440%, -5.145%]
Parameterization has too many items to render on hover (15).", "Arm 0_32

metric_1: 3.344% [2.204%, 4.485%]
metric_2: 3.158% [2.481%, 3.835%]
Parameterization has too many items to render on hover (15).", "Arm 0_33

metric_1: 3.298% [2.156%, 4.440%]
metric_2: 3.853% [3.167%, 4.539%]
Parameterization has too many items to render on hover (15).", "Arm 0_34

metric_1: -3.845% [-4.922%, -2.769%]
metric_2: 2.026% [1.357%, 2.695%]
Parameterization has too many items to render on hover (15).", "Arm 0_35

metric_1: 0.616% [-0.480%, 1.713%]
metric_2: 0.619% [-0.033%, 1.270%]
Parameterization has too many items to render on hover (15).", "Arm 0_36

metric_1: 0.606% [-0.493%, 1.704%]
metric_2: 1.328% [0.654%, 2.002%]
Parameterization has too many items to render on hover (15).", "Arm 0_37

metric_1: 0.801% [-0.301%, 1.904%]
metric_2: 4.539% [3.836%, 5.243%]
Parameterization has too many items to render on hover (15).", "Arm 0_38

metric_1: -1.23% [-2.310%, -0.151%]
metric_2: -0.74% [-1.407%, -0.074%]
Parameterization has too many items to render on hover (15).", "Arm 0_39

metric_1: -0.189% [-1.275%, 0.897%]
metric_2: 0.11% [-0.551%, 0.771%]
Parameterization has too many items to render on hover (15).", "Arm 0_4

metric_1: -2.877% [-3.920%, -1.834%]
metric_2: -1.538% [-2.193%, -0.883%]
Parameterization has too many items to render on hover (15).", "Arm 0_40

metric_1: -8.667% [-9.649%, -7.685%]
metric_2: -3.287% [-3.937%, -2.636%]
Parameterization has too many items to render on hover (15).", "Arm 0_41

metric_1: -5.112% [-6.129%, -4.096%]
metric_2: -1.776% [-2.424%, -1.128%]
Parameterization has too many items to render on hover (15).", "Arm 0_42

metric_1: -4.866% [-5.900%, -3.831%]
metric_2: -1.107% [-1.779%, -0.436%]
Parameterization has too many items to render on hover (15).", "Arm 0_43

metric_1: -3.621% [-4.659%, -2.582%]
metric_2: -1.374% [-2.031%, -0.717%]
Parameterization has too many items to render on hover (15).", "Arm 0_44

metric_1: 4.416% [3.261%, 5.571%]
metric_2: 7.531% [6.797%, 8.266%]
Parameterization has too many items to render on hover (15).", "Arm 0_45

metric_1: -3.094% [-4.140%, -2.047%]
metric_2: -2.19% [-2.854%, -1.526%]
Parameterization has too many items to render on hover (15).", "Arm 0_46

metric_1: -6.987% [-7.990%, -5.985%]
metric_2: -3.082% [-3.740%, -2.425%]
Parameterization has too many items to render on hover (15).", "Arm 0_47

metric_1: -3.456% [-4.507%, -2.404%]
metric_2: 3.964% [3.261%, 4.667%]
Parameterization has too many items to render on hover (15).", "Arm 0_48

metric_1: -4.487% [-5.521%, -3.452%]
metric_2: -1.178% [-1.827%, -0.528%]
Parameterization has too many items to render on hover (15).", "Arm 0_49

metric_1: -0.276% [-1.360%, 0.807%]
metric_2: 0.752% [0.081%, 1.422%]
Parameterization has too many items to render on hover (15).", "Arm 0_5

metric_1: -2.093% [-3.163%, -1.023%]
metric_2: -0.772% [-1.435%, -0.110%]
Parameterization has too many items to render on hover (15).", "Arm 0_50

metric_1: 0.34% [-0.750%, 1.431%]
metric_2: 2.197% [1.517%, 2.877%]
Parameterization has too many items to render on hover (15).", "Arm 0_51

metric_1: -4.611% [-5.633%, -3.589%]
metric_2: -1.337% [-2.011%, -0.664%]
Parameterization has too many items to render on hover (15).", "Arm 0_52

metric_1: -0.757% [-1.838%, 0.324%]
metric_2: 0.878% [0.208%, 1.547%]
Parameterization has too many items to render on hover (15).", "Arm 0_53

metric_1: -2.166% [-3.228%, -1.105%]
metric_2: -0.911% [-1.573%, -0.249%]
Parameterization has too many items to render on hover (15).", "Arm 0_54

metric_1: -3.611% [-4.643%, -2.579%]
metric_2: -1.267% [-1.920%, -0.613%]
Parameterization has too many items to render on hover (15).", "Arm 0_55

metric_1: -3.053% [-4.106%, -2.000%]
metric_2: -1.055% [-1.720%, -0.390%]
Parameterization has too many items to render on hover (15).", "Arm 0_56

metric_1: 0.115% [-0.976%, 1.206%]
metric_2: 1.138% [0.461%, 1.815%]
Parameterization has too many items to render on hover (15).", "Arm 0_57

metric_1: -8.012% [-9.011%, -7.013%]
metric_2: -3.221% [-3.878%, -2.563%]
Parameterization has too many items to render on hover (15).", "Arm 0_58

metric_1: -2.444% [-3.501%, -1.387%]
metric_2: -1.454% [-2.122%, -0.787%]
Parameterization has too many items to render on hover (15).", "Arm 0_59

metric_1: 2.105% [0.986%, 3.225%]
metric_2: 0.993% [0.320%, 1.665%]
Parameterization has too many items to render on hover (15).", "Arm 0_6

metric_1: -3.549% [-4.602%, -2.496%]
metric_2: -1.318% [-1.974%, -0.662%]
Parameterization has too many items to render on hover (15).", "Arm 0_60

metric_1: 2.966% [1.836%, 4.097%]
metric_2: 2.781% [2.106%, 3.455%]
Parameterization has too many items to render on hover (15).", "Arm 0_61

metric_1: 5.393% [4.212%, 6.573%]
metric_2: 10.553% [9.783%, 11.322%]
Parameterization has too many items to render on hover (15).", "Arm 0_62

metric_1: -13.026% [-13.978%, -12.075%]
metric_2: -3.787% [-4.450%, -3.125%]
Parameterization has too many items to render on hover (15).", "Arm 0_63

metric_1: -9.619% [-10.591%, -8.647%]
metric_2: -2.738% [-3.397%, -2.080%]
Parameterization has too many items to render on hover (15).", "Arm 0_7

metric_1: -4.118% [-5.156%, -3.080%]
metric_2: -1.61% [-2.273%, -0.947%]
Parameterization has too many items to render on hover (15).", "Arm 0_8

metric_1: -2.464% [-3.523%, -1.405%]
metric_2: -2.148% [-2.807%, -1.489%]
Parameterization has too many items to render on hover (15).", "Arm 0_9

metric_1: -7.957% [-8.951%, -6.962%]
metric_2: -2.794% [-3.447%, -2.142%]
Parameterization has too many items to render on hover (15).", "Arm status_quo

metric_1: -0.0% [-0.520%, 0.520%]
metric_2: -0.0% [-0.318%, 0.318%]
Parameterization has too many items to render on hover (15)." ], "type": "scatter", "visible": true, "x": [ -17.022045315104542, -11.473276363575263, -11.260309485449103, 2.9899364287480865, -3.2049806778237757, -2.827647808086468, -0.8072357684589017, -7.632201738791567, 2.4931910046363135, -1.1483177933819395, -1.4169105153422092, -14.735353419604191, -3.2108750242520334, 0.07332105300986574, 0.52794229128138, 1.7461298706241524, -6.404301715961119, 0.8357526716703064, 0.6988214644260046, 0.3185581570295636, -6.918118515914437, -7.462960459390675, 1.4797223030792654, -2.250057847895754, -4.649788400829341, -19.059723742205165, 3.3442377220374424, 3.2977920613814318, -3.845344844296581, 0.6160918095707203, 0.605873558955114, 0.8012482568804604, -1.2301194692314896, -0.18910424701708023, -2.8771572632426596, -8.666686434035148, -5.1123185935816275, -4.86569050603379, -3.620616821502621, 4.416040423353074, -3.093977759193243, -6.987479150965008, -3.45561072825807, -4.486522144334903, -0.27632651192689084, -2.092939160334957, 0.34040916740729205, -4.6111158309365, -0.7572230824447213, -2.1663524739549853, -3.611088388568022, -3.053027739941141, 0.11513485293184113, -8.011705772759294, -2.4442287384702572, 2.1052356784416553, -3.5492459592908614, 2.966172129108271, 5.392515165139946, -13.026404058841306, -9.619176461732291, -4.118044655666446, -2.4636655602657878, -7.956569241984238, -0.0 ], "y": [ -5.682890950120342, -3.5263840662392654, -3.5354295227440935, 2.542378866552539, -0.6285418092402999, -1.0553405972573464, -0.8320014955517723, -2.3193462375707914, 2.58163977526919, 0.32621268542555526, -1.3207797795363851, -3.314816695063182, -0.9921710458941605, -0.06250694414329654, 0.06640049636248754, -0.05540222390315327, -3.1495291448321576, 1.8934882315080965, 0.023529017577203423, 1.0773772731169438, 0.6512642256730126, -2.489527283561809, 5.857858819488534, -0.4484318638610448, -2.712182537468451, -5.792314875460171, 3.1580957792386144, 3.8533176800717563, 2.026262991103158, 0.6187928699607773, 1.3280865831055046, 4.5394243735749775, -0.7404858515521753, 0.10968492064137063, -1.5381048845570704, -3.2865434005114005, -1.7761680789956928, -1.1071508425287317, -1.3739381694523336, 7.5312245476719, -2.1901762525545276, -3.082214357601481, 3.963995297169013, -1.1779086966890597, 0.7516740725998268, -0.772463613395467, 2.196834434560228, -1.3372360602549658, 0.8775653888918572, -0.9111761852726025, -1.2666733248842346, -1.055168296030082, 1.1378418002527275, -3.2209479427630807, -1.4541461401402338, 0.9926717601692114, -1.3177840022000746, 2.7806338967179443, 10.552746740900881, -3.7873810524577847, -2.7384540934203034, -1.6100825481553491, -2.147955641504198, -2.7941662700346765, -0.0 ] }, { "error_x": { "array": [ 4.83875056447029, 5.102041394990379, 4.662392753925473 ], "color": "rgba(251,128,114,0.4)", "type": "data" }, "error_y": { "array": [ 2.808469142463111, 2.8179402082703007, 2.752087033496351 ], "color": "rgba(251,128,114,0.4)", "type": "data" }, "hoverinfo": "text", "marker": { "color": "rgba(251,128,114,1)" }, "mode": "markers", "name": "unconstrained", "showlegend": true, "text": [ "Arm b849

metric_1: 6.105% [1.266%, 10.944%]
metric_2: 8.281% [5.473%, 11.090%]
Parameterization has too many items to render on hover (15).", "Arm 3ad4

metric_1: 5.457% [0.355%, 10.559%]
metric_2: 6.539% [3.721%, 9.357%]
Parameterization has too many items to render on hover (15).", "Arm 998f

metric_1: 5.544% [0.881%, 10.206%]
metric_2: 8.915% [6.163%, 11.667%]
Parameterization has too many items to render on hover (15)." ], "type": "scatter", "visible": true, "x": [ 6.104854772592679, 5.4570750726576485, 5.543724025368393 ], "y": [ 8.281103305736018, 6.539035341740647, 8.915317158366165 ] }, { "error_x": { "array": [ 4.620799942815001, 4.341764333801678, 4.67115577126623 ], "color": "rgba(141,211,199,0.4)", "type": "data" }, "error_y": { "array": [ 2.5437250774011937, 2.2532861566721074, 2.6503979821172843 ], "color": "rgba(141,211,199,0.4)", "type": "data" }, "hoverinfo": "text", "marker": { "color": "rgba(141,211,199,1)" }, "mode": "markers", "name": "loose_constraint", "showlegend": true, "text": [ "Arm 2663

metric_1: 5.026% [0.405%, 9.647%]
metric_2: 3.761% [1.217%, 6.305%]
Parameterization has too many items to render on hover (15).", "Arm 5174

metric_1: 4.459% [0.117%, 8.800%]
metric_2: 3.719% [1.466%, 5.972%]
Parameterization has too many items to render on hover (15).", "Arm 7d93

metric_1: 4.418% [-0.254%, 9.089%]
metric_2: 3.722% [1.072%, 6.372%]
Parameterization has too many items to render on hover (15)." ], "type": "scatter", "visible": true, "x": [ 5.02607550500538, 4.458715026100761, 4.4175213815962335 ], "y": [ 3.7610174232852804, 3.7188506476744347, 3.7220056829809645 ] }, { "error_x": { "array": [ 4.660267831501327, 4.887460782453055, 5.635425857830732 ], "color": "rgba(188,128,189,0.4)", "type": "data" }, "error_y": { "array": [ 2.5193699490248327, 2.8125842237332273, 3.1483376877014773 ], "color": "rgba(188,128,189,0.4)", "type": "data" }, "hoverinfo": "text", "marker": { "color": "rgba(188,128,189,1)" }, "mode": "markers", "name": "tight_constraint", "showlegend": true, "text": [ "Arm b7d5

metric_1: 2.692% [-1.968%, 7.353%]
metric_2: 0.431% [-2.089%, 2.950%]
Parameterization has too many items to render on hover (15).", "Arm e3b1

metric_1: 2.395% [-2.493%, 7.282%]
metric_2: 0.867% [-1.946%, 3.679%]
Parameterization has too many items to render on hover (15).", "Arm 140e

metric_1: 1.953% [-3.682%, 7.589%]
metric_2: 0.672% [-2.476%, 3.820%]
Parameterization has too many items to render on hover (15)." ], "type": "scatter", "visible": true, "x": [ 2.692240778165111, 2.3947768825258926, 1.9532508957716572 ], "y": [ 0.4308677854504356, 0.866514006381011, 0.6719251619346646 ] } ], "layout": { "annotations": [ { "showarrow": false, "text": "Show CI", "x": 1.18, "xref": "paper", "y": 0.7, "yanchor": "middle", "yref": "paper" }, { "showarrow": false, "text": "Type", "x": 1.18, "xref": "paper", "y": 0.6, "yanchor": "middle", "yref": "paper" } ], "font": { "size": 10 }, "height": 600, "hovermode": "closest", "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "sequentialminus": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "title": { "text": "Objective Tradeoffs" }, "updatemenus": [ { "buttons": [ { "args": [ { "error_x.thickness": 2, "error_x.width": 4, "error_y.thickness": 2, "error_y.width": 4 } ], "label": "Yes", "method": "restyle" }, { "args": [ { "error_x.thickness": 0, "error_x.width": 0, "error_y.thickness": 0, "error_y.width": 0 } ], "label": "No", "method": "restyle" } ], "x": 1.25, "xanchor": "left", "y": 0.67, "yanchor": "middle" }, { "buttons": [ { "args": [ { "visible": [ false, true, true, true, true ] } ], "label": "Modeled", "method": "restyle" }, { "args": [ { "visible": [ true, false, false, false, false ] } ], "label": "Observed", "method": "restyle" } ], "x": 1.25, "xanchor": "left", "y": 0.57, "yanchor": "middle" } ], "width": 800, "xaxis": { "title": { "text": "metric_1 (%)" }, "zeroline": true, "zerolinecolor": "red" }, "yaxis": { "title": { "text": "metric_2 (%)" }, "zeroline": true, "zerolinecolor": "red" } } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "render(plot_multiple_metrics(\n", " gp,\n", " metric_x=METRIC_X_AXIS,\n", " metric_y=METRIC_Y_AXIS,\n", " generator_runs_dict={\n", " 'unconstrained': unconstrained,\n", " 'loose_constraint': constraint_5_results,\n", " 'tight_constraint': constraint_1_results,\n", " }\n", "))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Creating a New Trial\n", "\n", "Having done the analysis and candidate generation for three different optimization configs, we can easily create a new `BatchTrial` which combines the candidates from these three different optimizations. Each set of candidates looks promising -- the point estimates are higher along both metric values than in the previous batch. However, there is still a good bit of uncertainty in our predictions. It is hard to choose between the different constraint settings without reducing this noise, so we choose to run a new trial with all three constraint settings. However, we're generally convinced that the tight constraint is too conservative. We'd still like to reduce our uncertainty in that region, but we'll only take one arm from that set." ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "BatchTrial(experiment_name='human_in_the_loop_tutorial', index=1, status=TrialStatus.CANDIDATE)" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# We can add entire generator runs, when constructing a new trial. \n", "trial = experiment.new_batch_trial().add_generator_run(unconstrained).add_generator_run(constraint_5_results)\n", "\n", "# Or, we can hand-pick arms. \n", "trial.add_arm(constraint_1_results.arms[0])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The arms are combined into a single trial, along with the `status_quo` arm. Their generator can be accessed from the trial as well. " ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[Arm(name='1_0', parameters={'x_excellent': 0.6801095079537054, 'x_good': 0.3096643734731816, 'x_moderate': 0.2268465534619125, 'x_poor': 0.10301451222224842, 'x_unknown': 0.7558682175629852, 'y_excellent': 2.5539730045523883, 'y_good': 1.7638260143027995, 'y_moderate': 0.8266042026029538, 'y_poor': 0.29791108777321906, 'y_unknown': 1.9023532309000115, 'z_excellent': 4805436.190497896, 'z_good': 3664441.0826730602, 'z_moderate': 2518174.190936773, 'z_poor': 1419484.580961257, 'z_unknown': 2420953.975798388}),\n", " Arm(name='1_1', parameters={'x_excellent': 0.721196203475683, 'x_good': 0.37030653687056975, 'x_moderate': 0.32091999212594874, 'x_poor': 0.12452723170442845, 'x_unknown': 0.5697482206692012, 'y_excellent': 2.593720628021836, 'y_good': 1.9583758009715997, 'y_moderate': 1.03675274842991, 'y_poor': 0.31022389213183055, 'y_unknown': 1.1704675251489047, 'z_excellent': 4915915.909124852, 'z_good': 2974242.6526909843, 'z_moderate': 3021534.3809279236, 'z_poor': 2522372.2331922753, 'z_unknown': 1701225.1383448096}),\n", " Arm(name='1_2', parameters={'x_excellent': 0.7161239757005161, 'x_good': 0.2602096543554335, 'x_moderate': 0.17147983516570608, 'x_poor': 0.1070495980422025, 'x_unknown': 0.9300516881138716, 'y_excellent': 2.439133612815966, 'y_good': 1.6642794296688157, 'y_moderate': 0.5412182534268437, 'y_poor': 0.2999274189963448, 'y_unknown': 2.209757099037322, 'z_excellent': 4469148.538475683, 'z_good': 4152654.3539818586, 'z_moderate': 2031563.5151423083, 'z_poor': 726526.1083800276, 'z_unknown': 3189133.8213911387}),\n", " Arm(name='1_3', parameters={'x_excellent': 0.7808776142222997, 'x_good': 0.5324241675737768, 'x_moderate': 0.3273285916293381, 'x_poor': 0.09377434820944054, 'x_unknown': 0.6752134908415576, 'y_excellent': 2.645864672234603, 'y_good': 1.8060610412994977, 'y_moderate': 0.8201705878787885, 'y_poor': 0.3143423125181757, 'y_unknown': 1.6791407169767831, 'z_excellent': 4822208.429064664, 'z_good': 3404522.339038801, 'z_moderate': 3033439.927439515, 'z_poor': 1909307.8757109842, 'z_unknown': 2425377.396356267}),\n", " Arm(name='1_4', parameters={'x_excellent': 0.8357558230107469, 'x_good': 0.47528039671144734, 'x_moderate': 0.41934190502275576, 'x_poor': 0.25871498704975593, 'x_unknown': 0.6792363075062642, 'y_excellent': 2.7021793704488446, 'y_good': 2.179367393872921, 'y_moderate': 1.1656913172572276, 'y_poor': 0.5800573240657181, 'y_unknown': 1.5762345601224288, 'z_excellent': 4438883.541358687, 'z_good': 4236281.559372088, 'z_moderate': 4059782.51774204, 'z_poor': 2315582.5640817, 'z_unknown': 3639775.2683506617}),\n", " Arm(name='1_5', parameters={'x_excellent': 0.681680104269142, 'x_good': 0.5128244963638185, 'x_moderate': 0.3817560965554567, 'x_poor': 0.021336215925653074, 'x_unknown': 0.5047814820754957, 'y_excellent': 2.332084709925601, 'y_good': 1.6820353024882198, 'y_moderate': 1.0269542089224835, 'y_poor': 0.2619581816459108, 'y_unknown': 1.2579951674682204, 'z_excellent': 4652383.262944627, 'z_good': 2740171.231518933, 'z_moderate': 2474184.6916821515, 'z_poor': 3199070.669164602, 'z_unknown': 729670.6973235838}),\n", " Arm(name='1_6', parameters={'x_excellent': 0.7830416577575473, 'x_good': 0.6955560175728306, 'x_moderate': 0.4287696192119522, 'x_poor': 0.023271801209422925, 'x_unknown': 0.7597195547731662, 'y_excellent': 2.7737688078488363, 'y_good': 1.7888062539445546, 'y_moderate': 0.8109435377484814, 'y_poor': 0.14540668522121042, 'y_unknown': 1.3955110290010468, 'z_excellent': 4337292.166194241, 'z_good': 2754307.607383342, 'z_moderate': 1662601.7686083491, 'z_poor': 2738246.0195795083, 'z_unknown': 1869409.2943936042})]" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "experiment.trials[1].arms" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The original `GeneratorRuns` can be accessed from within the trial as well. This is useful for later analyses, allowing introspection of the `OptimizationConfig` used for generation (as well as other information, e.g. `SearchSpace` used for generation)." ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[GeneratorRunStruct(generator_run=GeneratorRun(3 arms, total weight 3.0), weight=1.0),\n", " GeneratorRunStruct(generator_run=GeneratorRun(3 arms, total weight 3.0), weight=1.0),\n", " GeneratorRunStruct(generator_run=GeneratorRun(1 arms, total weight 1.0), weight=1.0)]" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "experiment.trials[1]._generator_run_structs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here, we can see the unconstrained set-up used for our first set of candidates. " ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "OptimizationConfig(objective=Objective(metric_name=\"metric_1\", minimize=False), outcome_constraints=[])" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "experiment.trials[1]._generator_run_structs[0].generator_run.optimization_config" ] } ], "metadata": { "kernelspec": { "display_name": "python3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.1" } }, "nbformat": 4, "nbformat_minor": 2 }