ax.plot

Plots

Base

class ax.plot.base.AxPlotConfig[source]

Bases: ax.plot.base._AxPlotConfigBase

Config for plots

class ax.plot.base.AxPlotTypes[source]

Bases: enum.Enum

Enum of Ax plot types.

BANDIT_ROLLOUT = 4
CONTOUR = 0
GENERIC = 1
INTERACT_CONTOUR = 3
SLICE = 2
class ax.plot.base.PlotData[source]

Bases: tuple

Struct for plot data, including both in-sample and out-of-sample arms

in_sample

Alias for field number 1

metrics

Alias for field number 0

out_of_sample

Alias for field number 2

status_quo_name

Alias for field number 3

class ax.plot.base.PlotInSampleArm[source]

Bases: tuple

Struct for in-sample arms (both observed and predicted data)

context_stratum

Alias for field number 6

name

Alias for field number 0

parameters

Alias for field number 1

se

Alias for field number 4

se_hat

Alias for field number 5

y

Alias for field number 2

y_hat

Alias for field number 3

class ax.plot.base.PlotMetric[source]

Bases: tuple

Struct for metric

metric

Alias for field number 0

pred

Alias for field number 1

class ax.plot.base.PlotOutOfSampleArm[source]

Bases: tuple

Struct for out-of-sample arms (only predicted data)

context_stratum

Alias for field number 4

name

Alias for field number 0

parameters

Alias for field number 1

se_hat

Alias for field number 3

y_hat

Alias for field number 2

Color

class ax.plot.color.COLORS[source]

Bases: enum.Enum

An enumeration.

CORAL = (251, 128, 114)
LIGHT_PURPLE = (190, 186, 218)
ORANGE = (253, 180, 98)
PINK = (188, 128, 189)
STEELBLUE = (128, 177, 211)
TEAL = (141, 211, 199)
ax.plot.color.plotly_color_scale(list_of_rgb_tuples, reverse=False, alpha=1)[source]

Convert list of RGB tuples to list of tuples, where each tuple is break in [0, 1] and stringified RGBA color.

Return type:List[Tuple[float, str]]
ax.plot.color.rgba(rgb_tuple, alpha=1)[source]

Convert RGB tuple to an RGBA string.

Return type:str

Contour Plot

ax.plot.contour.interact_contour(model, metric_name, generator_runs_dict=None, relative=False, density=50, slice_values=None, lower_is_better=False)[source]

Create interactive plot with predictions for a 2-d slice of the parameter space.

Parameters:
  • model (ModelBridge) – ModelBridge that contains model for predictions
  • metric_name (str) – Name of metric to plot
  • generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.
  • relative (bool) – Predictions relative to status quo
  • density (int) – Number of points along slice to evaluate predictions.
  • slice_values (Optional[Dict[str, Any]]) – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters.
  • lower_is_better (bool) – Lower values for metric are better.
Return type:

AxPlotConfig

ax.plot.contour.plot_contour(model, param_x, param_y, metric_name, generator_runs_dict=None, relative=False, density=50, slice_values=None, lower_is_better=False)[source]

Plot predictions for a 2-d slice of the parameter space.

Parameters:
  • model (ModelBridge) – ModelBridge that contains model for predictions
  • param_x (str) – Name of parameter that will be sliced on x-axis
  • param_y (str) – Name of parameter that will be sliced on y-axis
  • metric_name (str) – Name of metric to plot
  • generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.
  • relative (bool) – Predictions relative to status quo
  • density (int) – Number of points along slice to evaluate predictions.
  • slice_values (Optional[Dict[str, Any]]) – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters.
  • lower_is_better (bool) – Lower values for metric are better.
Return type:

AxPlotConfig

Model Diagnostic Plot

ax.plot.diagnostic.interact_batch_comparison(observations, experiment, batch_x, batch_y, rel=False, status_quo_name=None)[source]

Compare repeated arms from two trials; select metric via dropdown.

Parameters:
  • observations (List[Observation]) – List of observations to compute comparison.
  • batch_x (int) – Index of batch for x-axis.
  • batch_y (int) – Index of bach for y-axis.
  • rel (bool) – Whether to relativize data against status_quo arm.
  • status_quo_name (Optional[str]) – Name of the status_quo arm.
Return type:

AxPlotConfig

ax.plot.diagnostic.interact_cross_validation(cv_results, show_context=True)[source]

Interactive cross-validation (CV) plotting; select metric via dropdown.

Note: uses the Plotly version of dropdown (which means that all data is stored within the notebook).

Parameters:
  • cv_results (List[CVResult]) – cross-validation results.
  • show_context (bool) – if True, show context on hover.
Return type:

AxPlotConfig

ax.plot.diagnostic.interact_empirical_model_validation(batch, data)[source]

Compare the model predictions for the batch arms against observed data.

Relies on the model predictions stored on the generator_runs of batch.

Parameters:
  • batch (BatchTrial) – Batch on which to perform analysis.
  • data (Data) – Observed data for the batch.
Return type:

AxPlotConfig

Returns:

AxPlotConfig for the plot.

ax.plot.diagnostic.tile_cross_validation(cv_results, show_arm_details_on_hover=True, show_context=True)[source]

Tile version of CV plots; sorted by ‘best fitting’ outcomes.

Plots are sorted in decreasing order using the p-value of a Fisher exact test statistic.

Parameters:
  • cv_results (List[CVResult]) – cross-validation results.
  • include_measurement_error – if True, include measurement_error metrics in plot.
  • show_arm_details_on_hover (bool) – if True, display parameterizations of arms on hover. Default is True.
  • show_context (bool) – if True (default), display context on hover.
Return type:

AxPlotConfig

Helpers

ax.plot.helper.arm_name_to_tuple(arm_name)[source]
Return type:Union[Tuple[int, int], Tuple[int]]
ax.plot.helper.get_fixed_values(model, slice_values=None)[source]

Get fixed values for parameters in a slice plot.

If there is an in-design status quo, those values will be used. Otherwise, the mean of RangeParameters or the mode of ChoiceParameters is used.

Any value in slice_values will override the above.

Parameters:
  • model (ModelBridge) – ModelBridge being used for plotting
  • slice_values (Optional[Dict[str, Any]]) – Map from parameter name to value at which is should be fixed.

Returns: Map from parameter name to fixed value.

Return type:Dict[str, Union[str, bool, float, int, None]]
ax.plot.helper.get_grid_for_parameter(parameter, density)[source]

Get a grid of points along the range of the parameter.

Will be a log-scale grid if parameter is log scale.

Parameters:
  • parameter (RangeParameter) – Parameter for which to generate grid.
  • density (int) – Number of points in the grid.
Return type:

ndarray

ax.plot.helper.get_plot_data(model, generator_runs_dict, metric_names=None)[source]

Format data object with metrics for in-sample and out-of-sample arms.

Calculate both observed and predicted metrics for in-sample arms. Calculate predicted metrics for out-of-sample arms passed via the generator_runs_dict argument.

In PlotData, in-sample observations are merged with IVW. In RawData, they are left un-merged and given as a list of dictionaries, one for each observation and having keys ‘arm_name’, ‘mean’, and ‘sem’.

Parameters:
  • model (ModelBridge) – The model.
  • generator_runs_dict (Dict[str, GeneratorRun]) – a mapping from generator run name to generator run.
  • metric_names (Optional[Set[str]]) – Restrict predictions to this set. If None, all metrics in the model will be returned.
Return type:

Tuple[PlotData, List[Dict[str, Union[float, str]]], Dict[str, Dict[str, Union[str, bool, float, int, None]]]]

Returns:

A tuple containing

  • PlotData object with in-sample and out-of-sample predictions.

  • List of observations like:

    {'metric_name': 'likes', 'arm_name': '0_1', 'mean': 1., 'sem': 0.1}.
    
  • Mapping from arm name to parameters.

ax.plot.helper.get_range_parameter(model, param_name)[source]

Get the range parameter with the given name from the model.

Throws if parameter doesn’t exist or is not a range parameter.

Parameters:
  • model (ModelBridge) – The model.
  • param_name (str) – The name of the RangeParameter to be found.

Returns: The RangeParameter named param_name.

Return type:RangeParameter
ax.plot.helper.get_range_parameters(model)[source]

Get a list of range parameters from a model.

Parameters:model (ModelBridge) – The model.

Returns: List of RangeParameters.

Return type:List[RangeParameter]
ax.plot.helper.resize_subtitles(figure, size)[source]

Rendering

ax.plot.render.plot_config_to_html(plot_config, plot_module_name='ax.plot', plot_resources={<AxPlotTypes.CONTOUR: 0>: 'contour.js', <AxPlotTypes.GENERIC: 1>: 'generic_plotly.js', <AxPlotTypes.INTERACT_CONTOUR: 3>: 'interact_contour.js', <AxPlotTypes.SLICE: 2>: 'slice.js', <AxPlotTypes.BANDIT_ROLLOUT: 4>: 'bandit_rollout.js'}, inject_helpers=False)[source]

Generate HTML + JS corresponding from a plot config.

Return type:str

Scatter Plots

ax.plot.scatter.interact_fitted(model, generator_runs_dict=None, rel=True, show_arm_details_on_hover=True, show_CI=True, arm_noun='arm')[source]

Interactive fitted outcome plots for each arm used in fitting the model.

Choose the outcome to plot using a dropdown.

Parameters:
  • model (ModelBridge) – model to use for predictions.
  • generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – a mapping from generator run name to generator run.
  • rel (bool) – if True, use relative effects. Default is True.
  • show_arm_details_on_hover (bool) – if True, display parameterizations of arms on hover. Default is True.
  • show_CI (bool) – if True, render confidence intervals.
  • arm_noun (str) – noun to use instead of “arm” (e.g. group)
Return type:

AxPlotConfig

ax.plot.scatter.lattice_multiple_metrics(model, generator_runs_dict=None, rel=True, show_arm_details_on_hover=False)[source]

Plot raw values or predictions of combinations of two metrics for arms.

Parameters:
  • model (ModelBridge) – model to draw predictions from.
  • generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – a mapping from generator run name to generator run.
  • rel (bool) – if True, use relative effects. Default is True.
  • show_arm_details_on_hover (bool) – if True, display parameterizations of arms on hover. Default is False.
Return type:

AxPlotConfig

ax.plot.scatter.plot_fitted(model, metric, generator_runs_dict=None, rel=True, custom_arm_order=None, custom_arm_order_name='Custom', show_CI=True)[source]

Plot fitted metrics.

Parameters:
  • model (ModelBridge) – model to use for predictions.
  • metric (str) – metric to plot predictions for.
  • generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – a mapping from generator run name to generator run.
  • rel (bool) – if True, use relative effects. Default is True.
  • custom_arm_order (Optional[List[str]]) – a list of arm names in the order corresponding to how they should be plotted on the x-axis. If not None, this is the default ordering.
  • custom_arm_order_name (str) – name for custom ordering to show in the ordering dropdown. Default is ‘Custom’.
  • show_CI (bool) – if True, render confidence intervals.
Return type:

AxPlotConfig

ax.plot.scatter.plot_multiple_metrics(model, metric_x, metric_y, generator_runs_dict=None, rel=True)[source]

Plot raw values or predictions of two metrics for arms.

All arms used in the model are included in the plot. Additional arms can be passed through the generator_runs_dict argument.

Parameters:
  • model (ModelBridge) – model to draw predictions from.
  • metric_x (str) – metric to plot on the x-axis.
  • metric_y (str) – metric to plot on the y-axis.
  • generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – a mapping from generator run name to generator run.
  • rel (bool) – if True, use relative effects. Default is True.
Return type:

AxPlotConfig

ax.plot.scatter.plot_objective_vs_constraints(model, objective, subset_metrics=None, generator_runs_dict=None, rel=True)[source]

Plot the tradeoff between an objetive and all other metrics in a model.

All arms used in the model are included in the plot. Additional arms can be passed through via the generator_runs_dict argument.

Parameters:
  • model (ModelBridge) – model to draw predictions from.
  • objective (str) – metric to optimize. Plotted on the x-axis.
  • subset_metrics (Optional[List[str]]) – list of metrics to plot on the y-axes if need a subset of all metrics in the model.
  • generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – a mapping from generator run name to generator run.
  • rel (bool) – if True, use relative effects. Default is True.
Return type:

AxPlotConfig

ax.plot.scatter.tile_fitted(model, generator_runs_dict=None, rel=True, show_arm_details_on_hover=False, show_CI=True, arm_noun='arm')[source]

Tile version of fitted outcome plots.

Parameters:
  • model (ModelBridge) – model to use for predictions.
  • generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – a mapping from generator run name to generator run.
  • rel (bool) – if True, use relative effects. Default is True.
  • show_arm_details_on_hover (bool) – if True, display parameterizations of arms on hover. Default is False.
  • show_CI (bool) – if True, render confidence intervals.
  • arm_noun (str) – noun to use instead of “arm” (e.g. group)
Return type:

AxPlotConfig

Slice Plot

ax.plot.slice.plot_slice(model, param_name, metric_name, generator_runs_dict=None, relative=False, density=50, slice_values=None)[source]

Plot predictions for a 1-d slice of the parameter space.

Parameters:
  • model (ModelBridge) – ModelBridge that contains model for predictions
  • param_name (str) – Name of parameter that will be sliced
  • metric_name (str) – Name of metric to plot
  • generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.
  • relative (bool) – Predictions relative to status quo
  • density (int) – Number of points along slice to evaluate predictions.
  • slice_values (Optional[Dict[str, Any]]) – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters.
Return type:

AxPlotConfig

Table

ax.plot.table_view.get_color(x, ci, rel, reverse)[source]

Determine the color of the table cell.

ax.plot.table_view.table_view_plot(experiment, data, use_empirical_bayes=True, only_data_frame=False, arm_noun='arm')[source]

Table of means and confidence intervals.

Table is of the form:

arm metric_1 metric_2
0_0 mean +- CI
0_1

Trace Plots

ax.plot.trace.generator_changes_scatter(generator_changes, y_range, generator_change_color=(141, 211, 199))[source]

Creates a graph object for the line(s) representing generator changes.

Parameters:
  • generator_changes (List[int]) – iterations, before which generators changed
  • y_range (List[float]) – upper and lower values of the y-range of the plot
  • generator_change_color (Tuple[int]) – tuple of 3 int values representing an RGB color. Defaults to orange.
Returns:

plotly graph objects for the lines representing generator

changes

Return type:

go.Scatter

ax.plot.trace.mean_trace_scatter(y, trace_color=(128, 177, 211), legend_label='mean')[source]

Creates a graph object for trace of the mean of the given series across runs.

Parameters:
  • y (ndarray) – (r x t) array with results from r runs and t trials.
  • trace_color (Tuple[int]) – tuple of 3 int values representing an RGB color. Defaults to blue.
  • legend_label (str) – label for this trace
Returns:

plotly graph object

Return type:

go.Scatter

ax.plot.trace.optimization_times(fit_times, gen_times, title='')[source]

Plots wall times for each method as a bar chart.

Parameters:
  • fit_times (Dict[str, List[float]]) – A map from method name to a list of the model fitting times.
  • gen_times (Dict[str, List[float]]) – A map from method name to a list of the gen times.
  • title (str) – Title for this plot.

Returns: AxPlotConfig with the plot

Return type:AxPlotConfig
ax.plot.trace.optimization_trace_all_methods(y_dict, optimum=None, title='', ylabel='', trace_colors=[(128, 177, 211), (251, 128, 114), (141, 211, 199), (188, 128, 189), (190, 186, 218), (253, 180, 98)], optimum_color=(253, 180, 98))[source]

Plots a comparison of optimization traces with 2-SEM bands for multiple methods on the same problem.

Parameters:
  • y – a mapping of method names to (r x t) arrays, where r is the number of runs in the test, and t is the number of trials.
  • optimum (Optional[float]) – value of the optimal objective.
  • title (str) – Title for this plot.
  • ylabel (str) – Label for y axis
  • trace_colors (List[Tuple[int]]) – tuples of 3 int values representing RGB colors to use for different methods shown in the combination plot. Defaults to Ax discrete color scale.
  • optimum_color (Tuple[int]) – tuple of 3 int values representing an RGB color. Defaults to orange.
Returns:

plot of the comparison of optimization traces with IQR

Return type:

AxPlotConfig

ax.plot.trace.optimization_trace_single_method(y, optimum=None, generator_changes=None, title='', ylabel='', trace_color=(128, 177, 211), optimum_color=(253, 180, 98), generator_change_color=(141, 211, 199))[source]

Plots an optimization trace with mean and 2 SEMs

Parameters:
  • y (ndarray) – (r x t) array; result to plot, with r runs and t trials
  • optimum (Optional[float]) – value of the optimal objective
  • generator_changes (Optional[List[int]]) – iterations, before which generators changed
  • title (str) – title of this plot
  • ylabel (str) – Label for y axis
  • trace_color (Tuple[int]) – tuple of 3 int values representing an RGB color. Defaults to orange.
  • optimum_color (Tuple[int]) – tuple of 3 int values representing an RGB color. Defaults to orange.
  • generator_change_color (Tuple[int]) – tuple of 3 int values representing an RGB color. Defaults to orange.
Returns:

plot of the optimization trace with IQR

Return type:

AxPlotConfig

ax.plot.trace.optimum_objective_scatter(optimum, num_iterations, optimum_color=(253, 180, 98))[source]

Creates a graph object for the line representing optimal objective.

Parameters:
  • optimum (float) – value of the optimal objective
  • num_iterations (int) – how many trials were in the optimization (used to determine the width of the plot)
  • trace_color – tuple of 3 int values representing an RGB color. Defaults to orange.
Returns:

plotly graph objects for the optimal objective line

Return type:

go.Scatter

ax.plot.trace.sem_range_scatter(y, trace_color=(128, 177, 211), legend_label='')[source]

Creates a graph object for trace of mean +/- 2 SEMs for y, across runs.

Parameters:
  • y (ndarray) – (r x t) array with results from r runs and t trials.
  • trace_color (Tuple[int]) – tuple of 3 int values representing an RGB color. Defaults to blue.
  • legend_label (str) – Label for the legend group.
Returns:

plotly graph objects for lower and upper bounds

Return type:

Tuple[go.Scatter]