ax.plot

Plots

Base

class ax.plot.base.AxPlotConfig[source]

Bases: ax.plot.base._AxPlotConfigBase

Config for plots

class ax.plot.base.AxPlotTypes[source]

Bases: enum.Enum

Enum of Ax plot types.

BANDIT_ROLLOUT = 4
CONTOUR = 0
GENERIC = 1
INTERACT_CONTOUR = 3
INTERACT_SLICE = 5
SLICE = 2
class ax.plot.base.PlotData[source]

Bases: tuple

Struct for plot data, including both in-sample and out-of-sample arms

property in_sample

Alias for field number 1

property metrics

Alias for field number 0

property out_of_sample

Alias for field number 2

property status_quo_name

Alias for field number 3

class ax.plot.base.PlotInSampleArm[source]

Bases: tuple

Struct for in-sample arms (both observed and predicted data)

property context_stratum

Alias for field number 6

property name

Alias for field number 0

property parameters

Alias for field number 1

property se

Alias for field number 4

property se_hat

Alias for field number 5

property y

Alias for field number 2

property y_hat

Alias for field number 3

class ax.plot.base.PlotMetric[source]

Bases: tuple

Struct for metric

property metric

Alias for field number 0

property pred

Alias for field number 1

class ax.plot.base.PlotOutOfSampleArm[source]

Bases: tuple

Struct for out-of-sample arms (only predicted data)

property context_stratum

Alias for field number 4

property name

Alias for field number 0

property parameters

Alias for field number 1

property se_hat

Alias for field number 3

property y_hat

Alias for field number 2

Color

class ax.plot.color.COLORS[source]

Bases: enum.Enum

An enumeration.

CORAL = (251, 128, 114)
LIGHT_PURPLE = (190, 186, 218)
ORANGE = (253, 180, 98)
PINK = (188, 128, 189)
STEELBLUE = (128, 177, 211)
TEAL = (141, 211, 199)
ax.plot.color.plotly_color_scale(list_of_rgb_tuples, reverse=False, alpha=1)[source]

Convert list of RGB tuples to list of tuples, where each tuple is break in [0, 1] and stringified RGBA color.

Return type

List[Tuple[float, str]]

ax.plot.color.rgba(rgb_tuple, alpha=1)[source]

Convert RGB tuple to an RGBA string.

Return type

str

Contour Plot

ax.plot.contour.interact_contour(model, metric_name, generator_runs_dict=None, relative=False, density=50, slice_values=None, lower_is_better=False, fixed_features=None, trial_index=None)[source]

Create interactive plot with predictions for a 2-d slice of the parameter space.

Parameters

  • model (ModelBridge) – ModelBridge that contains model for predictions

  • metric_name (str) – Name of metric to plot

  • generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.

  • relative (bool) – Predictions relative to status quo

  • density (int) – Number of points along slice to evaluate predictions.

  • slice_values (Optional[Dict[str, Any]]) – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters.

  • lower_is_better (bool) – Lower values for metric are better.

  • fixed_features (Optional[ObservationFeatures]) – An ObservationFeatures object containing the values of features (including non-parameter features like context) to be set in the slice.

Return type

AxPlotConfig

ax.plot.contour.plot_contour(model, param_x, param_y, metric_name, generator_runs_dict=None, relative=False, density=50, slice_values=None, lower_is_better=False, fixed_features=None, trial_index=None)[source]

Plot predictions for a 2-d slice of the parameter space.

Parameters

  • model (ModelBridge) – ModelBridge that contains model for predictions

  • param_x (str) – Name of parameter that will be sliced on x-axis

  • param_y (str) – Name of parameter that will be sliced on y-axis

  • metric_name (str) – Name of metric to plot

  • generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.

  • relative (bool) – Predictions relative to status quo

  • density (int) – Number of points along slice to evaluate predictions.

  • slice_values (Optional[Dict[str, Any]]) – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters.

  • lower_is_better (bool) – Lower values for metric are better.

  • fixed_features (Optional[ObservationFeatures]) – An ObservationFeatures object containing the values of features (including non-parameter features like context) to be set in the slice.

Return type

AxPlotConfig

ax.plot.contour.short_name(param_name)[source]
Return type

str

Model Diagnostic Plot

ax.plot.diagnostic.interact_batch_comparison(observations, experiment, batch_x, batch_y, rel=False, status_quo_name=None)[source]

Compare repeated arms from two trials; select metric via dropdown.

Parameters

  • observations (List[Observation]) – List of observations to compute comparison.

  • batch_x (int) – Index of batch for x-axis.

  • batch_y (int) – Index of bach for y-axis.

  • rel (bool) – Whether to relativize data against status_quo arm.

  • status_quo_name (Optional[str]) – Name of the status_quo arm.

Return type

AxPlotConfig

ax.plot.diagnostic.interact_cross_validation(cv_results, show_context=True)[source]

Interactive cross-validation (CV) plotting; select metric via dropdown.

Note: uses the Plotly version of dropdown (which means that all data is stored within the notebook).

Parameters

  • cv_results (List[CVResult]) – cross-validation results.

  • show_context (bool) – if True, show context on hover.

Return type

AxPlotConfig

ax.plot.diagnostic.interact_empirical_model_validation(batch, data)[source]

Compare the model predictions for the batch arms against observed data.

Relies on the model predictions stored on the generator_runs of batch.

Parameters

  • batch (BatchTrial) – Batch on which to perform analysis.

  • data (Data) – Observed data for the batch.

Return type

AxPlotConfig

Returns

AxPlotConfig for the plot.

ax.plot.diagnostic.tile_cross_validation(cv_results, show_arm_details_on_hover=True, show_context=True)[source]

Tile version of CV plots; sorted by ‘best fitting’ outcomes.

Plots are sorted in decreasing order using the p-value of a Fisher exact test statistic.

Parameters

  • cv_results (List[CVResult]) – cross-validation results.

  • include_measurement_error – if True, include measurement_error metrics in plot.

  • show_arm_details_on_hover (bool) – if True, display parameterizations of arms on hover. Default is True.

  • show_context (bool) – if True (default), display context on hover.

Return type

AxPlotConfig

Helpers

ax.plot.helper.arm_name_to_tuple(arm_name)[source]
Return type

Union[Tuple[int, int], Tuple[int]]

ax.plot.helper.axis_range(grid, is_log)[source]
Return type

List[float]

ax.plot.helper.contour_config_to_trace(config)[source]
ax.plot.helper.get_fixed_values(model, slice_values=None, trial_index=None)[source]

Get fixed values for parameters in a slice plot.

If there is an in-design status quo, those values will be used. Otherwise, the mean of RangeParameters or the mode of ChoiceParameters is used.

Any value in slice_values will override the above.

Parameters

  • model (ModelBridge) – ModelBridge being used for plotting

  • slice_values (Optional[Dict[str, Any]]) – Map from parameter name to value at which is should be fixed.

Returns: Map from parameter name to fixed value.

Return type

Dict[str, Union[str, float, int, None]]

ax.plot.helper.get_grid_for_parameter(parameter, density)[source]

Get a grid of points along the range of the parameter.

Will be a log-scale grid if parameter is log scale.

Parameters

  • parameter (RangeParameter) – Parameter for which to generate grid.

  • density (int) – Number of points in the grid.

Return type

ndarray

ax.plot.helper.get_plot_data(model, generator_runs_dict, metric_names=None, fixed_features=None)[source]

Format data object with metrics for in-sample and out-of-sample arms.

Calculate both observed and predicted metrics for in-sample arms. Calculate predicted metrics for out-of-sample arms passed via the generator_runs_dict argument.

In PlotData, in-sample observations are merged with IVW. In RawData, they are left un-merged and given as a list of dictionaries, one for each observation and having keys ‘arm_name’, ‘mean’, and ‘sem’.

Parameters

  • model (ModelBridge) – The model.

  • generator_runs_dict (Dict[str, GeneratorRun]) – a mapping from generator run name to generator run.

  • metric_names (Optional[Set[str]]) – Restrict predictions to this set. If None, all metrics in the model will be returned.

  • fixed_features (Optional[ObservationFeatures]) – Fixed features to use when making model predictions.

Return type

Tuple[PlotData, List[Dict[str, Union[str, float]]], Dict[str, Dict[str, Union[str, float, int, None]]]]

Returns

A tuple containing

  • PlotData object with in-sample and out-of-sample predictions.

  • List of observations like:

    {'metric_name': 'likes', 'arm_name': '0_1', 'mean': 1., 'sem': 0.1}.

  • Mapping from arm name to parameters.

ax.plot.helper.get_range_parameter(model, param_name)[source]

Get the range parameter with the given name from the model.

Throws if parameter doesn’t exist or is not a range parameter.

Parameters

  • model (ModelBridge) – The model.

  • param_name (str) – The name of the RangeParameter to be found.

Returns: The RangeParameter named param_name.

Return type

RangeParameter

ax.plot.helper.get_range_parameters(model)[source]

Get a list of range parameters from a model.

Parameters

model (ModelBridge) – The model.

Returns: List of RangeParameters.

Return type

List[RangeParameter]

ax.plot.helper.relativize_data(f, sd, rel, arm_data, metric)[source]
Return type

List[List[float]]

ax.plot.helper.resize_subtitles(figure, size)[source]
ax.plot.helper.rgb(arr)[source]
Return type

str

ax.plot.helper.slice_config_to_trace(arm_data, arm_name_to_parameters, f, fit_data, grid, metric, param, rel, setx, sd, is_log, visible)[source]

Rendering

ax.plot.render.plot_config_to_html(plot_config, plot_module_name='ax.plot', plot_resources={<AxPlotTypes.GENERIC: 1>: 'generic_plotly.js'}, inject_helpers=False)[source]

Generate HTML + JS corresponding from a plot config.

Return type

str

Scatter Plots

ax.plot.scatter.interact_fitted(model, generator_runs_dict=None, rel=True, show_arm_details_on_hover=True, show_CI=True, arm_noun='arm', metrics=None, fixed_features=None)[source]

Interactive fitted outcome plots for each arm used in fitting the model.

Choose the outcome to plot using a dropdown.

Parameters

  • model (ModelBridge) – model to use for predictions.

  • generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – a mapping from generator run name to generator run.

  • rel (bool) – if True, use relative effects. Default is True.

  • show_arm_details_on_hover (bool) – if True, display parameterizations of arms on hover. Default is True.

  • show_CI (bool) – if True, render confidence intervals.

  • arm_noun (str) – noun to use instead of “arm” (e.g. group)

  • metrics (Optional[List[str]]) – List of metric names to restrict to when plotting.

  • fixed_features (Optional[ObservationFeatures]) – Fixed features to use when making model predictions.

Return type

AxPlotConfig

ax.plot.scatter.lattice_multiple_metrics(model, generator_runs_dict=None, rel=True, show_arm_details_on_hover=False)[source]

Plot raw values or predictions of combinations of two metrics for arms.

Parameters

  • model (ModelBridge) – model to draw predictions from.

  • generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – a mapping from generator run name to generator run.

  • rel (bool) – if True, use relative effects. Default is True.

  • show_arm_details_on_hover (bool) – if True, display parameterizations of arms on hover. Default is False.

Return type

AxPlotConfig

ax.plot.scatter.plot_fitted(model, metric, generator_runs_dict=None, rel=True, custom_arm_order=None, custom_arm_order_name='Custom', show_CI=True)[source]

Plot fitted metrics.

Parameters

  • model (ModelBridge) – model to use for predictions.

  • metric (str) – metric to plot predictions for.

  • generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – a mapping from generator run name to generator run.

  • rel (bool) – if True, use relative effects. Default is True.

  • custom_arm_order (Optional[List[str]]) – a list of arm names in the order corresponding to how they should be plotted on the x-axis. If not None, this is the default ordering.

  • custom_arm_order_name (str) – name for custom ordering to show in the ordering dropdown. Default is ‘Custom’.

  • show_CI (bool) – if True, render confidence intervals.

Return type

AxPlotConfig

ax.plot.scatter.plot_multiple_metrics(model, metric_x, metric_y, generator_runs_dict=None, rel=True)[source]

Plot raw values or predictions of two metrics for arms.

All arms used in the model are included in the plot. Additional arms can be passed through the generator_runs_dict argument.

Parameters

  • model (ModelBridge) – model to draw predictions from.

  • metric_x (str) – metric to plot on the x-axis.

  • metric_y (str) – metric to plot on the y-axis.

  • generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – a mapping from generator run name to generator run.

  • rel (bool) – if True, use relative effects. Default is True.

Return type

AxPlotConfig

ax.plot.scatter.plot_objective_vs_constraints(model, objective, subset_metrics=None, generator_runs_dict=None, rel=True, fixed_features=None)[source]

Plot the tradeoff between an objetive and all other metrics in a model.

All arms used in the model are included in the plot. Additional arms can be passed through via the generator_runs_dict argument.

Fixed features input can be used to override fields of the insample arms when making model predictions.

Parameters

  • model (ModelBridge) – model to draw predictions from.

  • objective (str) – metric to optimize. Plotted on the x-axis.

  • subset_metrics (Optional[List[str]]) – list of metrics to plot on the y-axes if need a subset of all metrics in the model.

  • generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – a mapping from generator run name to generator run.

  • rel (bool) – if True, use relative effects. Default is True.

  • fixed_features (Optional[ObservationFeatures]) – Fixed features to use when making model predictions.

Return type

AxPlotConfig

ax.plot.scatter.tile_fitted(model, generator_runs_dict=None, rel=True, show_arm_details_on_hover=False, show_CI=True, arm_noun='arm', metrics=None, fixed_features=None)[source]

Tile version of fitted outcome plots.

Parameters

  • model (ModelBridge) – model to use for predictions.

  • generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – a mapping from generator run name to generator run.

  • rel (bool) – if True, use relative effects. Default is True.

  • show_arm_details_on_hover (bool) – if True, display parameterizations of arms on hover. Default is False.

  • show_CI (bool) – if True, render confidence intervals.

  • arm_noun (str) – noun to use instead of “arm” (e.g. group)

  • metrics (Optional[List[str]]) – List of metric names to restrict to when plotting.

  • fixed_features (Optional[ObservationFeatures]) – Fixed features to use when making model predictions.

Return type

AxPlotConfig

Slice Plot

ax.plot.slice.interact_slice(model, param_name, metric_name='', generator_runs_dict=None, relative=False, density=50, slice_values=None, fixed_features=None, trial_index=None)[source]

Create interactive plot with predictions for a 1-d slice of the parameter space.

Parameters

  • model (ModelBridge) – ModelBridge that contains model for predictions

  • param_name (str) – Name of parameter that will be sliced

  • metric_name (str) – Name of metric to plot

  • generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.

  • relative (bool) – Predictions relative to status quo

  • density (int) – Number of points along slice to evaluate predictions.

  • slice_values (Optional[Dict[str, Any]]) – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters. Ignored if fixed_features is specified.

  • fixed_features (Optional[ObservationFeatures]) – An ObservationFeatures object containing the values of features (including non-parameter features like context) to be set in the slice.

Return type

AxPlotConfig

ax.plot.slice.plot_slice(model, param_name, metric_name, generator_runs_dict=None, relative=False, density=50, slice_values=None, fixed_features=None, trial_index=None)[source]

Plot predictions for a 1-d slice of the parameter space.

Parameters

  • model (ModelBridge) – ModelBridge that contains model for predictions

  • param_name (str) – Name of parameter that will be sliced

  • metric_name (str) – Name of metric to plot

  • generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.

  • relative (bool) – Predictions relative to status quo

  • density (int) – Number of points along slice to evaluate predictions.

  • slice_values (Optional[Dict[str, Any]]) – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters. Ignored if fixed_features is specified.

  • fixed_features (Optional[ObservationFeatures]) – An ObservationFeatures object containing the values of features (including non-parameter features like context) to be set in the slice.

Return type

AxPlotConfig

Table

ax.plot.table_view.get_color(x, ci, rel, reverse)[source]

Determine the color of the table cell.

ax.plot.table_view.table_view_plot(experiment, data, use_empirical_bayes=True, only_data_frame=False, arm_noun='arm')[source]

Table of means and confidence intervals.

Table is of the form:

arm

metric_1

metric_2

0_0

mean +- CI

…

0_1

…

…

Trace Plots

ax.plot.trace.generator_changes_scatter(generator_changes, y_range, generator_change_color=(141, 211, 199))[source]

Creates a graph object for the line(s) representing generator changes.

Parameters

  • generator_changes (List[int]) – iterations, before which generators changed

  • y_range (List[float]) – upper and lower values of the y-range of the plot

  • generator_change_color (Tuple[int]) – tuple of 3 int values representing an RGB color. Defaults to orange.

Returns

plotly graph objects for the lines representing generator

changes

Return type

go.Scatter

ax.plot.trace.mean_trace_scatter(y, trace_color=(128, 177, 211), legend_label='mean', hover_labels=None)[source]

Creates a graph object for trace of the mean of the given series across runs.

Parameters

  • y (ndarray) – (r x t) array with results from r runs and t trials.

  • trace_color (Tuple[int]) – tuple of 3 int values representing an RGB color. Defaults to blue.

  • legend_label (str) – label for this trace.

  • hover_labels (Optional[List[str]]) – optional, text to show on hover; list where the i-th value corresponds to the i-th value in the value of the y argument.

Returns

plotly graph object

Return type

go.Scatter

ax.plot.trace.optimization_times(fit_times, gen_times, title='')[source]

Plots wall times for each method as a bar chart.

Parameters

  • fit_times (Dict[str, List[float]]) – A map from method name to a list of the model fitting times.

  • gen_times (Dict[str, List[float]]) – A map from method name to a list of the gen times.

  • title (str) – Title for this plot.

Returns: AxPlotConfig with the plot

Return type

AxPlotConfig

ax.plot.trace.optimization_trace_all_methods(y_dict, optimum=None, title='', ylabel='', hover_labels=None, trace_colors=[(128, 177, 211), (251, 128, 114), (141, 211, 199), (188, 128, 189), (190, 186, 218), (253, 180, 98)], optimum_color=(253, 180, 98))[source]

Plots a comparison of optimization traces with 2-SEM bands for multiple methods on the same problem.

Parameters

  • y – a mapping of method names to (r x t) arrays, where r is the number of runs in the test, and t is the number of trials.

  • optimum (Optional[float]) – value of the optimal objective.

  • title (str) – title for this plot.

  • ylabel (str) – label for the Y-axis.

  • hover_labels (Optional[List[str]]) – optional, text to show on hover; list where the i-th value corresponds to the i-th value in the value of the y argument.

  • trace_colors (List[Tuple[int]]) – tuples of 3 int values representing RGB colors to use for different methods shown in the combination plot. Defaults to Ax discrete color scale.

  • optimum_color (Tuple[int]) – tuple of 3 int values representing an RGB color. Defaults to orange.

Returns

plot of the comparison of optimization traces with IQR

Return type

AxPlotConfig

ax.plot.trace.optimization_trace_single_method(y, optimum=None, generator_changes=None, title='', ylabel='', hover_labels=None, trace_color=(128, 177, 211), optimum_color=(253, 180, 98), generator_change_color=(141, 211, 199))[source]

Plots an optimization trace with mean and 2 SEMs

Parameters

  • y (ndarray) – (r x t) array; result to plot, with r runs and t trials

  • optimum (Optional[float]) – value of the optimal objective

  • generator_changes (Optional[List[int]]) – iterations, before which generators changed

  • title (str) – title for this plot.

  • ylabel (str) – label for the Y-axis.

  • hover_labels (Optional[List[str]]) – optional, text to show on hover; list where the i-th value corresponds to the i-th value in the value of the y argument.

  • trace_color (Tuple[int]) – tuple of 3 int values representing an RGB color. Defaults to orange.

  • optimum_color (Tuple[int]) – tuple of 3 int values representing an RGB color. Defaults to orange.

  • generator_change_color (Tuple[int]) – tuple of 3 int values representing an RGB color. Defaults to orange.

Returns

plot of the optimization trace with IQR

Return type

AxPlotConfig

ax.plot.trace.optimum_objective_scatter(optimum, num_iterations, optimum_color=(253, 180, 98))[source]

Creates a graph object for the line representing optimal objective.

Parameters

  • optimum (float) – value of the optimal objective

  • num_iterations (int) – how many trials were in the optimization (used to determine the width of the plot)

  • trace_color – tuple of 3 int values representing an RGB color. Defaults to orange.

Returns

plotly graph objects for the optimal objective line

Return type

go.Scatter

ax.plot.trace.sem_range_scatter(y, trace_color=(128, 177, 211), legend_label='')[source]

Creates a graph object for trace of mean +/- 2 SEMs for y, across runs.

Parameters

  • y (ndarray) – (r x t) array with results from r runs and t trials.

  • trace_color (Tuple[int]) – tuple of 3 int values representing an RGB color. Defaults to blue.

  • legend_label (str) – Label for the legend group.

Returns

plotly graph objects for lower and upper bounds

Return type

Tuple[go.Scatter]