ax.plot

Rendering

ax.plot.render.plot_config_to_html(plot_config: AxPlotConfig, plot_module_name: str = 'ax.plot', plot_resources: Dict[Enum, str] = {AxPlotTypes.GENERIC: 'generic_plotly.js'}, inject_helpers: bool = False) str[source]

Generate HTML + JS corresponding from a plot config.

Plots

Base

class ax.plot.base.AxPlotConfig(data: Dict[str, Any], plot_type: Enum)[source]

Bases: _AxPlotConfigBase

Config for plots

class ax.plot.base.AxPlotTypes(value)[source]

Bases: Enum

Enum of Ax plot types.

BANDIT_ROLLOUT = 4
CONTOUR = 0
GENERIC = 1
HTML = 6
INTERACT_CONTOUR = 3
INTERACT_SLICE = 5
SLICE = 2
class ax.plot.base.PlotData(metrics: List[str], in_sample: Dict[str, PlotInSampleArm], out_of_sample: Optional[Dict[str, Dict[str, PlotOutOfSampleArm]]], status_quo_name: Optional[str])[source]

Bases: NamedTuple

Struct for plot data, including both in-sample and out-of-sample arms

in_sample: Dict[str, PlotInSampleArm]

Alias for field number 1

metrics: List[str]

Alias for field number 0

out_of_sample: Optional[Dict[str, Dict[str, PlotOutOfSampleArm]]]

Alias for field number 2

status_quo_name: Optional[str]

Alias for field number 3

class ax.plot.base.PlotInSampleArm(name: str, parameters: Dict[str, Union[None, str, bool, float, int]], y: Dict[str, float], y_hat: Dict[str, float], se: Dict[str, float], se_hat: Dict[str, float], context_stratum: Optional[Dict[str, Union[str, float]]])[source]

Bases: NamedTuple

Struct for in-sample arms (both observed and predicted data)

context_stratum: Optional[Dict[str, Union[str, float]]]

Alias for field number 6

name: str

Alias for field number 0

parameters: Dict[str, Union[None, str, bool, float, int]]

Alias for field number 1

se: Dict[str, float]

Alias for field number 4

se_hat: Dict[str, float]

Alias for field number 5

y: Dict[str, float]

Alias for field number 2

y_hat: Dict[str, float]

Alias for field number 3

class ax.plot.base.PlotMetric(metric: str, pred: bool, rel: bool)[source]

Bases: NamedTuple

Struct for metric

metric: str

Alias for field number 0

pred: bool

Alias for field number 1

rel: bool

Alias for field number 2

class ax.plot.base.PlotOutOfSampleArm(name: str, parameters: Dict[str, Union[None, str, bool, float, int]], y_hat: Dict[str, float], se_hat: Dict[str, float], context_stratum: Optional[Dict[str, Union[str, float]]])[source]

Bases: NamedTuple

Struct for out-of-sample arms (only predicted data)

context_stratum: Optional[Dict[str, Union[str, float]]]

Alias for field number 4

name: str

Alias for field number 0

parameters: Dict[str, Union[None, str, bool, float, int]]

Alias for field number 1

se_hat: Dict[str, float]

Alias for field number 3

y_hat: Dict[str, float]

Alias for field number 2

Bandit Rollout

ax.plot.bandit_rollout.plot_bandit_rollout(experiment: Experiment) AxPlotConfig[source]

Plot bandit rollout from ane experiement.

Benchmark

Contour Plot

ax.plot.contour.interact_contour(model: ModelBridge, metric_name: str, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, relative: bool = False, density: int = 50, slice_values: Optional[Dict[str, Any]] = None, lower_is_better: bool = False, fixed_features: Optional[ObservationFeatures] = None, trial_index: Optional[int] = None) AxPlotConfig[source]

Create interactive plot with predictions for a 2-d slice of the parameter space.

Parameters:
  • model – ModelBridge that contains model for predictions

  • metric_name – Name of metric to plot

  • generator_runs_dict – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.

  • relative – Predictions relative to status quo

  • density – Number of points along slice to evaluate predictions.

  • slice_values – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters.

  • lower_is_better – Lower values for metric are better.

  • fixed_features – An ObservationFeatures object containing the values of features (including non-parameter features like context) to be set in the slice.

Returns:

interactive plot of objective vs. parameters

Return type:

AxPlotConfig

ax.plot.contour.interact_contour_plotly(model: ModelBridge, metric_name: str, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, relative: bool = False, density: int = 50, slice_values: Optional[Dict[str, Any]] = None, lower_is_better: bool = False, fixed_features: Optional[ObservationFeatures] = None, trial_index: Optional[int] = None) Figure[source]

Create interactive plot with predictions for a 2-d slice of the parameter space.

Parameters:
  • model – ModelBridge that contains model for predictions

  • metric_name – Name of metric to plot

  • generator_runs_dict – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.

  • relative – Predictions relative to status quo

  • density – Number of points along slice to evaluate predictions.

  • slice_values – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters.

  • lower_is_better – Lower values for metric are better.

  • fixed_features – An ObservationFeatures object containing the values of features (including non-parameter features like context) to be set in the slice.

Returns:

interactive plot of objective vs. parameters

Return type:

go.Figure

ax.plot.contour.plot_contour(model: ModelBridge, param_x: str, param_y: str, metric_name: str, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, relative: bool = False, density: int = 50, slice_values: Optional[Dict[str, Any]] = None, lower_is_better: bool = False, fixed_features: Optional[ObservationFeatures] = None, trial_index: Optional[int] = None) AxPlotConfig[source]

Plot predictions for a 2-d slice of the parameter space.

Parameters:
  • model – ModelBridge that contains model for predictions

  • param_x – Name of parameter that will be sliced on x-axis

  • param_y – Name of parameter that will be sliced on y-axis

  • metric_name – Name of metric to plot

  • generator_runs_dict – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.

  • relative – Predictions relative to status quo

  • density – Number of points along slice to evaluate predictions.

  • slice_values – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters.

  • lower_is_better – Lower values for metric are better.

  • fixed_features – An ObservationFeatures object containing the values of features (including non-parameter features like context) to be set in the slice.

Returns:

contour plot of objective vs. parameter values

Return type:

AxPlotConfig

ax.plot.contour.plot_contour_plotly(model: ModelBridge, param_x: str, param_y: str, metric_name: str, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, relative: bool = False, density: int = 50, slice_values: Optional[Dict[str, Any]] = None, lower_is_better: bool = False, fixed_features: Optional[ObservationFeatures] = None, trial_index: Optional[int] = None) Figure[source]

Plot predictions for a 2-d slice of the parameter space.

Parameters:
  • model – ModelBridge that contains model for predictions

  • param_x – Name of parameter that will be sliced on x-axis

  • param_y – Name of parameter that will be sliced on y-axis

  • metric_name – Name of metric to plot

  • generator_runs_dict – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.

  • relative – Predictions relative to status quo

  • density – Number of points along slice to evaluate predictions.

  • slice_values – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters.

  • lower_is_better – Lower values for metric are better.

  • fixed_features – An ObservationFeatures object containing the values of features (including non-parameter features like context) to be set in the slice.

Returns:

contour plot of objective vs. parameter values

Return type:

go.Figure

ax.plot.contour.short_name(param_name: str) str[source]

Feature Importances

ax.plot.feature_importances.plot_feature_importance(df: pandas.DataFrame, title: str) AxPlotConfig[source]

Wrapper method to convert plot_feature_importance_plotly to AxPlotConfig

ax.plot.feature_importances.plot_feature_importance_by_feature(model: Optional[ModelBridge] = None, sensitivity_values: Optional[Dict[str, Dict[str, Union[float, ndarray]]]] = None, relative: bool = False, caption: str = '', importance_measure: str = '', label_dict: Optional[Dict[str, str]] = None) AxPlotConfig[source]

Wrapper method to convert plot_feature_importance_by_feature_plotly to AxPlotConfig

ax.plot.feature_importances.plot_feature_importance_by_feature_plotly(model: Optional[ModelBridge] = None, sensitivity_values: Optional[Dict[str, Dict[str, Union[float, ndarray]]]] = None, relative: bool = False, caption: str = '', importance_measure: str = '', label_dict: Optional[Dict[str, str]] = None) Figure[source]

One plot per metric, showing importances by feature.

If sensitivity values are not all positive, the absolute value will be shown and color will indicate positive or negative sign.

Parameters:
  • model – A model with a feature_importances method.

  • sensitivity_values – The sensitivity values for each metric in a dict format. It takes the following format if only the sensitivity value is plotted: {“metric1”:{“parameter1”:value1,”parameter2”:value2 …} …} It takes the following format if the sensitivity value and standard error are plotted: {“metric1”:{“parameter1”:[value1,var,se], “parameter2”:[[value2,var,se]]…}…}}.

  • relative – Whether to normalize feature importances so that they add to 1.

  • caption – An HTML-formatted string to place at the bottom of the plot.

  • importance_measure – The name of the importance metric to be added to the title.

  • label_dict – A dictionary mapping metric names to short labels.

Returns a go.Figure of feature importances.

ax.plot.feature_importances.plot_feature_importance_by_metric(model: ModelBridge) AxPlotConfig[source]

Wrapper method to convert plot_feature_importance_by_metric_plotly to AxPlotConfig

ax.plot.feature_importances.plot_feature_importance_by_metric_plotly(model: ModelBridge) Figure[source]

One plot per feature, showing importances by metric.

ax.plot.feature_importances.plot_feature_importance_plotly(df: pandas.DataFrame, title: str) Figure[source]
ax.plot.feature_importances.plot_relative_feature_importance(model: ModelBridge) AxPlotConfig[source]

Wrapper method to convert plot_relative_feature_importance_plotly to AxPlotConfig

ax.plot.feature_importances.plot_relative_feature_importance_plotly(model: ModelBridge) Figure[source]

Create a stacked bar chart of feature importances per metric

Marginal Effects

ax.plot.marginal_effects.plot_marginal_effects(model: ModelBridge, metric: str) AxPlotConfig[source]

Calculates and plots the marginal effects – the effect of changing one factor away from the randomized distribution of the experiment and fixing it at a particular level.

Parameters:
  • model – Model to use for estimating effects

  • metric – The metric for which to plot marginal effects.

Returns:

AxPlotConfig of the marginal effects

Model Diagnostics

ax.plot.diagnostic.interact_batch_comparison(observations: List[Observation], experiment: Experiment, batch_x: int, batch_y: int, rel: bool = False, status_quo_name: Optional[str] = None, x_label: Optional[str] = None, y_label: Optional[str] = None) AxPlotConfig[source]

Compare repeated arms from two trials; select metric via dropdown.

Parameters:
  • observations – List of observations to compute comparison.

  • batch_x – Index of batch for x-axis.

  • batch_y – Index of bach for y-axis.

  • rel – Whether to relativize data against status_quo arm.

  • status_quo_name – Name of the status_quo arm.

  • x_label – Label for the x-axis.

  • y_label – Label for the y-axis.

ax.plot.diagnostic.interact_cross_validation(cv_results: List[CVResult], show_context: bool = True, caption: str = '', label_dict: Optional[Dict[str, str]] = None, autoset_axis_limits: bool = True) AxPlotConfig[source]

Interactive cross-validation (CV) plotting; select metric via dropdown.

Note: uses the Plotly version of dropdown (which means that all data is stored within the notebook).

Parameters:
  • cv_results – cross-validation results.

  • show_context – if True, show context on hover.

  • label_dict – optional map from real metric names to shortened names

  • autoset_axis_limits – Automatically try to set the limit for each axis to focus on the region of interest.

Returns an AxPlotConfig

ax.plot.diagnostic.interact_cross_validation_plotly(cv_results: List[CVResult], show_context: bool = True, caption: str = '', label_dict: Optional[Dict[str, str]] = None, autoset_axis_limits: bool = True) Figure[source]

Interactive cross-validation (CV) plotting; select metric via dropdown.

Note: uses the Plotly version of dropdown (which means that all data is stored within the notebook).

Parameters:
  • cv_results – cross-validation results.

  • show_context – if True, show context on hover.

  • label_dict – optional map from real metric names to shortened names

  • autoset_axis_limits – Automatically try to set the limit for each axis to focus on the region of interest.

Returns a plotly.graph_objects.Figure

ax.plot.diagnostic.interact_empirical_model_validation(batch: BatchTrial, data: Data) AxPlotConfig[source]

Compare the model predictions for the batch arms against observed data.

Relies on the model predictions stored on the generator_runs of batch.

Parameters:
  • batch – Batch on which to perform analysis.

  • data – Observed data for the batch.

Returns:

AxPlotConfig for the plot.

ax.plot.diagnostic.tile_cross_validation(cv_results: List[CVResult], show_arm_details_on_hover: bool = True, show_context: bool = True, label_dict: Optional[Dict[str, str]] = None) AxPlotConfig[source]

Tile version of CV plots; sorted by ‘best fitting’ outcomes.

Plots are sorted in decreasing order using the p-value of a Fisher exact test statistic.

Parameters:
  • cv_results – cross-validation results.

  • include_measurement_error – if True, include measurement_error metrics in plot.

  • show_arm_details_on_hover – if True, display parameterizations of arms on hover. Default is True.

  • show_context – if True (default), display context on hover.

  • label_dict – optional map from real metric names to shortened names

Returns a plotly.graph_objects.Figure

Pareto Plots

class ax.plot.pareto_utils.ParetoFrontierResults(param_dicts: List[Dict[str, Union[None, str, bool, float, int]]], means: Dict[str, List[float]], sems: Dict[str, List[float]], primary_metric: str, secondary_metric: str, absolute_metrics: List[str], objective_thresholds: Optional[Dict[str, float]], arm_names: Optional[List[Optional[str]]])[source]

Bases: NamedTuple

Container for results from Pareto frontier computation.

Fields are: - param_dicts: The parameter dicts of the points generated on the Pareto Frontier. - means: The posterior mean predictions of the model for each metric (same order as the param dicts). These must be as a percent change relative to status quo for any metric not listed in absolute_metrics. - sems: The posterior sem predictions of the model for each metric (same order as the param dicts). Also must be relativized wrt status quo for any metric not listed in absolute_metrics. - primary_metric: The name of the primary metric. - secondary_metric: The name of the secondary metric. - absolute_metrics: List of outcome metrics that are NOT be relativized w.r.t. the status quo. All other metrics are assumed to be given here as % relative to status_quo. - objective_thresholds: Threshold for each objective. Must be on the same scale as means, so if means is relativized it should be the relative value, otherwise it should be absolute. - arm_names: Optional list of arm names for each parameterization.

absolute_metrics: List[str]

Alias for field number 5

arm_names: Optional[List[Optional[str]]]

Alias for field number 7

means: Dict[str, List[float]]

Alias for field number 1

objective_thresholds: Optional[Dict[str, float]]

Alias for field number 6

param_dicts: List[Dict[str, Union[None, str, bool, float, int]]]

Alias for field number 0

primary_metric: str

Alias for field number 3

secondary_metric: str

Alias for field number 4

sems: Dict[str, List[float]]

Alias for field number 2

ax.plot.pareto_utils.compute_posterior_pareto_frontier(experiment: Experiment, primary_objective: Metric, secondary_objective: Metric, data: Optional[Data] = None, outcome_constraints: Optional[List[OutcomeConstraint]] = None, absolute_metrics: Optional[List[str]] = None, num_points: int = 10, trial_index: Optional[int] = None, chebyshev: bool = True) ParetoFrontierResults[source]

Compute the Pareto frontier between two objectives. For experiments with batch trials, a trial index or data object must be provided.

This is done by fitting a GP and finding the pareto front according to the GP posterior mean.

Parameters:
  • experiment – The experiment to compute a pareto frontier for.

  • primary_objective – The primary objective to optimize.

  • secondary_objective – The secondary objective against which to trade off the primary objective.

  • outcome_constraints – Outcome constraints to be respected by the optimization. Can only contain constraints on metrics that are not primary or secondary objectives.

  • absolute_metrics – List of outcome metrics that should NOT be relativized w.r.t. the status quo (all other outcomes will be in % relative to status_quo).

  • num_points – The number of points to compute on the Pareto frontier.

  • chebyshev – Whether to use augmented_chebyshev_scalarization when computing Pareto Frontier points.

Returns:

A NamedTuple with fields listed in its definition.

Return type:

ParetoFrontierResults

ax.plot.pareto_utils.get_observed_pareto_frontiers(experiment: Experiment, data: Optional[Data] = None, rel: Optional[bool] = None, arm_names: Optional[List[str]] = None) List[ParetoFrontierResults][source]

Find all Pareto points from an experiment.

Uses only values as observed in the data; no modeling is involved. Makes no assumption about the search space or types of parameters. If “data” is provided will use that, otherwise will use all data already attached to the experiment.

Uses all arms present in data; does not filter according to experiment search space. If arm_names is specified, will filter to just those arm whose names are given in the list.

Assumes experiment has a multiobjective optimization config from which the objectives and outcome constraints will be extracted.

Will generate a ParetoFrontierResults for every pair of metrics in the experiment’s multiobjective optimization config.

Parameters:
  • experiment – The experiment.

  • data – Data to use for computing Pareto frontier. If not provided, will lookup data from experiment.

  • rel – Relativize results wrt experiment status quo. If None, then rel will be taken for each objective separately from its own objective threshold. rel must be specified if there are missing objective thresholds.

  • arm_names – If provided, computes Pareto frontier only from among the provided list of arm names, plus status quo if set on experiment.

Returns: ParetoFrontierResults that can be used with interact_pareto_frontier.

ax.plot.pareto_utils.get_tensor_converter_model(experiment: Experiment, data: Data) TorchModelBridge[source]

Constructs a minimal model for converting things to tensors.

Model fitting will instantiate all of the transforms but will not do any expensive (i.e. GP) fitting beyond that. The model will raise an error if it is used for predicting or generating.

Will work for any search space regardless of types of parameters.

Parameters:
  • experiment – Experiment.

  • data – Data for fitting the model.

Returns: A torch modelbridge with transforms set.

ax.plot.pareto_utils.infer_reference_point_from_experiment(experiment: Experiment) List[ObjectiveThreshold][source]

This functions is a wrapper around infer_reference_point to find the nadir point from the pareto front of an experiment. Aside from converting experiment to tensors, this wrapper transforms back and forth the objectives of the experiment so that they are appropriately used by infer_reference_point.

Parameters:

experiment – The experiment for which we want to infer the reference point.

Returns:

A list of objective thresholds representing the reference point.

ax.plot.pareto_utils.to_nonrobust_search_space(search_space: SearchSpace) SearchSpace[source]

Reduces a RobustSearchSpace to a SearchSpace.

This is a no-op for all other search spaces.

Scatter Plots

ax.plot.scatter.interact_fitted(model: ModelBridge, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, rel: bool = True, show_arm_details_on_hover: bool = True, show_CI: bool = True, arm_noun: str = 'arm', metrics: Optional[List[str]] = None, fixed_features: Optional[ObservationFeatures] = None, data_selector: Optional[Callable[[Observation], bool]] = None, label_dict: Optional[Dict[str, str]] = None, scalarized_metric_config: Optional[List[Dict[str, Any]]] = None) AxPlotConfig[source]

Interactive fitted outcome plots for each arm used in fitting the model.

Choose the outcome to plot using a dropdown.

Parameters:
  • model – model to use for predictions.

  • generator_runs_dict – a mapping from generator run name to generator run.

  • rel – if True, use relative effects. Default is True.

  • show_arm_details_on_hover – if True, display parameterizations of arms on hover. Default is True.

  • show_CI – if True, render confidence intervals.

  • arm_noun – noun to use instead of “arm” (e.g. group)

  • metrics – List of metric names to restrict to when plotting.

  • fixed_features – Fixed features to use when making model predictions.

  • data_selector – Function for selecting observations for plotting.

  • label_dict – A dictionary that maps the label to an alias to be used in the plot.

  • scalarized_metric_config – An optional list of dicts specifying how to aggregate multiple metrics into a single scalarized metric. For each dict, the key is the name of the new scalarized metric, and the value is a dictionary mapping each metric to its weight. e.g. {“name”: “metric1:agg”, “weight”: {“metric1_c1”: 0.5, “metric1_c2”: 0.5}}.

ax.plot.scatter.interact_fitted_plotly(model: ModelBridge, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, rel: bool = True, show_arm_details_on_hover: bool = True, show_CI: bool = True, arm_noun: str = 'arm', metrics: Optional[List[str]] = None, fixed_features: Optional[ObservationFeatures] = None, data_selector: Optional[Callable[[Observation], bool]] = None, label_dict: Optional[Dict[str, str]] = None, scalarized_metric_config: Optional[List[Dict[str, Any]]] = None) Figure[source]

Interactive fitted outcome plots for each arm used in fitting the model.

Choose the outcome to plot using a dropdown.

Parameters:
  • model – model to use for predictions.

  • generator_runs_dict – a mapping from generator run name to generator run.

  • rel – if True, use relative effects. Default is True.

  • show_arm_details_on_hover – if True, display parameterizations of arms on hover. Default is True.

  • show_CI – if True, render confidence intervals.

  • arm_noun – noun to use instead of “arm” (e.g. group)

  • metrics – List of metric names to restrict to when plotting.

  • fixed_features – Fixed features to use when making model predictions.

  • data_selector – Function for selecting observations for plotting.

  • label_dict – A dictionary that maps the label to an alias to be used in the plot.

  • scalarized_metric_config – An optional list of dicts specifying how to aggregate multiple metrics into a single scalarized metric. For each dict, the key is the name of the new scalarized metric, and the value is a dictionary mapping each metric to its weight. e.g. {“name”: “metric1:agg”, “weight”: {“metric1_c1”: 0.5, “metric1_c2”: 0.5}}.

ax.plot.scatter.lattice_multiple_metrics(model: ModelBridge, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, rel: bool = True, show_arm_details_on_hover: bool = False, data_selector: Optional[Callable[[Observation], bool]] = None) AxPlotConfig[source]

Plot raw values or predictions of combinations of two metrics for arms.

Parameters:
  • model – model to draw predictions from.

  • generator_runs_dict – a mapping from generator run name to generator run.

  • rel – if True, use relative effects. Default is True.

  • show_arm_details_on_hover – if True, display parameterizations of arms on hover. Default is False.

  • data_selector – Function for selecting observations for plotting.

ax.plot.scatter.plot_fitted(model: ModelBridge, metric: str, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, rel: bool = True, custom_arm_order: Optional[List[str]] = None, custom_arm_order_name: str = 'Custom', show_CI: bool = True, data_selector: Optional[Callable[[Observation], bool]] = None, scalarized_metric_config: Optional[List[Dict[str, Any]]] = None) AxPlotConfig[source]

Plot fitted metrics.

Parameters:
  • model – model to use for predictions.

  • metric – metric to plot predictions for.

  • generator_runs_dict – a mapping from generator run name to generator run.

  • rel – if True, use relative effects. Default is True.

  • custom_arm_order – a list of arm names in the order corresponding to how they should be plotted on the x-axis. If not None, this is the default ordering.

  • custom_arm_order_name – name for custom ordering to show in the ordering dropdown. Default is ‘Custom’.

  • show_CI – if True, render confidence intervals.

  • data_selector – Function for selecting observations for plotting.

  • scalarized_metric_config – An optional list of dicts specifying how to aggregate multiple metrics into a single scalarized metric. For each dict, the key is the name of the new scalarized metric, and the value is a dictionary mapping each metric to its weight. e.g. {“name”: “metric1:agg”, “weight”: {“metric1_c1”: 0.5, “metric1_c2”: 0.5}}.

ax.plot.scatter.plot_multiple_metrics(model: ModelBridge, metric_x: str, metric_y: str, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, rel_x: bool = True, rel_y: bool = True, fixed_features: Optional[ObservationFeatures] = None, data_selector: Optional[Callable[[Observation], bool]] = None, color_parameter: Optional[str] = None, color_metric: Optional[str] = None, **kwargs: Any) AxPlotConfig[source]

Plot raw values or predictions of two metrics for arms.

All arms used in the model are included in the plot. Additional arms can be passed through the generator_runs_dict argument.

Parameters:
  • model – model to draw predictions from.

  • metric_x – metric to plot on the x-axis.

  • metric_y – metric to plot on the y-axis.

  • generator_runs_dict – a mapping from generator run name to generator run.

  • rel_x – if True, use relative effects on metric_x.

  • rel_y – if True, use relative effects on metric_y.

  • data_selector – Function for selecting observations for plotting.

  • color_parameter – color points according to the specified parameter, cannot be used together with color_metric.

  • color_metric – color points according to the specified metric, cannot be used together with color_parameter.

ax.plot.scatter.plot_objective_vs_constraints(model: ModelBridge, objective: str, subset_metrics: Optional[List[str]] = None, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, rel: bool = True, infer_relative_constraints: Optional[bool] = False, fixed_features: Optional[ObservationFeatures] = None, data_selector: Optional[Callable[[Observation], bool]] = None, color_parameter: Optional[str] = None, color_metric: Optional[str] = None, label_dict: Optional[Dict[str, str]] = None) AxPlotConfig[source]

Plot the tradeoff between an objective and all other metrics in a model.

All arms used in the model are included in the plot. Additional arms can be passed through via the generator_runs_dict argument.

Fixed features input can be used to override fields of the insample arms when making model predictions.

Parameters:
  • model – model to draw predictions from.

  • objective – metric to optimize. Plotted on the x-axis.

  • subset_metrics – list of metrics to plot on the y-axes if need a subset of all metrics in the model.

  • generator_runs_dict – a mapping from generator run name to generator run.

  • rel – if True, use relative effects. Default is True.

  • infer_relative_constraints – if True, read relative spec from model’s optimization config. Absolute constraints will not be relativized; relative ones will be. Objectives will respect the rel parameter. Metrics that are not constraints will be relativized.

  • fixed_features – Fixed features to use when making model predictions.

  • data_selector – Function for selecting observations for plotting.

  • color_parameter – color points according to the specified parameter, cannot be used together with color_metric.

  • color_metric – color points according to the specified metric, cannot be used together with color_parameter.

  • label_dict – A dictionary that maps the label to an alias to be used in the plot.

ax.plot.scatter.tile_fitted(model: ModelBridge, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, rel: bool = True, show_arm_details_on_hover: bool = False, show_CI: bool = True, arm_noun: str = 'arm', metrics: Optional[List[str]] = None, fixed_features: Optional[ObservationFeatures] = None, data_selector: Optional[Callable[[Observation], bool]] = None, scalarized_metric_config: Optional[List[Dict[str, Any]]] = None, label_dict: Optional[Dict[str, str]] = None) AxPlotConfig[source]

Tile version of fitted outcome plots.

Parameters:
  • model – model to use for predictions.

  • generator_runs_dict – a mapping from generator run name to generator run.

  • rel – if True, use relative effects. Default is True.

  • show_arm_details_on_hover – if True, display parameterizations of arms on hover. Default is False.

  • show_CI – if True, render confidence intervals.

  • arm_noun – noun to use instead of “arm” (e.g. group)

  • metrics – List of metric names to restrict to when plotting.

  • fixed_features – Fixed features to use when making model predictions.

  • data_selector – Function for selecting observations for plotting.

  • scalarized_metric_config – An optional list of dicts specifying how to aggregate multiple metrics into a single scalarized metric. For each dict, the key is the name of the new scalarized metric, and the value is a dictionary mapping each metric to its weight. e.g. {“name”: “metric1:agg”, “weight”: {“metric1_c1”: 0.5, “metric1_c2”: 0.5}}.

  • label_dict – A dictionary that maps the label to an alias to be used in the plot.

ax.plot.scatter.tile_observations(experiment: Experiment, data: Optional[Data] = None, rel: bool = True, metrics: Optional[List[str]] = None, arm_names: Optional[List[str]] = None, arm_noun: str = 'arm', label_dict: Optional[Dict[str, str]] = None) AxPlotConfig[source]

Tiled plot with all observed outcomes.

Will plot all observed arms. If data is provided will use that, otherwise will fetch data from experiment. Will plot all metrics in data unless a list is provided in metrics. If arm_names is provided will limit the plot to only arms in that list.

Parameters:
  • experiment – Experiment

  • data – Data to use, otherwise will fetch data from experiment.

  • rel – Plot relative values, if experiment has status quo.

  • metrics – Limit results to this set of metrics.

  • arm_names – Limit results to this set of arms.

  • arm_noun – Noun to use instead of “arm”.

  • label_dict – A dictionary that maps the label to an alias to be used in the plot.

Returns: Plot config for the plot.

Slice Plot

ax.plot.slice.interact_slice(model: ModelBridge, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, relative: bool = False, density: int = 50, slice_values: Optional[Dict[str, Any]] = None, fixed_features: Optional[ObservationFeatures] = None, trial_index: Optional[int] = None) AxPlotConfig[source]

Create interactive plot with predictions for a 1-d slice of the parameter space.

Parameters:
  • model – ModelBridge that contains model for predictions

  • generator_runs_dict – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.

  • relative – Predictions relative to status quo

  • density – Number of points along slice to evaluate predictions.

  • slice_values – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters. Ignored if fixed_features is specified.

  • fixed_features – An ObservationFeatures object containing the values of features (including non-parameter features like context) to be set in the slice.

Returns:

interactive plot of objective vs. parameter

Return type:

AxPlotConfig

ax.plot.slice.interact_slice_plotly(model: ModelBridge, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, relative: bool = False, density: int = 50, slice_values: Optional[Dict[str, Any]] = None, fixed_features: Optional[ObservationFeatures] = None, trial_index: Optional[int] = None) Figure[source]

Create interactive plot with predictions for a 1-d slice of the parameter space.

Parameters:
  • model – ModelBridge that contains model for predictions

  • generator_runs_dict – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.

  • relative – Predictions relative to status quo

  • density – Number of points along slice to evaluate predictions.

  • slice_values – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters. Ignored if fixed_features is specified.

  • fixed_features – An ObservationFeatures object containing the values of features (including non-parameter features like context) to be set in the slice.

Returns:

interactive plot of objective vs. parameter

Return type:

go.Figure

ax.plot.slice.plot_slice(model: ModelBridge, param_name: str, metric_name: str, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, relative: bool = False, density: int = 50, slice_values: Optional[Dict[str, Any]] = None, fixed_features: Optional[ObservationFeatures] = None, trial_index: Optional[int] = None) AxPlotConfig[source]

Plot predictions for a 1-d slice of the parameter space.

Parameters:
  • model – ModelBridge that contains model for predictions

  • param_name – Name of parameter that will be sliced

  • metric_name – Name of metric to plot

  • generator_runs_dict – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.

  • relative – Predictions relative to status quo

  • density – Number of points along slice to evaluate predictions.

  • slice_values – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters. Ignored if fixed_features is specified.

  • fixed_features – An ObservationFeatures object containing the values of features (including non-parameter features like context) to be set in the slice.

Returns:

plot of objective vs. parameter value

Return type:

AxPlotConfig

ax.plot.slice.plot_slice_plotly(model: ModelBridge, param_name: str, metric_name: str, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, relative: bool = False, density: int = 50, slice_values: Optional[Dict[str, Any]] = None, fixed_features: Optional[ObservationFeatures] = None, trial_index: Optional[int] = None) Figure[source]

Plot predictions for a 1-d slice of the parameter space.

Parameters:
  • model – ModelBridge that contains model for predictions

  • param_name – Name of parameter that will be sliced

  • metric_name – Name of metric to plot

  • generator_runs_dict – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.

  • relative – Predictions relative to status quo

  • density – Number of points along slice to evaluate predictions.

  • slice_values – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters. Ignored if fixed_features is specified.

  • fixed_features – An ObservationFeatures object containing the values of features (including non-parameter features like context) to be set in the slice.

Returns:

plot of objective vs. parameter value

Return type:

go.Figure

Table

ax.plot.table_view.get_color(x: float, ci: float, rel: bool, reverse: bool) str[source]

Determine the color of the table cell.

ax.plot.table_view.table_view_plot(experiment: Experiment, data: Data, use_empirical_bayes: bool = True, only_data_frame: bool = False, arm_noun: str = 'arm') Tuple[pandas.core.frame.DataFrame][source]

Table of means and confidence intervals.

Table is of the form:

arm

metric_1

metric_2

0_0

mean +- CI

0_1

Trace Plots

Parallel Coordinates

Plotting Utilities

class ax.plot.color.COLORS(value)[source]

Bases: Enum

An enumeration.

CORAL = (251, 128, 114)
LIGHT_PURPLE = (190, 186, 218)
ORANGE = (253, 180, 98)
PINK = (188, 128, 189)
STEELBLUE = (128, 177, 211)
TEAL = (141, 211, 199)
ax.plot.color.plotly_color_scale(list_of_rgb_tuples: List[Tuple[Real, ...]], reverse: bool = False, alpha: float = 1) List[Tuple[float, str]][source]

Convert list of RGB tuples to list of tuples, where each tuple is break in [0, 1] and stringified RGBA color.

ax.plot.color.rgba(rgb_tuple: Tuple[Real, ...], alpha: float = 1) str[source]

Convert RGB tuple to an RGBA string.

ax.plot.helper.arm_name_to_sort_key(arm_name: str) Tuple[str, int, int][source]

Parses arm name into tuple suitable for reverse sorting by key

Example

arm_names = [“0_0”, “1_10”, “1_2”, “10_0”, “control”] sorted(arm_names, key=arm_name_to_sort_key, reverse=True) [“control”, “0_0”, “1_2”, “1_10”, “10_0”]

ax.plot.helper.arm_name_to_tuple(arm_name: str) Union[Tuple[int, int], Tuple[int]][source]
ax.plot.helper.axis_range(grid: List[float], is_log: bool) List[float][source]
ax.plot.helper.build_filter_trial(keep_trial_indices: List[int]) Callable[[Observation], bool][source]

Creates a callable that filters observations based on trial_index

ax.plot.helper.compose_annotation(caption: str, x: float = 0.0, y: float = -0.15) List[Dict[str, Any]][source]
ax.plot.helper.contour_config_to_trace(config) List[Dict[str, Any]][source]
ax.plot.helper.extend_range(lower: float, upper: float, percent: int = 10, log_scale: bool = False) Tuple[float, float][source]

Given a range of minimum and maximum values taken by values on a given axis, extend it in both directions by a given percentage to have some margin within the plot around its meaningful part.

ax.plot.helper.get_fixed_values(model: ModelBridge, slice_values: Optional[Dict[str, Any]] = None, trial_index: Optional[int] = None) Dict[str, Union[None, str, bool, float, int]][source]

Get fixed values for parameters in a slice plot.

If there is an in-design status quo, those values will be used. Otherwise, the mean of RangeParameters or the mode of ChoiceParameters is used.

Any value in slice_values will override the above.

Parameters:
  • model – ModelBridge being used for plotting

  • slice_values – Map from parameter name to value at which is should be fixed.

Returns: Map from parameter name to fixed value.

ax.plot.helper.get_grid_for_parameter(parameter: RangeParameter, density: int) ndarray[source]

Get a grid of points along the range of the parameter.

Will be a log-scale grid if parameter is log scale.

Parameters:
  • parameter – Parameter for which to generate grid.

  • density – Number of points in the grid.

ax.plot.helper.get_plot_data(model: ModelBridge, generator_runs_dict: Dict[str, GeneratorRun], metric_names: Optional[Set[str]] = None, fixed_features: Optional[ObservationFeatures] = None, data_selector: Optional[Callable[[Observation], bool]] = None, scalarized_metric_config: Optional[List[Dict[str, Dict[str, float]]]] = None) Tuple[PlotData, List[Dict[str, Union[str, float]]], Dict[str, Dict[str, Union[None, str, bool, float, int]]]][source]

Format data object with metrics for in-sample and out-of-sample arms.

Calculate both observed and predicted metrics for in-sample arms. Calculate predicted metrics for out-of-sample arms passed via the generator_runs_dict argument.

In PlotData, in-sample observations are merged with IVW. In RawData, they are left un-merged and given as a list of dictionaries, one for each observation and having keys ‘arm_name’, ‘mean’, and ‘sem’.

Parameters:
  • model – The model.

  • generator_runs_dict – a mapping from generator run name to generator run.

  • metric_names – Restrict predictions to this set. If None, all metrics in the model will be returned.

  • fixed_features – Fixed features to use when making model predictions.

  • data_selector – Function for selecting observations for plotting.

  • scalarized_metric_config – An optional list of dicts specifying how to aggregate multiple metrics into a single scalarized metric. For each dict, the key is the name of the new scalarized metric, and the value is a dictionary mapping each metric to its weight. e.g. {“name”: “metric1:agg”, “weight”: {“metric1_c1”: 0.5, “metric1_c2”: 0.5}}.

Returns:

A tuple containing

  • PlotData object with in-sample and out-of-sample predictions.

  • List of observations like:

    {'metric_name': 'likes', 'arm_name': '0_1', 'mean': 1., 'sem': 0.1}.
    
  • Mapping from arm name to parameters.

ax.plot.helper.get_range_parameter(model: ModelBridge, param_name: str) RangeParameter[source]

Get the range parameter with the given name from the model.

Throws if parameter doesn’t exist or is not a range parameter.

Parameters:
  • model – The model.

  • param_name – The name of the RangeParameter to be found.

Returns: The RangeParameter named param_name.

ax.plot.helper.get_range_parameters(model: ModelBridge, min_num_values: int = 0) List[RangeParameter][source]

Get a list of range parameters from a model.

Parameters:
  • model – The model.

  • min_num_values – Minimum number of values

Returns: List of RangeParameters.

ax.plot.helper.get_range_parameters_from_list(parameters: List[Parameter], min_num_values: int = 0) List[RangeParameter][source]

Get a list of range parameters from a model.

Parameters:
  • parameters – List of parameters

  • min_num_values – Minimum number of values

Returns: List of RangeParameters.

ax.plot.helper.infer_is_relative(model: ModelBridge, metrics: List[str], non_constraint_rel: bool) Dict[str, bool][source]

Determine whether or not to relativize a metric.

Metrics that are constraints will get this decision from their relative flag. Other metrics will use the default_rel.

Parameters:
  • model – model fit on metrics.

  • metrics – list of metric names.

  • non_constraint_rel – whether or not to relativize non-constraint metrics

Returns:

Dict[str, bool] containing whether or not to relativize each input metric.

ax.plot.helper.relativize(m_t: float, sem_t: float, m_c: float, sem_c: float) List[float][source]
ax.plot.helper.relativize_data(f: List[float], sd: List[float], rel: bool, arm_data: Dict[Any, Any], metric: str) List[List[float]][source]
ax.plot.helper.resize_subtitles(figure: Dict[str, Any], size: int) Dict[str, Any][source]
ax.plot.helper.rgb(arr: List[int]) str[source]
ax.plot.helper.slice_config_to_trace(arm_data, arm_name_to_parameters, f: List[float], fit_data, grid, metric: str, param, rel: bool, setx, sd: List[float], is_log, visible) List[Dict[str, Any]][source]