ax.plot¶

Base¶

class ax.plot.base.AxPlotConfig(data: Dict[str, Any], plot_type: Enum)[source]¶

Bases: _AxPlotConfigBase

Config for plots

class ax.plot.base.AxPlotTypes(value)[source]¶

Bases: Enum

Enum of Ax plot types.

BANDIT_ROLLOUT = 4¶

CONTOUR = 0¶

GENERIC = 1¶

HTML = 6¶

INTERACT_CONTOUR = 3¶

INTERACT_SLICE = 5¶

SLICE = 2¶

class ax.plot.base.PlotData(metrics: List[str], in_sample: Dict[str, PlotInSampleArm], out_of_sample: Optional[Dict[str, Dict[str, PlotOutOfSampleArm]]], status_quo_name: Optional[str])[source]¶

Bases: tuple

Struct for plot data, including both in-sample and out-of-sample arms

in_sample: Dict[str, PlotInSampleArm]¶

Alias for field number 1

metrics: List[str]¶

Alias for field number 0

out_of_sample: Optional[Dict[str, Dict[str, PlotOutOfSampleArm]]]¶

Alias for field number 2

status_quo_name: Optional[str]¶

Alias for field number 3

class ax.plot.base.PlotInSampleArm(name: str, parameters: Dict[str, Optional[Union[str, bool, float, int]]], y: Dict[str, float], y_hat: Dict[str, float], se: Dict[str, float], se_hat: Dict[str, float], context_stratum: Optional[Dict[str, Union[str, float]]])[source]¶

Bases: tuple

Struct for in-sample arms (both observed and predicted data)

context_stratum: Optional[Dict[str, Union[str, float]]]¶

Alias for field number 6

name: str¶

Alias for field number 0

parameters: Dict[str, Optional[Union[str, bool, float, int]]]¶

Alias for field number 1

se: Dict[str, float]¶

Alias for field number 4

se_hat: Dict[str, float]¶

Alias for field number 5

y: Dict[str, float]¶

Alias for field number 2

y_hat: Dict[str, float]¶

Alias for field number 3

class ax.plot.base.PlotMetric(metric: str, pred: bool, rel: bool)[source]¶

Bases: tuple

Struct for metric

metric: str¶

Alias for field number 0

pred: bool¶

Alias for field number 1

rel: bool¶

Alias for field number 2

class ax.plot.base.PlotOutOfSampleArm(name: str, parameters: Dict[str, Optional[Union[str, bool, float, int]]], y_hat: Dict[str, float], se_hat: Dict[str, float], context_stratum: Optional[Dict[str, Union[str, float]]])[source]¶

Bases: tuple

Struct for out-of-sample arms (only predicted data)

context_stratum: Optional[Dict[str, Union[str, float]]]¶

Alias for field number 4

name: str¶

Alias for field number 0

parameters: Dict[str, Optional[Union[str, bool, float, int]]]¶

Alias for field number 1

se_hat: Dict[str, float]¶

Alias for field number 3

y_hat: Dict[str, float]¶

Alias for field number 2

Bandit Rollout¶

ax.plot.bandit_rollout.plot_bandit_rollout(experiment: Experiment) → AxPlotConfig[source]¶

Plot bandit rollout from ane experiement.

Benchmark¶

ax.plot.benchmark.plot_modeling_times(aggregated_results: Iterable[AggregatedBenchmarkResult]) → AxPlotConfig[source]¶

Plots wall times of each method’s fit and gen calls as a stack bar chart.

ax.plot.benchmark.plot_optimization_trace(aggregated_results: List[AggregatedBenchmarkResult], optimum: Optional[float] = None, by_progression: bool = False, final_progression_only: bool = False) → AxPlotConfig[source]¶

Plots optimization trace for each aggregated result with mean and SEM. When by_progression is True, the results are plotted with progressions on the x-axis. In that case, if final_progression_only is True, then the value of a trial is taken to be the value of its final progression.

If an optimum is provided (can represent either an optimal value or maximum hypervolume in the case of multi-objective problems) it will be plotted as an orange dashed line as well.

Contour Plot¶

ax.plot.contour.interact_contour(model: ModelBridge, metric_name: str, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, relative: bool = False, density: int = 50, slice_values: Optional[Dict[str, Any]] = None, lower_is_better: bool = False, fixed_features: Optional[ObservationFeatures] = None, trial_index: Optional[int] = None) → AxPlotConfig[source]¶

Create interactive plot with predictions for a 2-d slice of the parameter space.

Parameters:

• model – ModelBridge that contains model for predictions

• metric_name – Name of metric to plot

• generator_runs_dict – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.

• relative – Predictions relative to status quo

• density – Number of points along slice to evaluate predictions.

• slice_values – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters.

• lower_is_better – Lower values for metric are better.

• fixed_features – An ObservationFeatures object containing the values of features (including non-parameter features like context) to be set in the slice.

Returns:

interactive plot of objective vs. parameters

Return type:

AxPlotConfig

ax.plot.contour.interact_contour_plotly(model: ModelBridge, metric_name: str, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, relative: bool = False, density: int = 50, slice_values: Optional[Dict[str, Any]] = None, lower_is_better: bool = False, fixed_features: Optional[ObservationFeatures] = None, trial_index: Optional[int] = None) → Figure[source]¶

Create interactive plot with predictions for a 2-d slice of the parameter space.

Parameters:

• model – ModelBridge that contains model for predictions

• metric_name – Name of metric to plot

• generator_runs_dict – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.

• relative – Predictions relative to status quo

• density – Number of points along slice to evaluate predictions.

• slice_values – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters.

• lower_is_better – Lower values for metric are better.

• fixed_features – An ObservationFeatures object containing the values of features (including non-parameter features like context) to be set in the slice.

Returns:

interactive plot of objective vs. parameters

Return type:

go.Figure

ax.plot.contour.plot_contour(model: ModelBridge, param_x: str, param_y: str, metric_name: str, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, relative: bool = False, density: int = 50, slice_values: Optional[Dict[str, Any]] = None, lower_is_better: bool = False, fixed_features: Optional[ObservationFeatures] = None, trial_index: Optional[int] = None) → AxPlotConfig[source]¶

Plot predictions for a 2-d slice of the parameter space.

Parameters:

• model – ModelBridge that contains model for predictions

• param_x – Name of parameter that will be sliced on x-axis

• param_y – Name of parameter that will be sliced on y-axis

• metric_name – Name of metric to plot

• generator_runs_dict – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.

• relative – Predictions relative to status quo

• density – Number of points along slice to evaluate predictions.

• slice_values – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters.

• lower_is_better – Lower values for metric are better.

• fixed_features – An ObservationFeatures object containing the values of features (including non-parameter features like context) to be set in the slice.

Returns:

contour plot of objective vs. parameter values

Return type:

AxPlotConfig

ax.plot.contour.plot_contour_plotly(model: ModelBridge, param_x: str, param_y: str, metric_name: str, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, relative: bool = False, density: int = 50, slice_values: Optional[Dict[str, Any]] = None, lower_is_better: bool = False, fixed_features: Optional[ObservationFeatures] = None, trial_index: Optional[int] = None) → Figure[source]¶

Plot predictions for a 2-d slice of the parameter space.

Parameters:

• model – ModelBridge that contains model for predictions

• param_x – Name of parameter that will be sliced on x-axis

• param_y – Name of parameter that will be sliced on y-axis

• metric_name – Name of metric to plot

• generator_runs_dict – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.

• relative – Predictions relative to status quo

• density – Number of points along slice to evaluate predictions.

• slice_values – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters.

• lower_is_better – Lower values for metric are better.

• fixed_features – An ObservationFeatures object containing the values of features (including non-parameter features like context) to be set in the slice.

Returns:

contour plot of objective vs. parameter values

Return type:

go.Figure

ax.plot.contour.short_name(param_name: str) → str[source]¶

Feature Importances¶

ax.plot.feature_importances.plot_feature_importance(df: pandas.DataFrame, title: str) → AxPlotConfig[source]¶

Wrapper method to convert plot_feature_importance_plotly to AxPlotConfig

ax.plot.feature_importances.plot_feature_importance_by_feature(model: Optional[ModelBridge] = None, sensitivity_values: Optional[Dict[str, Dict[str, Union[float, ndarray]]]] = None, relative: bool = False, caption: str = '', importance_measure: str = '') → AxPlotConfig[source]¶

Wrapper method to convert plot_feature_importance_by_feature_plotly to AxPlotConfig

ax.plot.feature_importances.plot_feature_importance_by_feature_plotly(model: Optional[ModelBridge] = None, sensitivity_values: Optional[Dict[str, Dict[str, Union[float, ndarray]]]] = None, relative: bool = False, caption: str = '', importance_measure: str = '') → Figure[source]¶

One plot per metric, showing importances by feature.

Parameters:

• model – A model with a feature_importances method.

• sensitivity_values – The sensitivity values for each metric in a dict format. It takes the following format if only the sensitivity value is plotted: {“metric1”:{“parameter1”:value1,”parameter2”:value2 …} …} It takes the following format if the sensitivity value and standard error are plotted: {“metric1”:{“parameter1”:[value1,var,se], “parameter2”:[[value2,var,se]]…}…}}.

• relative – Whether to normalize feature importances so that they add to 1.

• caption – An HTML-formatted string to place at the bottom of the plot.

• importance_measure – The name of the importance metric to be added to the title.

Returns a go.Figure of feature importances.

ax.plot.feature_importances.plot_feature_importance_by_metric(model: ModelBridge) → AxPlotConfig[source]¶

Wrapper method to convert plot_feature_importance_by_metric_plotly to AxPlotConfig

ax.plot.feature_importances.plot_feature_importance_by_metric_plotly(model: ModelBridge) → Figure[source]¶

One plot per feature, showing importances by metric.

ax.plot.feature_importances.plot_feature_importance_plotly(df: pandas.DataFrame, title: str) → Figure[source]¶

ax.plot.feature_importances.plot_relative_feature_importance(model: ModelBridge) → AxPlotConfig[source]¶

Wrapper method to convert plot_relative_feature_importance_plotly to AxPlotConfig

ax.plot.feature_importances.plot_relative_feature_importance_plotly(model: ModelBridge) → Figure[source]¶

Create a stacked bar chart of feature importances per metric

Marginal Effects¶

ax.plot.marginal_effects.plot_marginal_effects(model: ModelBridge, metric: str) → AxPlotConfig[source]¶

Calculates and plots the marginal effects – the effect of changing one factor away from the randomized distribution of the experiment and fixing it at a particular level.

Parameters:

• model – Model to use for estimating effects

• metric – The metric for which to plot marginal effects.

Returns:

AxPlotConfig of the marginal effects

Model Diagnostics¶

ax.plot.diagnostic.interact_batch_comparison(observations: List[Observation], experiment: Experiment, batch_x: int, batch_y: int, rel: bool = False, status_quo_name: Optional[str] = None) → AxPlotConfig[source]¶

Compare repeated arms from two trials; select metric via dropdown.

Parameters:

• observations – List of observations to compute comparison.

• batch_x – Index of batch for x-axis.

• batch_y – Index of bach for y-axis.

• rel – Whether to relativize data against status_quo arm.

• status_quo_name – Name of the status_quo arm.

ax.plot.diagnostic.interact_cross_validation(cv_results: List[CVResult], show_context: bool = True, label_dict: Optional[Dict[str, str]] = None) → AxPlotConfig[source]¶

Interactive cross-validation (CV) plotting; select metric via dropdown.

Note: uses the Plotly version of dropdown (which means that all data is stored within the notebook).

Parameters:

• cv_results – cross-validation results.

• show_context – if True, show context on hover.

• label_dict – optional map from real metric names to shortened names

Returns an AxPlotConfig

ax.plot.diagnostic.interact_cross_validation_plotly(cv_results: List[CVResult], show_context: bool = True, caption: str = '', label_dict: Optional[Dict[str, str]] = None) → Figure[source]¶

Interactive cross-validation (CV) plotting; select metric via dropdown.

Note: uses the Plotly version of dropdown (which means that all data is stored within the notebook).

Parameters:

• cv_results – cross-validation results.

• show_context – if True, show context on hover.

• label_dict – optional map from real metric names to shortened names

Returns a plotly.graph_objects.Figure

ax.plot.diagnostic.interact_empirical_model_validation(batch: BatchTrial, data: Data) → AxPlotConfig[source]¶

Compare the model predictions for the batch arms against observed data.

Relies on the model predictions stored on the generator_runs of batch.

Parameters:

• batch – Batch on which to perform analysis.

• data – Observed data for the batch.

Returns:

AxPlotConfig for the plot.

ax.plot.diagnostic.tile_cross_validation(cv_results: List[CVResult], show_arm_details_on_hover: bool = True, show_context: bool = True, label_dict: Optional[Dict[str, str]] = None) → AxPlotConfig[source]¶

Tile version of CV plots; sorted by ‘best fitting’ outcomes.

Plots are sorted in decreasing order using the p-value of a Fisher exact test statistic.

Parameters:

• cv_results – cross-validation results.

• include_measurement_error – if True, include measurement_error metrics in plot.

• show_arm_details_on_hover – if True, display parameterizations of arms on hover. Default is True.

• show_context – if True (default), display context on hover.

• label_dict – optional map from real metric names to shortened names

Returns a plotly.graph_objects.Figure

Pareto Plots¶

ax.plot.pareto_frontier.interact_multiple_pareto_frontier(frontier_lists: Dict[str, List[ParetoFrontierResults]], CI_level: float = 0.9, show_parameterization_on_hover: bool = True) → AxPlotConfig[source]¶

Plot a Pareto frontiers from a list of lists of NamedParetoFrontierResults objects that we want to compare.

Parameters:

• frontier_lists (Dict[List[ParetoFrontierResults]]) – A dictionary of multiple lists of Pareto frontier computation results to plot for comparison. Each list of ParetoFrontierResults contains a list of the results of the same pareto frontier but under different pairs of metrics. Different List[ParetoFrontierResults] must contain the the same pairs of metrics for this function to work.

• CI_level (float, optional) – The confidence level, i.e. 0.95 (95%)

• show_parameterization_on_hover (bool, optional) – If True, show the parameterization of the points on the frontier on hover.

Returns:

The resulting Plotly plot definition.

Return type:

AEPlotConfig

ax.plot.pareto_frontier.interact_pareto_frontier(frontier_list: List[ParetoFrontierResults], CI_level: float = 0.9, show_parameterization_on_hover: bool = True) → AxPlotConfig[source]¶

Plot a pareto frontier from a list of objects

ax.plot.pareto_frontier.plot_multiple_pareto_frontiers(frontiers: Dict[str, ParetoFrontierResults], CI_level: float = 0.9, show_parameterization_on_hover: bool = True) → AxPlotConfig[source]¶

Plot a Pareto frontier from a ParetoFrontierResults object.

Parameters:

• frontiers (Dict[str, ParetoFrontierResults]) – The results of the Pareto frontier computation.

• CI_level (float, optional) – The confidence level, i.e. 0.95 (95%)

• show_parameterization_on_hover (bool, optional) – If True, show the parameterization of the points on the frontier on hover.

Returns:

The resulting Plotly plot definition.

Return type:

AEPlotConfig

ax.plot.pareto_frontier.plot_pareto_frontier(frontier: ParetoFrontierResults, CI_level: float = 0.9, show_parameterization_on_hover: bool = True) → AxPlotConfig[source]¶

Plot a Pareto frontier from a ParetoFrontierResults object.

Parameters:

• frontier (ParetoFrontierResults) – The results of the Pareto frontier computation.

• CI_level (float, optional) – The confidence level, i.e. 0.95 (95%)

• show_parameterization_on_hover (bool, optional) – If True, show the parameterization of the points on the frontier on hover.

Returns:

The resulting Plotly plot definition.

Return type:

AEPlotConfig

ax.plot.pareto_frontier.scatter_plot_with_pareto_frontier(Y: ndarray, Y_pareto: ndarray, metric_x: str, metric_y: str, reference_point: Tuple[float, float], minimize: bool = True) → AxPlotConfig[source]¶

ax.plot.pareto_frontier.scatter_plot_with_pareto_frontier_plotly(Y: ndarray, Y_pareto: Optional[ndarray], metric_x: Optional[str], metric_y: Optional[str], reference_point: Optional[Tuple[float, float]], minimize: Optional[Union[bool, Tuple[bool, bool]]] = True) → Figure[source]¶

Plots a scatter of all points in Y for metric_x and metric_y with a reference point and Pareto frontier from Y_pareto.

Points in the scatter are colored in a gradient representing their trial index, with metric_x on x-axis and metric_y on y-axis. Reference point is represented as a star and Pareto frontier –– as a line. The frontier connects to the reference point via projection lines.

NOTE: Both metrics should have the same minimization setting, passed as minimize.

Parameters:

• Y – Array of outcomes, of which the first two will be plotted.

• Y_pareto – Array of Pareto-optimal points, first two outcomes in which will be plotted.

• metric_x – Name of first outcome in Y.

• metric_Y – Name of second outcome in Y.

• reference_point – Reference point for metric_x and metric_y.

• minimize – Whether the two metrics in the plot are being minimized or maximized.

class ax.plot.pareto_utils.ParetoFrontierResults(param_dicts: List[Dict[str, Optional[Union[str, bool, float, int]]]], means: Dict[str, List[float]], sems: Dict[str, List[float]], primary_metric: str, secondary_metric: str, absolute_metrics: List[str], objective_thresholds: Optional[Dict[str, float]], arm_names: Optional[List[Optional[str]]])[source]¶

Bases: tuple

Container for results from Pareto frontier computation.

Fields are: - param_dicts: The parameter dicts of the points generated on the Pareto Frontier. - means: The posterior mean predictions of the model for each metric (same order as the param dicts). These must be as a percent change relative to status quo for any metric not listed in absolute_metrics. - sems: The posterior sem predictions of the model for each metric (same order as the param dicts). Also must be relativized wrt status quo for any metric not listed in absolute_metrics. - primary_metric: The name of the primary metric. - secondary_metric: The name of the secondary metric. - absolute_metrics: List of outcome metrics that are NOT be relativized w.r.t. the status quo. All other metrics are assumed to be given here as % relative to status_quo. - objective_thresholds: Threshold for each objective. Must be on the same scale as means, so if means is relativized it should be the relative value, otherwise it should be absolute. - arm_names: Optional list of arm names for each parameterization.

absolute_metrics: List[str]¶

Alias for field number 5

arm_names: Optional[List[Optional[str]]]¶

Alias for field number 7

means: Dict[str, List[float]]¶

Alias for field number 1

objective_thresholds: Optional[Dict[str, float]]¶

Alias for field number 6

param_dicts: List[Dict[str, Optional[Union[str, bool, float, int]]]]¶

Alias for field number 0

primary_metric: str¶

Alias for field number 3

secondary_metric: str¶

Alias for field number 4

sems: Dict[str, List[float]]¶

Alias for field number 2

ax.plot.pareto_utils.compute_posterior_pareto_frontier(experiment: Experiment, primary_objective: Metric, secondary_objective: Metric, data: Optional[Data] = None, outcome_constraints: Optional[List[OutcomeConstraint]] = None, absolute_metrics: Optional[List[str]] = None, num_points: int = 10, trial_index: Optional[int] = None, chebyshev: bool = True) → ParetoFrontierResults[source]¶

Compute the Pareto frontier between two objectives. For experiments with batch trials, a trial index or data object must be provided.

This is done by fitting a GP and finding the pareto front according to the GP posterior mean.

Parameters:

• experiment – The experiment to compute a pareto frontier for.

• primary_objective – The primary objective to optimize.

• secondary_objective – The secondary objective against which to trade off the primary objective.

• outcome_constraints – Outcome constraints to be respected by the optimization. Can only contain constraints on metrics that are not primary or secondary objectives.

• absolute_metrics – List of outcome metrics that should NOT be relativized w.r.t. the status quo (all other outcomes will be in % relative to status_quo).

• num_points – The number of points to compute on the Pareto frontier.

• chebyshev – Whether to use augmented_chebyshev_scalarization when computing Pareto Frontier points.

Returns:

A NamedTuple with fields listed in its definition.

Return type:

ParetoFrontierResults

ax.plot.pareto_utils.get_observed_pareto_frontiers(experiment: Experiment, data: Optional[Data] = None, rel: bool = True, arm_names: Optional[List[str]] = None) → List[ParetoFrontierResults][source]¶

Find all Pareto points from an experiment.

Uses only values as observed in the data; no modeling is involved. Makes no assumption about the search space or types of parameters. If “data” is provided will use that, otherwise will use all data attached to the experiment.

Uses all arms present in data; does not filter according to experiment search space. If arm_names is specified, will filter to just those arm whose names are given in the list.

Assumes experiment has a multiobjective optimization config from which the objectives and outcome constraints will be extracted.

Will generate a ParetoFrontierResults for every pair of metrics in the experiment’s multiobjective optimization config.

Parameters:

• experiment – The experiment.

• data – Data to use for computing Pareto frontier. If not provided, will fetch data from experiment.

• rel – Relativize, if status quo on experiment.

• arm_names – If provided, computes Pareto frontier only from among the provided list of arm names, plus status quo if set on experiment.

Returns: ParetoFrontierResults that can be used with interact_pareto_frontier.

ax.plot.pareto_utils.get_tensor_converter_model(experiment: Experiment, data: Data) → TorchModelBridge[source]¶

Constructs a minimal model for converting things to tensors.

Model fitting will instantiate all of the transforms but will not do any expensive (i.e. GP) fitting beyond that. The model will raise an error if it is used for predicting or generating.

Will work for any search space regardless of types of parameters.

Parameters:

• experiment – Experiment.

• data – Data for fitting the model.

Returns: A torch modelbridge with transforms set.

ax.plot.pareto_utils.infer_reference_point_from_experiment(experiment: Experiment) → List[ObjectiveThreshold][source]¶

This functions is a wrapper around infer_reference_point to find the nadir point from the pareto front of an experiment. Aside from converting experiment to tensors, this wrapper transforms back and forth the objectives of the experiment so that they are appropriately used by infer_reference_point.

Parameters:

experiment – The experiment for which we want to infer the reference point.

Returns:

List of obejective thresholds representing the reference point.

Scatter Plots¶

ax.plot.scatter.interact_fitted(model: ModelBridge, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, rel: bool = True, show_arm_details_on_hover: bool = True, show_CI: bool = True, arm_noun: str = 'arm', metrics: Optional[List[str]] = None, fixed_features: Optional[ObservationFeatures] = None, data_selector: Optional[Callable[[Observation], bool]] = None, label_dict: Optional[Dict[str, str]] = None) → AxPlotConfig[source]¶

Interactive fitted outcome plots for each arm used in fitting the model.

Choose the outcome to plot using a dropdown.

Parameters:

• model – model to use for predictions.

• generator_runs_dict – a mapping from generator run name to generator run.

• rel – if True, use relative effects. Default is True.

• show_arm_details_on_hover – if True, display parameterizations of arms on hover. Default is True.

• show_CI – if True, render confidence intervals.

• arm_noun – noun to use instead of “arm” (e.g. group)

• metrics – List of metric names to restrict to when plotting.

• fixed_features – Fixed features to use when making model predictions.

• data_selector – Function for selecting observations for plotting.

• label_dict – A dictionary that maps the label to an alias to be used in the plot.

ax.plot.scatter.interact_fitted_plotly(model: ModelBridge, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, rel: bool = True, show_arm_details_on_hover: bool = True, show_CI: bool = True, arm_noun: str = 'arm', metrics: Optional[List[str]] = None, fixed_features: Optional[ObservationFeatures] = None, data_selector: Optional[Callable[[Observation], bool]] = None, label_dict: Optional[Dict[str, str]] = None) → Figure[source]¶

Interactive fitted outcome plots for each arm used in fitting the model.

Choose the outcome to plot using a dropdown.

Parameters:

• model – model to use for predictions.

• generator_runs_dict – a mapping from generator run name to generator run.

• rel – if True, use relative effects. Default is True.

• show_arm_details_on_hover – if True, display parameterizations of arms on hover. Default is True.

• show_CI – if True, render confidence intervals.

• arm_noun – noun to use instead of “arm” (e.g. group)

• metrics – List of metric names to restrict to when plotting.

• fixed_features – Fixed features to use when making model predictions.

• data_selector – Function for selecting observations for plotting.

• label_dict – A dictionary that maps the label to an alias to be used in the plot.

ax.plot.scatter.lattice_multiple_metrics(model: ModelBridge, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, rel: bool = True, show_arm_details_on_hover: bool = False, data_selector: Optional[Callable[[Observation], bool]] = None) → AxPlotConfig[source]¶

Plot raw values or predictions of combinations of two metrics for arms.

Parameters:

• model – model to draw predictions from.

• generator_runs_dict – a mapping from generator run name to generator run.

• rel – if True, use relative effects. Default is True.

• show_arm_details_on_hover – if True, display parameterizations of arms on hover. Default is False.

• data_selector – Function for selecting observations for plotting.

ax.plot.scatter.plot_fitted(model: ModelBridge, metric: str, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, rel: bool = True, custom_arm_order: Optional[List[str]] = None, custom_arm_order_name: str = 'Custom', show_CI: bool = True, data_selector: Optional[Callable[[Observation], bool]] = None) → AxPlotConfig[source]¶

Plot fitted metrics.

Parameters:

• model – model to use for predictions.

• metric – metric to plot predictions for.

• generator_runs_dict – a mapping from generator run name to generator run.

• rel – if True, use relative effects. Default is True.

• custom_arm_order – a list of arm names in the order corresponding to how they should be plotted on the x-axis. If not None, this is the default ordering.

• custom_arm_order_name – name for custom ordering to show in the ordering dropdown. Default is ‘Custom’.

• show_CI – if True, render confidence intervals.

• data_selector – Function for selecting observations for plotting.

ax.plot.scatter.plot_multiple_metrics(model: ModelBridge, metric_x: str, metric_y: str, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, rel_x: bool = True, rel_y: bool = True, fixed_features: Optional[ObservationFeatures] = None, data_selector: Optional[Callable[[Observation], bool]] = None, color_parameter: Optional[str] = None, color_metric: Optional[str] = None, **kwargs: Any) → AxPlotConfig[source]¶

Plot raw values or predictions of two metrics for arms.

All arms used in the model are included in the plot. Additional arms can be passed through the generator_runs_dict argument.

Parameters:

• model – model to draw predictions from.

• metric_x – metric to plot on the x-axis.

• metric_y – metric to plot on the y-axis.

• generator_runs_dict – a mapping from generator run name to generator run.

• rel_x – if True, use relative effects on metric_x.

• rel_y – if True, use relative effects on metric_y.

• data_selector – Function for selecting observations for plotting.

• color_parameter – color points according to the specified parameter, cannot be used together with color_metric.

• color_metric – color points according to the specified metric, cannot be used together with color_parameter.

ax.plot.scatter.plot_objective_vs_constraints(model: ModelBridge, objective: str, subset_metrics: Optional[List[str]] = None, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, rel: bool = True, infer_relative_constraints: Optional[bool] = False, fixed_features: Optional[ObservationFeatures] = None, data_selector: Optional[Callable[[Observation], bool]] = None, color_parameter: Optional[str] = None, color_metric: Optional[str] = None, label_dict: Optional[Dict[str, str]] = None) → AxPlotConfig[source]¶

Plot the tradeoff between an objective and all other metrics in a model.

All arms used in the model are included in the plot. Additional arms can be passed through via the generator_runs_dict argument.

Fixed features input can be used to override fields of the insample arms when making model predictions.

Parameters:

• model – model to draw predictions from.

• objective – metric to optimize. Plotted on the x-axis.

• subset_metrics – list of metrics to plot on the y-axes if need a subset of all metrics in the model.

• generator_runs_dict – a mapping from generator run name to generator run.

• rel – if True, use relative effects. Default is True.

• infer_relative_constraints – if True, read relative spec from model’s optimization config. Absolute constraints will not be relativized; relative ones will be. Objectives will respect the rel parameter. Metrics that are not constraints will be relativized.

• fixed_features – Fixed features to use when making model predictions.

• data_selector – Function for selecting observations for plotting.

• color_parameter – color points according to the specified parameter, cannot be used together with color_metric.

• color_metric – color points according to the specified metric, cannot be used together with color_parameter.

• label_dict – A dictionary that maps the label to an alias to be used in the plot.

ax.plot.scatter.tile_fitted(model: ModelBridge, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, rel: bool = True, show_arm_details_on_hover: bool = False, show_CI: bool = True, arm_noun: str = 'arm', metrics: Optional[List[str]] = None, fixed_features: Optional[ObservationFeatures] = None, data_selector: Optional[Callable[[Observation], bool]] = None) → AxPlotConfig[source]¶

Tile version of fitted outcome plots.

Parameters:

• model – model to use for predictions.

• generator_runs_dict – a mapping from generator run name to generator run.

• rel – if True, use relative effects. Default is True.

• show_arm_details_on_hover – if True, display parameterizations of arms on hover. Default is False.

• show_CI – if True, render confidence intervals.

• arm_noun – noun to use instead of “arm” (e.g. group)

• metrics – List of metric names to restrict to when plotting.

• fixed_features – Fixed features to use when making model predictions.

• data_selector – Function for selecting observations for plotting.

ax.plot.scatter.tile_observations(experiment: Experiment, data: Optional[Data] = None, rel: bool = True, metrics: Optional[List[str]] = None, arm_names: Optional[List[str]] = None) → AxPlotConfig[source]¶

Tiled plot with all observed outcomes.

Will plot all observed arms. If data is provided will use that, otherwise will fetch data from experiment. Will plot all metrics in data unless a list is provided in metrics. If arm_names is provided will limit the plot to only arms in that list.

Parameters:

• experiment – Experiment

• data – Data to use, otherwise will fetch data from experiment.

• rel – Plot relative values, if experiment has status quo.

• metrics – Limit results to this set of metrics.

• arm_names – Limit results to this set of arms.

Returns: Plot config for the plot.

Slice Plot¶

ax.plot.slice.interact_slice(model: ModelBridge, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, relative: bool = False, density: int = 50, slice_values: Optional[Dict[str, Any]] = None, fixed_features: Optional[ObservationFeatures] = None, trial_index: Optional[int] = None) → AxPlotConfig[source]¶

Create interactive plot with predictions for a 1-d slice of the parameter space.

Parameters:

• model – ModelBridge that contains model for predictions

• generator_runs_dict – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.

• relative – Predictions relative to status quo

• density – Number of points along slice to evaluate predictions.

• slice_values – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters. Ignored if fixed_features is specified.

• fixed_features – An ObservationFeatures object containing the values of features (including non-parameter features like context) to be set in the slice.

Returns:

interactive plot of objective vs. parameter

Return type:

AxPlotConfig

ax.plot.slice.interact_slice_plotly(model: ModelBridge, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, relative: bool = False, density: int = 50, slice_values: Optional[Dict[str, Any]] = None, fixed_features: Optional[ObservationFeatures] = None, trial_index: Optional[int] = None) → Figure[source]¶

Create interactive plot with predictions for a 1-d slice of the parameter space.

Parameters:

• model – ModelBridge that contains model for predictions

• generator_runs_dict – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.

• relative – Predictions relative to status quo

• density – Number of points along slice to evaluate predictions.

• slice_values – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters. Ignored if fixed_features is specified.

• fixed_features – An ObservationFeatures object containing the values of features (including non-parameter features like context) to be set in the slice.

Returns:

interactive plot of objective vs. parameter

Return type:

go.Figure

ax.plot.slice.plot_slice(model: ModelBridge, param_name: str, metric_name: str, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, relative: bool = False, density: int = 50, slice_values: Optional[Dict[str, Any]] = None, fixed_features: Optional[ObservationFeatures] = None, trial_index: Optional[int] = None) → AxPlotConfig[source]¶

Plot predictions for a 1-d slice of the parameter space.

Parameters:

• model – ModelBridge that contains model for predictions

• param_name – Name of parameter that will be sliced

• metric_name – Name of metric to plot

• generator_runs_dict – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.

• relative – Predictions relative to status quo

• density – Number of points along slice to evaluate predictions.

• slice_values – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters. Ignored if fixed_features is specified.

• fixed_features – An ObservationFeatures object containing the values of features (including non-parameter features like context) to be set in the slice.

Returns:

plot of objective vs. parameter value

Return type:

AxPlotConfig

ax.plot.slice.plot_slice_plotly(model: ModelBridge, param_name: str, metric_name: str, generator_runs_dict: Optional[Dict[str, GeneratorRun]] = None, relative: bool = False, density: int = 50, slice_values: Optional[Dict[str, Any]] = None, fixed_features: Optional[ObservationFeatures] = None, trial_index: Optional[int] = None) → Figure[source]¶

Plot predictions for a 1-d slice of the parameter space.

Parameters:

• model – ModelBridge that contains model for predictions

• param_name – Name of parameter that will be sliced

• metric_name – Name of metric to plot

• generator_runs_dict – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.

• relative – Predictions relative to status quo

• density – Number of points along slice to evaluate predictions.

• slice_values – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters. Ignored if fixed_features is specified.

• fixed_features – An ObservationFeatures object containing the values of features (including non-parameter features like context) to be set in the slice.

Returns:

plot of objective vs. parameter value

Return type:

go.Figure

Table¶

ax.plot.table_view.get_color(x: float, ci: float, rel: bool, reverse: bool) → str[source]¶

Determine the color of the table cell.

ax.plot.table_view.table_view_plot(experiment: Experiment, data: Data, use_empirical_bayes: bool = True, only_data_frame: bool = False, arm_noun: str = 'arm')[source]¶

Table of means and confidence intervals.

Table is of the form:

arm	metric_1	metric_2
0_0	mean +- CI	…
0_1	…	…

Trace Plots¶

ax.plot.trace.get_running_trials_per_minute(experiment: Experiment, show_until_latest_end_plus_timedelta: timedelta = datetime.timedelta(seconds=300)) → AxPlotConfig[source]¶

ax.plot.trace.mean_markers_scatter(y: ndarray, marker_color: Tuple[int] = (190, 186, 218), legend_label: str = '', hover_labels: Optional[List[str]] = None) → Scatter[source]¶

Creates a graph object for trace of the mean of the given series across runs, with errorbars.

Parameters:

• y – (r x t) array with results from r runs and t trials.

• trace_color – tuple of 3 int values representing an RGB color. Defaults to light purple.

• legend_label – label for this trace.

• hover_labels – optional, text to show on hover; list where the i-th value corresponds to the i-th value in the value of the y argument.

Returns:

plotly graph object

Return type:

go.Scatter

ax.plot.trace.mean_trace_scatter(y: ndarray, trace_color: Tuple[int] = (128, 177, 211), legend_label: str = 'mean', hover_labels: Optional[List[str]] = None) → Scatter[source]¶

Creates a graph object for trace of the mean of the given series across runs.

Parameters:

• y – (r x t) array with results from r runs and t trials.

• trace_color – tuple of 3 int values representing an RGB color. Defaults to blue.

• legend_label – label for this trace.

• hover_labels – optional, text to show on hover; list where the i-th value corresponds to the i-th value in the value of the y argument.

Returns:

plotly graph object

Return type:

go.Scatter