ax.analysis

Analysis

class ax.analysis.analysis.Analysis(*args, **kwargs)[source]

Bases: Protocol

An Analysis is a class that given either and Experiment, a GenerationStrategy, or both can compute some data intended for end-user consumption. The data is returned to the user in the form of an AnalysisCard which contains the raw data, a blob (the data processed for end-user consumption), and miscellaneous metadata that can be useful for rendering the card or a collection of cards.

The AnalysisCard is a thin wrapper around the raw data and the processed blob; Analyses impose structure on their blob should subclass Analysis. See PlotlyAnalysis for an example which produces cards where the blob is always a Plotly Figure object.

A good pattern to follow when implementing your own Analyses is to configure “settings” (like which parameter or metrics to operate on, or whether to use observed or modeled effects) in your Analyses’ __init__ methods, then to consume these settings in the compute method.

property attributes: dict[str, Any]

The attributes the AnalysisCard will be given in compute.

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → AnalysisCard[source]
compute_result(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → Result[AnalysisCard, AnalysisE][source]

Utility method to compute an AnalysisCard as a Result. This can be useful for computing many Analyses at once and handling Exceptions later.

property name: str

The name the AnalysisCard will be given in compute.

class ax.analysis.analysis.AnalysisCard(name: str, title: str, subtitle: str, level: int, df: DataFrame, blob: str, attributes: dict[str, Any] | None = None)[source]

Bases: Base

attributes: dict[str, Any]
blob: str
blob_annotation = 'dataframe'
df: DataFrame
level: int
name: str
subtitle: str
title: str
class ax.analysis.analysis.AnalysisCardLevel(value)[source]

Bases: IntEnum

An enumeration.

CRITICAL = 40
DEBUG = 0
HIGH = 30
LOW = 10
MID = 20
class ax.analysis.analysis.AnalysisE(message: str, exception: Exception, analysis: Analysis)[source]

Bases: ExceptionE

analysis: Analysis
ax.analysis.analysis.display_cards(cards: Iterable[AnalysisCard], minimum_level: int = AnalysisCardLevel.LOW) → None[source]

Display a collection of AnalysisCards in IPython environments (ex. Jupyter).

Parameters:

  • cards – Collection of AnalysisCards to display.

  • minimum_level – Minimum level of cards to display.

Cross Validation Analysis

class ax.analysis.plotly.cross_validation.CrossValidationPlot(metric_name: str | None = None, folds: int = -1, untransform: bool = True, trial_index: int | None = None)[source]

Bases: PlotlyAnalysis

Plotly Scatter plot for cross validation for model predictions using the current model on the GenerationStrategy. This plot is useful for understanding how well the model is able to predict out-of-sample which in turn is indicative of its ability to suggest valuable candidates.

Splits the model’s training data into train/test folds and makes out-of-sample predictions on the test folds.

A well fit model will have points clustered around the y=x line, and a model with poor fit may have points in a horizontal band in the center of the plot indicating a tendency to predict the observed mean of the specificed metric for all arms.

The DataFrame computed will contain one row per arm and the following columns:

  • arm_name: The name of the arm

  • observed: The observed mean of the metric specified

  • observed_sem: The SEM of the observed mean of the metric specified

  • predicted: The predicted mean of the metric specified

  • predicted_sem: The SEM of the predicted mean of the metric specified

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → PlotlyAnalysisCard[source]

Markdown Analysis

class ax.analysis.markdown.markdown_analysis.MarkdownAnalysis(*args, **kwargs)[source]

Bases: Analysis

An Analysis that computes a paragraph of Markdown formatted text.

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → MarkdownAnalysisCard[source]
class ax.analysis.markdown.markdown_analysis.MarkdownAnalysisCard(name: str, title: str, subtitle: str, level: int, df: DataFrame, blob: str, attributes: dict[str, Any] | None = None)[source]

Bases: AnalysisCard

blob_annotation = 'markdown'
get_markdown() → str[source]
ax.analysis.markdown.markdown_analysis.markdown_analysis_card_from_analysis_e(analysis_e: AnalysisE) → MarkdownAnalysisCard[source]

Plotly Analysis

class ax.analysis.plotly.plotly_analysis.PlotlyAnalysis(*args, **kwargs)[source]

Bases: Analysis

An Analysis that computes a Plotly figure.

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → PlotlyAnalysisCard[source]
class ax.analysis.plotly.plotly_analysis.PlotlyAnalysisCard(name: str, title: str, subtitle: str, level: int, df: DataFrame, blob: str, attributes: dict[str, Any] | None = None)[source]

Bases: AnalysisCard

blob_annotation = 'plotly'
get_figure() → plotly.graph_objects.Figure[source]

Can Generate Candidates Healthcheck Analysis

class ax.analysis.healthcheck.can_generate_candidates.CanGenerateCandidatesAnalysis(can_generate_candidates: bool, reason: str, days_till_fail: int)[source]

Bases: HealthcheckAnalysis

LAST_RUN_TEMPLATE: str = '\n\nLAST TRIAL RUN: {days} day(s) ago'
REASON_PREFIX: str = 'This experiment cannot generate candidates.\nREASON: '
compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → HealthcheckAnalysisCard[source]

Should Generate Candidates Healthcheck Analysis

class ax.analysis.healthcheck.should_generate_candidates.ShouldGenerateCandidates(should_generate: bool, reason: str, trial_index: int)[source]

Bases: HealthcheckAnalysis

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → HealthcheckAnalysisCard[source]

Healthcheck Analysis

class ax.analysis.healthcheck.healthcheck_analysis.HealthcheckAnalysis(*args, **kwargs)[source]

Bases: Analysis

An analysis that performs a health check.

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → HealthcheckAnalysisCard[source]
class ax.analysis.healthcheck.healthcheck_analysis.HealthcheckAnalysisCard(name: str, title: str, subtitle: str, level: int, df: DataFrame, blob: str, attributes: dict[str, Any] | None = None)[source]

Bases: AnalysisCard

blob_annotation = 'healthcheck'
get_status() → HealthcheckStatus[source]
class ax.analysis.healthcheck.healthcheck_analysis.HealthcheckStatus(value)[source]

Bases: IntEnum

An enumeration.

FAIL = 1
PASS = 0
WARNING = 2

Constraints Feasibility Analysis

class ax.analysis.healthcheck.constraints_feasibility.ConstraintsFeasibilityAnalysis(prob_threshold: float = 0.95)[source]

Bases: HealthcheckAnalysis

Analysis for checking the feasibility of the constraints for the experiment. A constraint is considered feasible if the probability of constraints violation is below the threshold for at least one arm.

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → HealthcheckAnalysisCard[source]

Compute the feasibility of the constraints for the experiment.

Parameters:

  • experiment – Ax experiment.

  • generation_strategy – Ax generation strategy.

  • prob_threhshold – Threshold for the probability of constraint violation. Constraints are considered feasible if the probability of constraint violation is below the threshold for at least one arm.

Returns:

A HealthcheckAnalysisCard object with the information on infeasible metrics, i.e., metrics for which the constraints are infeasible for all test groups (arms).

ax.analysis.healthcheck.constraints_feasibility.constraints_feasibility(optimization_config: OptimizationConfig, model: ModelBridge, prob_threshold: float = 0.99) → Tuple[bool, DataFrame][source]

Check the feasibility of the constraints for the experiment.

Parameters:

  • optimization_config – Ax optimization config.

  • model – Ax model to use for predictions.

  • prob_threshold – Threshold for the probability of constraint violation.

Returns:

A tuple of a boolean indicating whether the constraints are feasible and a dataframe with information on the probabilities of constraints violation for each arm.

Search Space Analysis

class ax.analysis.healthcheck.search_space_analysis.SearchSpaceAnalysis(trial_index: int, boundary_proportion_threshold: float = 0.5)[source]

Bases: HealthcheckAnalysis

Analysis for checking wehther the search space of the experiment should be expanded. It checks whether the suggested parameters land at the boundary of the search space and recommends expanding the search space if the proportion of the suggested parameters that land at the boundary is above the threshold.

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → HealthcheckAnalysisCard[source]
Parameters:

  • experiment – Ax experiment.

  • generation_strategy – Ax generation strategy.

Returns:

A HealthcheckAnalysisCard object with the information on the parameters and parameter constraints whose boundaries are recommended to be expanded.

ax.analysis.healthcheck.search_space_analysis.boundary_proportions_message(boundary_proportions_df: DataFrame, boundary_proportion_threshold: float = 0.5) → str[source]

Construct a message explaning what parameter or parameter constraints bounds to change based on the proportions of the parametrizations that landed at the search spaces boundaries. A proportion should be above the theshold in order to recommend expanding the search space along the corresponding parameter or parameter constraint.

Parameters:

  • boundary_proportions_df –

    A dataframe with the following columns * parameter_or_constraint: the parameter or constraint object

    containing this row’s search space boundary.

    • boundary: a string representation of the function defining this boundary.

    • proportion: the proportion of provided parameterizations within

      tol of this boundary.

    • bound: whether this is an upper or lower bound.

  • parameter (A dataframe containing parameters and) – constraints along with the proportions of the parametrizations that landed at the lower or upper bounds of the parameters or at the constraints boundary.

  • boundary_proportion_threshold – The minimal proportion of suggested parametrizations that land at the boundary of the search space for us to recommend expanding the search space.

Returns:

A string explaning what parameter or parameter constraints bounds to change in order to expand the search space.

ax.analysis.healthcheck.search_space_analysis.search_space_boundary_proportions(search_space: SearchSpace, parametrizations: list[dict[str, None | str | bool | float | int]], tol: float = 1e-06) → DataFrame[source]

Compute the fractions of parametrizations that landed at the parameter and parameter constraint boundaies of the search space.

Parameters:

  • search_space – Search space.

  • parametrizations – A list of suggested parametrizations (parameter values).

  • tol – Relative tolerance for the difference between parameters and the boundary bounds.

Returns:

A dataframe containing parameters along with the fractions of parametrizations that landed at the parameter lower and upper limit in case of range and ordered choice parameters and containing parameter constraints along with the fractions of parametrizations that landed at the constraint boundary.

InSample Effects Analysis

class ax.analysis.plotly.arm_effects.insample_effects.InSampleEffectsPlot(metric_name: str, trial_index: int, use_modeled_effects: bool)[source]

Bases: PlotlyAnalysis

Plotly Insample Effects plot for a single metric on a single trial, with one point per unique arm across all trials. The plot may either use modeled effects, or raw / observed data.

This plot is useful for understanding how arms compare to eachother for a given metric.

TODO: Allow trial index to be optional so we can plot all trials for non batch experiments.

The DataFrame computed will contain one row per arm and the following columns:

  • source: In-sample or model key that geneerated the candidate

  • arm_name: The name of the arm

  • mean: The observed or predicted mean of the metric specified

  • sem: The observed or predicted sem of the metric specified

  • error_margin: The 95% CI of the metric specified for the arm

  • size_column: The size of the circle in the plot, which represents

    the probability that the arm is feasible (does not violate any constraints).

  • parameters: A string representation of the parameters for the arm

    to be viewed in the tooltip.

  • constraints_violated: A string representation of the probability

    each constraint is violated for the arm, to be viewed in the tooltip.

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → PlotlyAnalysisCard[source]
property name: str

The name the AnalysisCard will be given in compute.

Parallel Coordinates Analysis

class ax.analysis.plotly.parallel_coordinates.ParallelCoordinatesPlot(metric_name: str | None = None)[source]

Bases: PlotlyAnalysis

Plotly Parcoords plot for a single metric, with one line per arm and dimensions for each parameter in the search space. This plot is useful for understanding how thoroughly the search space is explored as well as for identifying if there is any clusertering for either good or bad parameterizations.

The DataFrame computed will contain one row per arm and the following columns:

  • arm_name: The name of the arm

  • METRIC_NAME: The observed mean of the metric specified

  • **PARAMETER_NAME: The value of said parameter for the arm, for each parameter

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → PlotlyAnalysisCard[source]

Predicted Effects Analysis

class ax.analysis.plotly.arm_effects.predicted_effects.PredictedEffectsPlot(metric_name: str)[source]

Bases: PlotlyAnalysis

Plotly Predicted Effecs plot for a single metric, with one point per unique arm across all trials. It plots all observed points, as well as predictions for the most recently generated trial.

This plot is useful for understanding how arms in a candidate trial can be expected to perform.

The DataFrame computed will contain one row per arm and the following columns:

  • source: In-sample or model key that geneerated the candidate

  • arm_name: The name of the arm

  • mean: The observed or predicted mean of the metric specified

  • sem: The observed or predicted sem of the metric specified

  • error_margin: The 95% CI of the metric specified for the arm

  • size_column: The size of the circle in the plot, which represents

    the probability that the arm is feasible (does not violate any constraints).

  • parameters: A string representation of the parameters for the arm

    to be viewed in the tooltip.

  • constraints_violated: A string representation of the probability

    each constraint is violated for the arm, to be viewed in the tooltip.

CARD_NAME = 'PredictedEffectsPlot'
compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → PlotlyAnalysisCard[source]
trial_index: int | None = None

Plotly Arm Effects Utils

ax.analysis.plotly.arm_effects.utils.format_parameters_for_effects_by_arm_plot(parameters: dict[str, None | str | bool | float | int], max_num_params: int = 5) → str[source]

Format the parameters for tooltips in the predicted or insample effects plot.

ax.analysis.plotly.arm_effects.utils.get_predictions_by_arm(model: ModelBridge, metric_name: str, outcome_constraints: list[OutcomeConstraint], gr: GeneratorRun | None = None) → list[dict[str, Any]][source]
ax.analysis.plotly.arm_effects.utils.prepare_arm_effects_plot(df: DataFrame, metric_name: str, outcome_constraints: list[OutcomeConstraint]) → plotly.graph_objects.Figure[source]

Prepare a plotly figure for the predicted effects based on the data in df.

Parameters:

  • metric_name – The name of the metric to plot.

  • outcome_constraints – The outcome constraints for the experiment used to determine if the metric is a constraint, and if so, what the bound is so the bound can be rendered in the plot.

  • df –

    A dataframe of data to plot with the following columns: - source: In-sample or model key that geneerated the candidate - arm_name: The name of the arm - mean: The observed or predicted mean of the metric specified - sem: The observed or predicted sem of the metric specified - error_margin: The 95% CI of the metric specified for the arm - size_column: The size of the circle in the plot, which represents

    the probability that the arm is feasible (does not violate any constraints).

    • parameters: A string representation of the parameters for the arm

      to be viewed in the tooltip.

    • constraints_violated: A string representation of the probability

      each constraint is violated for the arm, to be viewed in the tooltip.

Plotly Analysis Utils

ax.analysis.plotly.utils.format_constraint_violated_probabilities(constraints_violated: dict[str, float]) → str[source]

Format the constraints violated for the tooltip.

ax.analysis.plotly.utils.get_constraint_violated_probabilities(predictions: list[tuple[dict[str, float], dict[str, float]]], outcome_constraints: list[OutcomeConstraint]) → dict[str, list[float]][source]

Get the probability that each arm violates the outcome constraints.

Parameters:

  • predictions – List of predictions for each observation feature generated by predict_at_point. It should include predictions for all outcome constraint metrics.

  • outcome_constraints – List of outcome constraints to check.

Returns:

A dict of probabilities that each arm violates the outcome constraint provided, and for “any_constraint_violated” the probability that the arm violates any outcome constraint provided.

ax.analysis.plotly.utils.is_predictive(model: ModelBridge) → bool[source]

Check if a model is predictive. Basically, we’re checking if predict() is implemented.

NOTE: This does not mean it’s capable of out of sample prediction.

ax.analysis.plotly.utils.select_metric(experiment: Experiment) → str[source]

Select the most relevant metric to plot from an Experiment.

Scatter Plot Analysis

class ax.analysis.plotly.scatter.ScatterPlot(x_metric_name: str, y_metric_name: str, show_pareto_frontier: bool = False)[source]

Bases: PlotlyAnalysis

Plotly Scatter plot for any two metrics. Each arm is represented by a single point, whose color indicates the arm’s trial index. Optionally, the Pareto frontier can be shown. This plot is useful for understanding the relationship and/or tradeoff between two metrics.

The DataFrame computed will contain one row per arm and the following columns:

  • trial_index: The trial index of the arm

  • arm_name: The name of the arm

  • X_METRIC_NAME: The observed mean of the metric specified

  • Y_METRIC_NAME: The observed mean of the metric specified

  • is_optimal: Whether the arm is on the Pareto frontier

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → PlotlyAnalysisCard[source]

Summary

class ax.analysis.summary.Summary(omit_empty_columns: bool = True)[source]

Bases: Analysis

High-level summary of the Experiment with one row per arm. Any values missing at compute time will be represented as None. Columns where every value is None will be omitted by default.

The DataFrame computed will contain one row per arm and the following columns:

  • trial_index: The trial index of the arm

  • arm_name: The name of the arm

  • status: The status of the trial (e.g. RUNNING, SUCCEDED, FAILED)

  • failure_reason: The reason for the failure, if applicable

  • generation_method: The model_key of the model that generated the arm

  • generation_node: The name of the GenerationNode that generated the arm

  • **METADATA: Any metadata associated with the trial, as specified by the

    Experiment’s runner.run_metadata_report_keys field

  • **METRIC_NAME: The observed mean of the metric specified, for each metric

  • **PARAMETER_NAME: The value of said parameter for the arm, for each parameter

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → AnalysisCard[source]

Interaction

class ax.analysis.plotly.interaction.InteractionPlot(metric_name: str | None = None, fit_interactions: bool = True, most_important: bool = True, seed: int = 0, torch_device: device | None = None)[source]

Bases: PlotlyAnalysis

Analysis class which tries to explain the data of an experiment as one- or two- dimensional additive components with a level of sparsity in the components. The relative importance of each component is quantified by its Sobol index. Each component may be visualized through slice or contour plots depending on if it is a first order or second order component, respectively.

The DataFrame computed will contain just the sensitivity analyisis with one row per parameter and the following columns:

  • feature: The name of the first or second order component

  • sensitivity: The sensitivity of the component

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → PlotlyAnalysisCard[source]

Compute Sobol index sensitivity for one metric of an experiment. Sensitivity is comptuted by component, where a compoent may be either one variable (main effect) or two variables (interaction effect). The sensitivity is computed using a model fit with an OAK kernel, which decomposes the objective to be a sum of components, and where marginal effects can be computed accurately.

Contour

class ax.analysis.plotly.surface.contour.ContourPlot(x_parameter_name: str, y_parameter_name: str, metric_name: str | None = None)[source]

Bases: PlotlyAnalysis

Plot a 2D surface of the surrogate model’s predicted outcomes for a given pair of parameters, where all other parameters are held fixed at their status-quo value or mean if no status quo is available.

The DataFrame computed will contain the following columns:

  • PARAMETER_NAME: The value of the x parameter specified

  • PARAMETER_NAME: The value of the y parameter specified

  • METRIC_NAME: The predected mean of the metric specified

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → PlotlyAnalysisCard[source]

Slice

class ax.analysis.plotly.surface.slice.SlicePlot(parameter_name: str, metric_name: str | None = None)[source]

Bases: PlotlyAnalysis

Plot a 1D “slice” of the surrogate model’s predicted outcomes for a given parameter, where all other parameters are held fixed at their status-quo value or mean if no status quo is available.

The DataFrame computed will contain the following columns:

  • PARAMETER_NAME: The value of the parameter specified

  • METRIC_NAME_mean: The predected mean of the metric specified

  • METRIC_NAME_sem: The predected sem of the metric specified

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → PlotlyAnalysisCard[source]

Utils

ax.analysis.plotly.surface.utils.get_parameter_values(parameter: Parameter, density: int = 100) → list[None | str | bool | float | int][source]

Get a list of parameter values to predict over for a given parameter.

ax.analysis.plotly.surface.utils.is_axis_log_scale(parameter: Parameter) → bool[source]

Check if the parameter is log scale.

ax.analysis.plotly.surface.utils.select_fixed_value(parameter: Parameter) → None | str | bool | float | int[source]

Select a fixed value for a parameter. Use mean for RangeParameter, “middle” value for ChoiceParameter, and value for FixedParameter.

Utils

ax.analysis.utils.choose_analyses(experiment: Experiment) → list[Analysis][source]

Choose a default set of Analyses to compute based on the current state of the Experiment.