ax.analysis¶

Analysis¶

class ax.analysis.analysis.Analysis(*args, **kwargs)[source]¶

Bases: Protocol

An Analysis is a class that given either and Experiment, a GenerationStrategy, or both can compute some data intended for end-user consumption. The data is returned to the user in the form of an AnalysisCard which contains the raw data, a blob (the data processed for end-user consumption), and miscellaneous metadata that can be useful for rendering the card or a collection of cards.

The AnalysisCard is a thin wrapper around the raw data and the processed blob; Analyses impose structure on their blob should subclass Analysis. See PlotlyAnalysis for an example which produces cards where the blob is always a Plotly Figure object.

A good pattern to follow when implementing your own Analyses is to configure “settings” (like which parameter or metrics to operate on, or whether to use observed or modeled effects) in your Analyses’ __init__ methods, then to consume these settings in the compute method.

property attributes: dict[str, Any]¶

The attributes the AnalysisCard will be given in compute.

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → AnalysisCard[source]¶

compute_result(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → Result[AnalysisCard, ExceptionE][source]¶

Utility method to compute an AnalysisCard as a Result. This can be useful for computing many Analyses at once and handling Exceptions later.

property name: str¶

The name the AnalysisCard will be given in compute.

class ax.analysis.analysis.AnalysisCard(name: str, title: str, subtitle: str, level: int, df: DataFrame, blob: str, attributes: dict[str, Any] | None = None)[source]¶

Bases: Base

attributes: dict[str, Any]¶

blob: str¶

blob_annotation = 'dataframe'¶

df: DataFrame¶

level: int¶

name: str¶

subtitle: str¶

title: str¶

class ax.analysis.analysis.AnalysisCardLevel(value)[source]¶

Bases: IntEnum

An enumeration.

CRITICAL = 40¶

DEBUG = 0¶

HIGH = 30¶

LOW = 10¶

MID = 20¶

class ax.analysis.analysis.AnalysisE(message: str, exception: Exception, analysis: Analysis)[source]¶

Bases: ExceptionE

analysis: Analysis¶

ax.analysis.analysis.display_cards(cards: Iterable[AnalysisCard], minimum_level: int = AnalysisCardLevel.LOW) → None[source]¶

Display a collection of AnalysisCards in IPython environments (ex. Jupyter).

Parameters:

• cards – Collection of AnalysisCards to display.

• minimum_level – Minimum level of cards to display.

Cross Validation Analysis¶

class ax.analysis.plotly.cross_validation.CrossValidationPlot(metric_name: str | None = None, folds: int = -1, untransform: bool = True, trial_index: int | None = None)[source]¶

Bases: PlotlyAnalysis

Plotly Scatter plot for cross validation for model predictions using the current model on the GenerationStrategy. This plot is useful for understanding how well the model is able to predict out-of-sample which in turn is indicative of its ability to suggest valuable candidates.

Splits the model’s training data into train/test folds and makes out-of-sample predictions on the test folds.

A well fit model will have points clustered around the y=x line, and a model with poor fit may have points in a horizontal band in the center of the plot indicating a tendency to predict the observed mean of the specificed metric for all arms.

The DataFrame computed will contain one row per arm and the following columns:

• arm_name: The name of the arm

• observed: The observed mean of the metric specified

• observed_sem: The SEM of the observed mean of the metric specified

• predicted: The predicted mean of the metric specified

• predicted_sem: The SEM of the predicted mean of the metric specified

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → PlotlyAnalysisCard[source]¶

Markdown Analysis¶

class ax.analysis.markdown.markdown_analysis.MarkdownAnalysis(*args, **kwargs)[source]¶

Bases: Analysis

An Analysis that computes a paragraph of Markdown formatted text.

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → MarkdownAnalysisCard[source]¶

class ax.analysis.markdown.markdown_analysis.MarkdownAnalysisCard(name: str, title: str, subtitle: str, level: int, df: DataFrame, blob: str, attributes: dict[str, Any] | None = None)[source]¶

Bases: AnalysisCard

blob_annotation = 'markdown'¶

get_markdown() → str[source]¶

Plotly Analysis¶

class ax.analysis.plotly.plotly_analysis.PlotlyAnalysis(*args, **kwargs)[source]¶

Bases: Analysis

An Analysis that computes a Plotly figure.

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → PlotlyAnalysisCard[source]¶

class ax.analysis.plotly.plotly_analysis.PlotlyAnalysisCard(name: str, title: str, subtitle: str, level: int, df: DataFrame, blob: str, attributes: dict[str, Any] | None = None)[source]¶

Bases: AnalysisCard

blob_annotation = 'plotly'¶

get_figure() → plotly.graph_objects.Figure[source]¶

Can Generate Candidates Healthcheck Analysis¶

class ax.analysis.healthcheck.can_generate_candidates.CanGenerateCandidatesAnalysis(can_generate_candidates: bool, reason: str, days_till_fail: int)[source]¶

Bases: HealthcheckAnalysis

LAST_RUN_TEMPLATE: str = '\n\nLAST TRIAL RUN: {days} day(s) ago'¶

REASON_PREFIX: str = 'This experiment cannot generate candidates.\nREASON: '¶

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → HealthcheckAnalysisCard[source]¶

Should Generate Candidates Healthcheck Analysis¶

class ax.analysis.healthcheck.should_generate_candidates.ShouldGenerateCandidates(should_generate: bool, reason: str, trial_index: int)[source]¶

Bases: HealthcheckAnalysis

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → HealthcheckAnalysisCard[source]¶

Healthcheck Analysis¶

class ax.analysis.healthcheck.healthcheck_analysis.HealthcheckAnalysis(*args, **kwargs)[source]¶

Bases: Analysis

An analysis that performs a health check.

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → HealthcheckAnalysisCard[source]¶

class ax.analysis.healthcheck.healthcheck_analysis.HealthcheckAnalysisCard(name: str, title: str, subtitle: str, level: int, df: DataFrame, blob: str, attributes: dict[str, Any] | None = None)[source]¶

Bases: AnalysisCard

blob_annotation = 'healthcheck'¶

get_status() → HealthcheckStatus[source]¶

class ax.analysis.healthcheck.healthcheck_analysis.HealthcheckStatus(value)[source]¶

Bases: IntEnum

An enumeration.

FAIL = 1¶

PASS = 0¶

WARNING = 2¶

InSample Effects Analysis¶

class ax.analysis.plotly.arm_effects.insample_effects.InSampleEffectsPlot(metric_name: str, trial_index: int, use_modeled_effects: bool)[source]¶

Bases: PlotlyAnalysis

Plotly Insample Effects plot for a single metric on a single trial, with one point per unique arm across all trials. The plot may either use modeled effects, or raw / observed data.

This plot is useful for understanding how arms compare to eachother for a given metric.

TODO: Allow trial index to be optional so we can plot all trials for non batch experiments.

The DataFrame computed will contain one row per arm and the following columns:

• source: In-sample or model key that geneerated the candidate

• arm_name: The name of the arm

• mean: The observed or predicted mean of the metric specified

• sem: The observed or predicted sem of the metric specified

• error_margin: The 95% CI of the metric specified for the arm

• size_column: The size of the circle in the plot, which represents

  the probability that the arm is feasible (does not violate any constraints).

• parameters: A string representation of the parameters for the arm

  to be viewed in the tooltip.

• constraints_violated: A string representation of the probability

  each constraint is violated for the arm, to be viewed in the tooltip.

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → PlotlyAnalysisCard[source]¶

property name: str¶

The name the AnalysisCard will be given in compute.

Parallel Coordinates Analysis¶

class ax.analysis.plotly.parallel_coordinates.ParallelCoordinatesPlot(metric_name: str | None = None)[source]¶

Bases: PlotlyAnalysis

Plotly Parcoords plot for a single metric, with one line per arm and dimensions for each parameter in the search space. This plot is useful for understanding how thoroughly the search space is explored as well as for identifying if there is any clusertering for either good or bad parameterizations.

The DataFrame computed will contain one row per arm and the following columns:

• arm_name: The name of the arm

• METRIC_NAME: The observed mean of the metric specified

• **PARAMETER_NAME: The value of said parameter for the arm, for each parameter

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → PlotlyAnalysisCard[source]¶

Predicted Effects Analysis¶

class ax.analysis.plotly.arm_effects.predicted_effects.PredictedEffectsPlot(metric_name: str)[source]¶

Bases: PlotlyAnalysis

Plotly Predicted Effecs plot for a single metric, with one point per unique arm across all trials. It plots all observed points, as well as predictions for the most recently generated trial.

This plot is useful for understanding how arms in a candidate trial can be expected to perform.

The DataFrame computed will contain one row per arm and the following columns:

• source: In-sample or model key that geneerated the candidate

• arm_name: The name of the arm

• mean: The observed or predicted mean of the metric specified

• sem: The observed or predicted sem of the metric specified

• error_margin: The 95% CI of the metric specified for the arm

• size_column: The size of the circle in the plot, which represents

  the probability that the arm is feasible (does not violate any constraints).

• parameters: A string representation of the parameters for the arm

  to be viewed in the tooltip.

• constraints_violated: A string representation of the probability

  each constraint is violated for the arm, to be viewed in the tooltip.

CARD_NAME = 'PredictedEffectsPlot'¶

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → PlotlyAnalysisCard[source]¶

trial_index: int | None = None¶

Plotly Arm Effects Utils¶

ax.analysis.plotly.arm_effects.utils.format_parameters_for_effects_by_arm_plot(parameters: dict[str, None | str | bool | float | int], max_num_params: int = 5) → str[source]¶

Format the parameters for tooltips in the predicted or insample effects plot.

ax.analysis.plotly.arm_effects.utils.get_predictions_by_arm(model: ModelBridge, metric_name: str, outcome_constraints: list[OutcomeConstraint], gr: GeneratorRun | None = None) → list[dict[str, Any]][source]¶

ax.analysis.plotly.arm_effects.utils.prepare_arm_effects_plot(df: DataFrame, metric_name: str, outcome_constraints: list[OutcomeConstraint]) → plotly.graph_objects.Figure[source]¶

Prepare a plotly figure for the predicted effects based on the data in df.

Parameters:

• metric_name – The name of the metric to plot.

• outcome_constraints – The outcome constraints for the experiment used to determine if the metric is a constraint, and if so, what the bound is so the bound can be rendered in the plot.

• df –

  A dataframe of data to plot with the following columns: - source: In-sample or model key that geneerated the candidate - arm_name: The name of the arm - mean: The observed or predicted mean of the metric specified - sem: The observed or predicted sem of the metric specified - error_margin: The 95% CI of the metric specified for the arm - size_column: The size of the circle in the plot, which represents

  the probability that the arm is feasible (does not violate any constraints).

  • parameters: A string representation of the parameters for the arm

    to be viewed in the tooltip.

  • constraints_violated: A string representation of the probability

    each constraint is violated for the arm, to be viewed in the tooltip.

Plotly Analysis Utils¶

ax.analysis.plotly.utils.format_constraint_violated_probabilities(constraints_violated: dict[str, float]) → str[source]¶

Format the constraints violated for the tooltip.

ax.analysis.plotly.utils.get_constraint_violated_probabilities(predictions: list[tuple[dict[str, float], dict[str, float]]], outcome_constraints: list[OutcomeConstraint]) → dict[str, list[float]][source]¶

Get the probability that each arm violates the outcome constraints.

Parameters:

• predictions – List of predictions for each observation feature generated by predict_at_point. It should include predictions for all outcome constraint metrics.

• outcome_constraints – List of outcome constraints to check.

Returns:

A dict of probabilities that each arm violates the outcome constraint provided, and for “any_constraint_violated” the probability that the arm violates any outcome constraint provided.

ax.analysis.plotly.utils.is_predictive(model: ModelBridge) → bool[source]¶

Check if a model is predictive. Basically, we’re checking if predict() is implemented.

NOTE: This does not mean it’s capable of out of sample prediction.

ax.analysis.plotly.utils.select_metric(experiment: Experiment) → str[source]¶

Select the most relevant metric to plot from an Experiment.

Scatter Plot Analysis¶

class ax.analysis.plotly.scatter.ScatterPlot(x_metric_name: str, y_metric_name: str, show_pareto_frontier: bool = False)[source]¶

Bases: PlotlyAnalysis

Plotly Scatter plot for any two metrics. Each arm is represented by a single point, whose color indicates the arm’s trial index. Optionally, the Pareto frontier can be shown. This plot is useful for understanding the relationship and/or tradeoff between two metrics.

The DataFrame computed will contain one row per arm and the following columns:

• trial_index: The trial index of the arm

• arm_name: The name of the arm

• X_METRIC_NAME: The observed mean of the metric specified

• Y_METRIC_NAME: The observed mean of the metric specified

• is_optimal: Whether the arm is on the Pareto frontier

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → PlotlyAnalysisCard[source]¶

Summary¶

class ax.analysis.summary.Summary(omit_empty_columns: bool = True)[source]¶

Bases: Analysis

High-level summary of the Experiment with one row per arm. Any values missing at compute time will be represented as None. Columns where every value is None will be omitted by default.

The DataFrame computed will contain one row per arm and the following columns:

• trial_index: The trial index of the arm

• arm_name: The name of the arm

• status: The status of the trial (e.g. RUNNING, SUCCEDED, FAILED)

• failure_reason: The reason for the failure, if applicable

• generation_method: The model_key of the model that generated the arm

• generation_node: The name of the GenerationNode that generated the arm

• **METADATA: Any metadata associated with the trial, as specified by the

  Experiment’s runner.run_metadata_report_keys field

• **METRIC_NAME: The observed mean of the metric specified, for each metric

• **PARAMETER_NAME: The value of said parameter for the arm, for each parameter

compute(experiment: Experiment | None = None, generation_strategy: GenerationStrategyInterface | None = None) → AnalysisCard[source]¶