ax.core¶

AbstractData¶

class ax.core.abstract_data.AbstractData(description: Optional[str] = None)[source]

Bases: abc.ABC, ax.utils.common.base.Base

Abstract Base Class for storing data for an experiment.

abstract static from_multiple_data(data: Iterable[ax.core.abstract_data.AbstractData], subset_metrics: Optional[Iterable[str]] = None) → ax.core.abstract_data.AbstractData[source]

Combines multiple data objects into one (with the concatenated underlying dataframe).

NOTE: if one or more data objects in the iterable is of a custom subclass of this Data type, object of that class will be returned. If the iterable contains incompatible types of Data, an error will be raised.

Parameters

• data – Iterable of data objects to combine.

• subset_metrics – If specified, combined data will only contain metrics that appear in this iterable.

abstract property metric_names

Set of metrics contained in this data.

abstract to_observation_data() → List[ObservationData][source]

Convert to ObservationData

class ax.core.abstract_data.AbstractDataFrameData(df: Optional[pandas.DataFrame] = None, description: Optional[str] = None)[source]

Bases: ax.core.abstract_data.AbstractData, ax.utils.common.base.Base

Abstract Base Class for storing DataFrame-backed Data for an experiment.

df

DataFrame with underlying data, and required columns.

description

Human-readable description of data.

COLUMN_DATA_TYPES = {'arm_name': <class 'str'>, 'mean': <class 'numpy.float64'>, 'metric_name': <class 'str'>, 'sem': <class 'numpy.float64'>, 'trial_index': <class 'numpy.int64'>}

REQUIRED_COLUMNS = {}

classmethod column_data_types(extra_column_types: Optional[Dict[str, Type]] = None) → Dict[str, Type][source]

Type specification for all supported columns.

copy_structure_with_df(df: pandas.DataFrame) → ax.core.abstract_data.AbstractDataFrameData[source]

Serialize the structural properties needed to initialize this Data. Used for storage and to help construct new similar Data. All kwargs other than “dataframe” and “description” are considered structural.

classmethod deserialize_init_args(args: Dict[str, Any]) → Dict[str, Any][source]

Given a dictionary, extract the properties needed to initialize the metric. Used for storage.

property df

Return a flattened DataFrame representation of this data’s metrics.

property df_hash

Compute hash of pandas DataFrame.

This first serializes the DataFrame and computes the md5 hash on the resulting string. Note that this may cause performance issue for very large DataFrames.

Parameters

df – The DataFrame for which to compute the hash.

Returns

str: The hash of the DataFrame.

get_filtered_results(**filters: Dict[str, Any]) → pandas.DataFrame[source]

Return filtered subset of data.

Parameters

filter – Column names and values they must match.

Returns

df: The filtered DataFrame.

property metric_names

Set of metric names that appear in the underlying dataframe of this object.

classmethod required_columns() → Set[str][source]

Names of required columns.

classmethod serialize_init_args(data: ax.core.abstract_data.AbstractDataFrameData) → Dict[str, Any][source]

Serialize the class-dependent properties needed to initialize this Data. Used for storage and to help construct new similar Data. All kwargs other than “dataframe” and “description” are considered structural.

classmethod supported_columns(extra_column_names: Optional[Iterable[str]] = None) → Set[str][source]

Names of columns supported (but not necessarily required) by this class.

to_observation_data() → List[ObservationData][source]

Convert to ObservationData

Arm¶

class ax.core.arm.Arm(parameters: Dict[str, Optional[Union[str, bool, float, int]]], name: Optional[str] = None)[source]¶

Bases: ax.utils.common.base.SortableBase

Base class for defining arms.

Randomization in experiments assigns units to a given arm. Thus, the arm encapsulates the parametrization needed by the unit.

clone(clear_name: bool = False) → ax.core.arm.Arm[source]¶

Create a copy of this arm.

Parameters

clear_name – whether this cloned copy should set its name to None instead of the name of the arm being cloned. Defaults to False.

property has_name¶

Return true if arm’s name is not None.

static md5hash(parameters: Dict[str, Optional[Union[str, bool, float, int]]]) → str[source]¶

Return unique identifier for arm’s parameters.

Parameters

parameters – Parameterization; mapping of param name to value.

Returns

Hash of arm’s parameters.

property name¶

Get arm name. Throws if name is None.

property name_or_short_signature¶

Returns arm name if exists; else last 4 characters of the hash.

Used for presentation of candidates (e.g. plotting and tables), where the candidates do not yet have names (since names are automatically set upon addition to a trial).

property parameters¶

Get mapping from parameter names to values.

property signature¶

Get unique representation of a arm.

BaseTrial¶

class ax.core.base_trial.BaseTrial(experiment: core.experiment.Experiment, trial_type: Optional[str] = None, ttl_seconds: Optional[int] = None, index: Optional[int] = None)[source]¶

Bases: abc.ABC, ax.utils.common.base.SortableBase

Base class for representing trials.

Trials are containers for arms that are deployed together. There are two kinds of trials: regular Trial, which only contains a single arm, and BatchTrial, which contains an arbitrary number of arms.

Parameters

• experiment – Experiment, of which this trial is a part

• trial_type – Type of this trial, if used in MultiTypeExperiment.

• ttl_seconds – If specified, trials will be considered failed after this many seconds since the time the trial was ran, unless the trial is completed before then. Meant to be used to detect ‘dead’ trials, for which the evaluation process might have crashed etc., and which should be considered failed after their ‘time to live’ has passed.

• index – If specified, the trial’s index will be set accordingly. This should generally not be specified, as in the index will be automatically determined based on the number of existing trials. This is only used for the purpose of loading from storage.

abstract property abandoned_arms¶

All abandoned arms, associated with this trial.

property abandoned_reason¶

abstract property arms¶

abstract property arms_by_name¶

assign_runner() → ax.core.base_trial.BaseTrial[source]¶

Assigns default experiment runner if trial doesn’t already have one.

complete(reason: Optional[str] = None) → ax.core.base_trial.BaseTrial[source]¶

Stops the trial if functionality is defined on runner

and marks trial completed.

Parameters

reason – A message containing information why the trial is to be completed.

Returns

The trial instance.

property completed_successfully¶

Checks if trial status is COMPLETED.

property deployed_name¶

Name of the experiment created in external framework.

This property is derived from the name field in run_metadata.

property did_not_complete¶

Checks if trial status is terminal, but not COMPLETED.

property experiment¶

The experiment this trial belongs to.

fetch_data(metrics: Optional[List[ax.core.metric.Metric]] = None, **kwargs: Any) → ax.core.abstract_data.AbstractDataFrameData[source]¶

Fetch data for this trial for all metrics on experiment.

Parameters

• trial_index – The index of the trial to fetch data for.

• metrics – If provided, fetch data for these metrics instead of the ones defined on the experiment.

• kwargs – keyword args to pass to underlying metrics’ fetch data functions.

Returns

Data for this trial.

abstract property generator_runs¶

All generator runs associated with this trial.

property index¶

The index of this trial within the experiment’s trial list.

property is_abandoned¶

Whether this trial is abandoned.

lookup_data(keep_latest_map_values_only: bool = True) → ax.core.abstract_data.AbstractDataFrameData[source]¶

Lookup cached data on experiment for this trial.

Parameters

keep_latest_map_values_only – If true, then if Data is an instance of MapData, we keep only the latest value for each map key. This way, the returned dataframe will only contain one row for each trial, arm, and metric.

Returns

If not merging across timestamps, the latest Data object associated with the trial. If merging, all data for trial, merged.

mark_abandoned(reason: Optional[str] = None) → ax.core.base_trial.BaseTrial[source]¶

Mark trial as abandoned.

NOTE: Arms in abandoned trials are considered to be ‘pending points’ in experiment after their abandonment to avoid Ax models suggesting the same arm again as a new candidate. Arms in abandoned trials are also excluded from model training data unless fit_abandoned option is specified to model bridge.

Parameters

abandoned_reason – The reason the trial was abandoned.

Returns

The trial instance.

mark_arm_abandoned(arm_name: str, reason: Optional[str] = None) → ax.core.base_trial.BaseTrial[source]¶

mark_as(status: ax.core.base_trial.TrialStatus, **kwargs: Any) → ax.core.base_trial.BaseTrial[source]¶

Mark trial with a new TrialStatus.

Parameters

• status – The new status of the trial.

• kwargs – Additional keyword args, as can be ued in the respective mark_ methods associated with the trial status.

Returns

The trial instance.

mark_completed() → ax.core.base_trial.BaseTrial[source]¶

Mark trial as completed.

Returns

The trial instance.

mark_early_stopped() → ax.core.base_trial.BaseTrial[source]¶

Mark trial as early stopped.

Returns

The trial instance.

mark_failed() → ax.core.base_trial.BaseTrial[source]¶

Mark trial as failed.

Returns

The trial instance.

mark_running(no_runner_required: bool = False) → ax.core.base_trial.BaseTrial[source]¶

Mark trial has started running.

Returns

The trial instance.

mark_staged() → ax.core.base_trial.BaseTrial[source]¶

Mark the trial as being staged for running.

Returns

The trial instance.

run() → ax.core.base_trial.BaseTrial[source]¶

Deploys the trial according to the behavior on the runner.

The runner returns a run_metadata dict containining metadata of the deployment process. It also returns a deployed_name of the trial within the system to which it was deployed. Both these fields are set on the trial.

Returns

The trial instance.

property run_metadata¶

Dict containing metadata from the deployment process.

This is set implicitly during trial.run().

property runner¶

The runner object defining how to deploy the trial.

property status¶

The status of the trial in the experimentation lifecycle.

stop(new_status: ax.core.base_trial.TrialStatus, reason: Optional[str] = None) → ax.core.base_trial.BaseTrial[source]¶

Stops the trial according to the behavior on the runner.

The runner returns a stop_metadata dict containining metadata of the stopping process.

Parameters

• new_status – The new TrialStatus. Must be one of {TrialStatus.COMPLETED, TrialStatus.ABANDONED, TrialStatus.EARLY_STOPPED}

• reason – A message containing information why the trial is to be stopped.

Returns

The trial instance.

property stop_metadata¶

Dict containing metadata from the stopping process.

This is set implicitly during trial.stop().

property time_completed¶

Completion time of the trial.

property time_created¶

Creation time of the trial.

property time_run_started¶

Time the trial was started running (i.e. collecting data).

property time_staged¶

Staged time of the trial.

property trial_type¶

The type of the trial.

Relevant for experiments containing different kinds of trials (e.g. different deployment types).

property ttl_seconds¶

This trial’s time-to-live once ran, in seconds. If not set, trial will never be automatically considered failed (i.e. infinite TTL). Reflects after how many seconds since the time the trial was run it will be considered failed unless completed.

update_run_metadata(metadata: Dict[str, Any]) → Dict[str, Any][source]¶

Updates the run metadata dict stored on this trial and returns the updated dict.

update_stop_metadata(metadata: Dict[str, Any]) → Dict[str, Any][source]¶

Updates the stop metadata dict stored on this trial and returns the updated dict.

class ax.core.base_trial.TrialStatus(value)[source]¶

Bases: int, enum.Enum

Enum of trial status.

General lifecycle of a trial is::

CANDIDATE --> STAGED --> RUNNING --> COMPLETED
          ------------->         --> FAILED (retryable)
                                 --> EARLY_STOPPED (deemed unpromising)
          -------------------------> ABANDONED (non-retryable)

Trial is marked as a CANDIDATE immediately upon its creation.

Trials may be abandoned at any time prior to completion or failure. The difference between abandonment and failure is that the FAILED state is meant to express a possibly transient or retryable error, so trials in that state may be re-run and arm(s) in them may be resuggested by Ax models to be added to new trials.

ABANDONED trials on the other end, indicate that the trial (and arms(s) in it) should not be rerun or added to new trials. A trial might be marked ABANDONED as a result of human-initiated action (if some trial in experiment is poorly-performing, deterministically failing etc., and should not be run again in the experiment). It might also be marked ABANDONED in an automated way if the trial’s execution encounters an error that indicates that the arm(s) in the trial should bot be evaluated in the experiment again (e.g. the parameterization in a given arm deterministically causes trial evaluation to fail). Note that it’s also possible to abandon a single arm in a BatchTrial via batch.mark_arm_abandoned.

Early-stopped refers to trials that were deemed unpromising by an early-stopping strategy and therefore terminated.

Additionally, when trials are deployed, they may be in an intermediate staged state (e.g. scheduled but waiting for resources) or immediately transition to running. Note that STAGED trial status is not always applicable and depends on the Runner trials are deployed with (and whether a Runner is present at all; for example, in Ax Service API, trials are marked as RUNNING immediately when generated from get_next_trial, skipping the STAGED status).

NOTE: Data for abandoned trials (or abandoned arms in batch trials) is not passed to the model as part of training data, unless fit_abandoned option is specified to model bridge.

ABANDONED = 5¶

CANDIDATE = 0¶

COMPLETED = 3¶

DISPATCHED = 6¶

EARLY_STOPPED = 7¶

FAILED = 2¶

RUNNING = 4¶

STAGED = 1¶

property expecting_data¶

True if trial is expecting data.

property is_abandoned¶

True if this trial is an abandoned one.

property is_candidate¶

True if this trial is a candidate.

property is_completed¶

True if this trial is a successfully completed one.

property is_deployed¶

True if trial has been deployed but not completed.

property is_early_stopped¶

True if this trial is an early stopped one.

property is_failed¶

True if this trial is a failed one.

property is_running¶

True if this trial is a running one.

property is_terminal¶

True if trial is completed.

ax.core.base_trial.immutable_once_run(func: Callable) → Callable[source]¶

Decorator for methods that should throw Error when trial is running or has ever run and immutable.

BatchTrial¶

class ax.core.batch_trial.AbandonedArm(name: str, time: datetime.datetime, reason: Optional[str] = None)[source]¶

Bases: ax.utils.common.base.SortableBase

Class storing metadata of arm that has been abandoned within a BatchTrial.

name: str¶

reason: Optional[str] = None¶

time: datetime.datetime¶

class ax.core.batch_trial.BatchTrial(experiment: core.experiment.Experiment, generator_run: Optional[GeneratorRun] = None, trial_type: Optional[str] = None, optimize_for_power: Optional[bool] = False, ttl_seconds: Optional[int] = None, index: Optional[int] = None)[source]¶

Bases: ax.core.base_trial.BaseTrial

Batched trial that has multiple attached arms, meant to be deployed and evaluated together, and possibly arm weights, which are a measure of how much of the total resources allocated to evaluating a batch should go towards evaluating the specific arm. For instance, for field experiments the weights could describe the fraction of the total experiment population assigned to the different treatment arms. Interpretation of the weights is defined in Runner.

NOTE: A BatchTrial is not just a trial with many arms; it is a trial, for which it is important that the arms are evaluated simultaneously, e.g. in an A/B test where the evaluation results are subject to nonstationarity. For cases where multiple arms are evaluated separately and independently of each other, use multiple Trial objects with a single arm each.

Parameters

• experiment – Experiment, to which this trial is attached

• generator_run – GeneratorRun, associated with this trial. This can a also be set later through add_arm or add_generator_run, but a trial’s associated generator run is immutable once set.

• trial_type – Type of this trial, if used in MultiTypeExperiment.

• optimize_for_power – Whether to optimize the weights of arms in this trial such that the experiment’s power to detect effects of certain size is as high as possible. Refer to documentation of BatchTrial.set_status_quo_and_optimize_power for more detail.

• ttl_seconds – If specified, trials will be considered failed after this many seconds since the time the trial was ran, unless the trial is completed before then. Meant to be used to detect ‘dead’ trials, for which the evaluation process might have crashed etc., and which should be considered failed after their ‘time to live’ has passed.

• index – If specified, the trial’s index will be set accordingly. This should generally not be specified, as in the index will be automatically determined based on the number of existing trials. This is only used for the purpose of loading from storage.

property abandoned_arm_names¶

Set of names of arms that have been abandoned within this trial.

property abandoned_arms¶

List of arms that have been abandoned within this trial.

property abandoned_arms_metadata¶

add_arm(*args, **kwargs)¶

add_arms_and_weights(*args, **kwargs)¶

add_generator_run(*args, **kwargs)¶

property arm_weights¶

The set of arms and associated weights for the trial.

These are constructed by merging the arms and weights from each generator run that is attached to the trial.

property arms¶

All arms contained in the trial.

property arms_by_name¶

Map from arm name to object for all arms in trial.

clone() → ax.core.batch_trial.BatchTrial[source]¶

Clone the trial.

Returns

A new instance of the trial.

property experiment¶

The experiment this batch belongs to.

property generator_run_structs¶

List of generator run structs attached to this trial.

Struct holds generator_run object and the weight with which it was added.

property generator_runs¶

All generator runs associated with this trial.

property index¶

The index of this batch within the experiment’s batch list.

property is_factorial¶

Return true if the trial’s arms are a factorial design with no linked factors.

mark_arm_abandoned(arm_name: str, reason: Optional[str] = None) → ax.core.batch_trial.BatchTrial[source]¶

Mark a arm abandoned.

Usually done after deployment when one arm causes issues but user wants to continue running other arms in the batch.

NOTE: Abandoned arms are considered to be ‘pending points’ in experiment after their abandonment to avoid Ax models suggesting the same arm again as a new candidate. Abandoned arms are also excluded from model training data unless fit_abandoned option is specified to model bridge.

Parameters

• arm_name – The name of the arm to abandon.

• reason – The reason for abandoning the arm.

Returns

The batch instance.

normalized_arm_weights(total: float = 1, trunc_digits: Optional[int] = None) → MutableMapping[ax.core.arm.Arm, float][source]¶

Returns arms with a new set of weights normalized to the given total.

This method is useful for many runners where we need to normalize weights to a certain total without mutating the weights attached to a trial.

Parameters

• total – The total weight to which to normalize. Default is 1, in which case arm weights can be interpreted as probabilities.

• trunc_digits – The number of digits to keep. If the resulting total weight is not equal to total, re-allocate weight in such a way to maintain relative weights as best as possible.

Returns

Mapping from arms to the new set of weights.

run() → ax.core.batch_trial.BatchTrial[source]¶

Deploys the trial according to the behavior on the runner.

The runner returns a run_metadata dict containining metadata of the deployment process. It also returns a deployed_name of the trial within the system to which it was deployed. Both these fields are set on the trial.

Returns

The trial instance.

set_status_quo_and_optimize_power(*args, **kwargs)¶

set_status_quo_with_weight(*args, **kwargs)¶

property status_quo¶

The control arm for this batch.

unset_status_quo() → None[source]¶

Set the status quo to None.

property weights¶

Weights corresponding to arms contained in the trial.

class ax.core.batch_trial.GeneratorRunStruct(generator_run: ax.core.generator_run.GeneratorRun, weight: float)[source]¶

Bases: ax.utils.common.base.SortableBase

Stores GeneratorRun object as well as the weight with which it was added.

generator_run: ax.core.generator_run.GeneratorRun¶

weight: float¶

Data¶

class ax.core.data.Data(df: Optional[pandas.DataFrame] = None, description: Optional[str] = None)[source]

Bases: ax.core.abstract_data.AbstractDataFrameData

Class storing data for an experiment.

The dataframe is retrieved via the df property. The data can be stored to an external store for future use by attaching it to an experiment using experiment.attach_data() (this requires a description to be set.)

df

DataFrame with underlying data, and required columns.

description

Human-readable description of data.

COLUMN_DATA_TYPES = {'arm_name': <class 'str'>, 'end_time': pandas.Timestamp, 'fidelities': <class 'str'>, 'frac_nonnull': <class 'numpy.float64'>, 'mean': <class 'numpy.float64'>, 'metric_name': <class 'str'>, 'n': <class 'numpy.int64'>, 'random_split': <class 'numpy.int64'>, 'sem': <class 'numpy.float64'>, 'start_time': pandas.Timestamp, 'trial_index': <class 'numpy.int64'>}

REQUIRED_COLUMNS = {'arm_name', 'mean', 'metric_name', 'sem'}

static from_evaluations(evaluations: Dict[str, Dict[str, Union[float, numpy.floating, numpy.integer, Tuple[Union[float, numpy.floating, numpy.integer], Optional[Union[float, numpy.floating, numpy.integer]]]]]], trial_index: int, sample_sizes: Optional[Dict[str, int]] = None, start_time: Optional[int] = None, end_time: Optional[int] = None) → ax.core.data.Data[source]

Convert dict of evaluations to Ax data object.

Parameters

• evaluations – Map from arm name to metric outcomes (itself a mapping of metric names to tuples of mean and optionally a SEM).

• trial_index – Trial index to which this data belongs.

• sample_sizes – Number of samples collected for each arm.

• start_time – Optional start time of run of the trial that produced this data, in milliseconds.

• end_time – Optional end time of run of the trial that produced this data, in milliseconds.

Returns

Ax Data object.

static from_fidelity_evaluations(evaluations: Dict[str, List[Tuple[Dict[str, Optional[Union[str, bool, float, int]]], Dict[str, Union[float, numpy.floating, numpy.integer, Tuple[Union[float, numpy.floating, numpy.integer], Optional[Union[float, numpy.floating, numpy.integer]]]]]]]], trial_index: int, sample_sizes: Optional[Dict[str, int]] = None, start_time: Optional[int] = None, end_time: Optional[int] = None) → ax.core.data.Data[source]

Convert dict of fidelity evaluations to Ax data object.

Parameters

• evaluations – Map from arm name to list of (fidelity, metric outcomes) (where metric outcomes is itself a mapping of metric names to tuples of mean and SEM).

• trial_index – Trial index to which this data belongs.

• sample_sizes – Number of samples collected for each arm.

• start_time – Optional start time of run of the trial that produced this data, in milliseconds.

• end_time – Optional end time of run of the trial that produced this data, in milliseconds.

Returns

Ax Data object.

static from_multiple_data(data: Iterable[ax.core.data.Data], subset_metrics: Optional[Iterable[str]] = None) → ax.core.data.Data[source]

Combines multiple data objects into one (with the concatenated underlying dataframe).

NOTE: if one or more data objects in the iterable is of a custom subclass of Data, object of that class will be returned. If the iterable contains multiple types of Data, an error will be raised.

Parameters

• data – Iterable of Ax Data objects to combine.

• subset_metrics – If specified, combined Data will only contain metrics, names of which appear in this iterable, in the underlying dataframe.

ax.core.data.clone_without_metrics(data: ax.core.data.Data, excluded_metric_names: Iterable[str]) → ax.core.data.Data[source]

Returns a new data object where rows containing the metrics specified by metric_names are filtered out. Used to sanitize data before using it as training data for a model that requires data rectangularity.

Parameters

• data – Original data to clone.

• excluded_metric_names – Metrics to avoid copying

Returns

new version of the original data without specified metrics.

ax.core.data.custom_data_class(column_data_types: Optional[Dict[str, Type]] = None, required_columns: Optional[Set[str]] = None, time_columns: Optional[Set[str]] = None) → Type[ax.core.data.Data][source]

Creates a custom data class with additional columns.

All columns and their designations on the base data class are preserved, the inputs here are appended to the definitions on the base class.

Parameters

• column_data_types – Dict from column name to column type.

• required_columns – Set of additional columns required for this data object.

• time_columns – Set of additional columns to cast to timestamp.

Returns

New data subclass with amended column definitions.

ax.core.data.set_single_trial(data: ax.core.data.Data) → ax.core.data.Data[source]

Returns a new Data object where we set all rows to have the same trial index (i.e. 0). This is meant to be used with our IVW transform, which will combine multiple observations of the same metric.

Experiment¶

class ax.core.experiment.DataType(value)[source]¶

Bases: enum.Enum

An enumeration.

DATA = 1¶

MAP_DATA = 2¶

class ax.core.experiment.Experiment(search_space: ax.core.search_space.SearchSpace, name: Optional[str] = None, optimization_config: Optional[ax.core.optimization_config.OptimizationConfig] = None, tracking_metrics: Optional[List[ax.core.metric.Metric]] = None, runner: Optional[ax.core.runner.Runner] = None, status_quo: Optional[ax.core.arm.Arm] = None, description: Optional[str] = None, is_test: bool = False, experiment_type: Optional[str] = None, properties: Optional[Dict[str, Any]] = None, default_data_type: Optional[ax.core.experiment.DataType] = None)[source]¶

Bases: ax.utils.common.base.Base

Base class for defining an experiment.

add_tracking_metric(metric: ax.core.metric.Metric) → Experiment[source]¶

Add a new metric to the experiment.

Parameters

metric – Metric to be added.

add_tracking_metrics(metrics: List[ax.core.metric.Metric]) → Experiment[source]¶

Add a list of new metrics to the experiment.

If any of the metrics are already defined on the experiment, we raise an error and don’t add any of them to the experiment

Parameters

metrics – Metrics to be added.

property arms_by_name¶

The arms belonging to this experiment, by their name.

property arms_by_signature¶

The arms belonging to this experiment, by their signature.

attach_data(data: ax.core.abstract_data.AbstractDataFrameData, combine_with_last_data: bool = False, overwrite_existing_data: bool = False) → int[source]¶

Attach data to experiment. Stores data in experiment._data_by_trial, to be looked up via experiment.lookup_data_for_trial.

Parameters

• data – Data object to store.

• combine_with_last_data –

  By default, when attaching data, it’s identified by its timestamp, and experiment.lookup_data_for_trial returns data by most recent timestamp. Sometimes, however, we want to combine the data from multiple calls to attach_data into one dataframe. This might be because:

  • We attached data for some metrics at one point and data for

  the rest of the metrics later on. - We attached data for some fidelity at one point and data for another fidelity later one.

  To achieve that goal, set combine_with_last_data to True. In this case, we will take the most recent previously attached data, append the newly attached data to it, attach a new Data object with the merged result, and delete the old one. Afterwards, calls to lookup_data_for_trial will return this new combined data object. This operation will also validate that the newly added data does not contain observations for metrics that already have observations at the same fidelity in the most recent data.

• overwrite_existing_data – By default, we keep around all data that has ever been attached to the experiment. However, if we know that the incoming data contains all the information we need for a given trial, we can replace the existing data for that trial, thereby reducing the amount we need to store in the database.

Returns

Timestamp of storage in millis.

property data_by_trial¶

Data stored on the experiment, indexed by trial index and storage time.

First key is trial index and second key is storage time in milliseconds. For a given trial, data is ordered by storage time, so first added data will appear first in the list.

property default_data_constructor¶

property default_data_type¶

property default_trial_type¶

Default trial type assigned to trials in this experiment.

In the base experiment class this is always None. For experiments with multiple trial types, use the MultiTypeExperiment class.

property experiment_type¶

The type of the experiment.

fetch_data(metrics: Optional[List[ax.core.metric.Metric]] = None, **kwargs: Any) → ax.core.abstract_data.AbstractDataFrameData[source]¶

Fetches data for all trials on this experiment and for either the specified metrics or all metrics currently on the experiment, if metrics argument is not specified.

NOTE: For metrics that are not available while trial is running, the data may be retrieved from cache on the experiment. Data is cached on the experiment via calls to experiment.attach_data and whetner a given metric class is available while trial is running is determined by the boolean returned from its is_available_while_running class method.

Parameters

• metrics – If provided, fetch data for these metrics instead of the ones defined on the experiment.

• kwargs – keyword args to pass to underlying metrics’ fetch data functions.

Returns

Data for the experiment.

fetch_trials_data(trial_indices: Iterable[int], metrics: Optional[List[ax.core.metric.Metric]] = None, **kwargs: Any) → ax.core.abstract_data.AbstractDataFrameData[source]¶

Fetches data for specific trials on the experiment.

NOTE: For metrics that are not available while trial is running, the data may be retrieved from cache on the experiment. Data is cached on the experiment via calls to experiment.attach_data and whetner a given metric class is available while trial is running is determined by the boolean returned from its is_available_while_running class method.

Parameters

• trial_indices – Indices of trials, for which to fetch data.

• metrics – If provided, fetch data for these metrics instead of the ones defined on the experiment.

• kwargs – Keyword args to pass to underlying metrics’ fetch data functions.

Returns

Data for the specific trials on the experiment.

get_trials_by_indices(trial_indices: Iterable[int]) → List[ax.core.base_trial.BaseTrial][source]¶

Grabs trials on this experiment by their indices.

property has_name¶

Return true if experiment’s name is not None.

property immutable_search_space_and_opt_config¶

Boolean representing whether search space and metrics on this experiment are mutable (by default they are).

NOTE: For experiments with immutable search spaces and metrics, generator runs will not store copies of search space and metrics, which improves storage layer performance. Not keeping copies of those on generator runs also disables keeping track of changes to search space and metrics, thereby necessitating that those attributes be immutable on experiment.

property is_simple_experiment¶

Whether this experiment is a regular Experiment or the subclassing SimpleExperiment.

property is_test¶

Get whether the experiment is a test.

lookup_data(trial_indices: Optional[Iterable[int]] = None, keep_latest_map_values_only: bool = True) → ax.core.abstract_data.AbstractDataFrameData[source]¶

Lookup data for all trials on this experiment and for either the specified metrics or all metrics currently on the experiment, if metrics argument is not specified.

Parameters

• trial_indices – Indices of trials, for which to fetch data.

• keep_latest_map_values_only – If true, then if Data is an instance of MapData, we keep only the latest value for each map key. This way, the returned dataframe will only contain one row for each trial, arm, and metric.

Returns

Data for the experiment.

lookup_data_for_trial(trial_index: int, keep_latest_map_values_only: bool = True) → Tuple[ax.core.abstract_data.AbstractDataFrameData, int][source]¶

Lookup stored data for a specific trial.

Returns latest data object, and its storage timestamp, present for this trial. Returns empty data and -1 if no data present.

Parameters

• trial_index – The index of the trial to lookup data for.

• keep_latest_map_values_only – If true, then if Data is an instance of MapData, we keep only the latest value for each map key. This way, the returned dataframe will only contain one row for each trial, arm, and metric.

Returns

The requested data object, and its storage timestamp in milliseconds.

lookup_data_for_ts(timestamp: int) → ax.core.abstract_data.AbstractDataFrameData[source]¶

Collect data for all trials stored at this timestamp.

Useful when many trials’ data was fetched and stored simultaneously and user wants to retrieve same collection of data later.

Can also be used to lookup specific data for a single trial when storage time is known.

Parameters

timestamp – Timestamp in millis at which data was stored.

Returns

Data object with all data stored at the timestamp.

property metrics¶

The metrics attached to the experiment.

property name¶

Get experiment name. Throws if name is None.

new_batch_trial(generator_run: Optional[ax.core.generator_run.GeneratorRun] = None, trial_type: Optional[str] = None, optimize_for_power: Optional[bool] = False, ttl_seconds: Optional[int] = None) → ax.core.batch_trial.BatchTrial[source]¶

Create a new batch trial associated with this experiment.

Parameters

• generator_run – GeneratorRun, associated with this trial. This can a also be set later through add_arm or add_generator_run, but a trial’s associated generator run is immutable once set.

• trial_type – Type of this trial, if used in MultiTypeExperiment.

• optimize_for_power – Whether to optimize the weights of arms in this trial such that the experiment’s power to detect effects of certain size is as high as possible. Refer to documentation of BatchTrial.set_status_quo_and_optimize_power for more detail.

• ttl_seconds – If specified, trials will be considered failed after this many seconds since the time the trial was ran, unless the trial is completed before then. Meant to be used to detect ‘dead’ trials, for which the evaluation process might have crashed etc., and which should be considered failed after their ‘time to live’ has passed.

new_trial(generator_run: Optional[ax.core.generator_run.GeneratorRun] = None, trial_type: Optional[str] = None, ttl_seconds: Optional[int] = None) → ax.core.trial.Trial[source]¶

Create a new trial associated with this experiment.

Parameters

• generator_run – GeneratorRun, associated with this trial. Trial has only one arm attached to it and this generator_run must therefore contain one arm. This arm can also be set later through add_arm or add_generator_run, but a trial’s associated generator run is immutable once set.

• trial_type – Type of this trial, if used in MultiTypeExperiment.

• ttl_seconds – If specified, trials will be considered failed after this many seconds since the time the trial was ran, unless the trial is completed before then. Meant to be used to detect ‘dead’ trials, for which the evaluation process might have crashed etc., and which should be considered failed after their ‘time to live’ has passed.

property num_abandoned_arms¶

How many arms attached to this experiment are abandoned.

property num_trials¶

How many trials are associated with this experiment.

property optimization_config¶

The experiment’s optimization config.

property parameters¶

The parameters in the experiment’s search space.

remove_tracking_metric(metric_name: str) → Experiment[source]¶

Remove a metric that already exists on the experiment.

Parameters

metric_name – Unique name of metric to remove.

reset_runners(runner: ax.core.runner.Runner) → None[source]¶

Replace all candidate trials runners.

Parameters

runner – New runner to replace with.

runner_for_trial(trial: ax.core.base_trial.BaseTrial) → Optional[ax.core.runner.Runner][source]¶

The default runner to use for a given trial.

In the base experiment class, this is always the default experiment runner. For experiments with multiple trial types, use the MultiTypeExperiment class.

property running_trial_indices¶

Indices of running trials, associated with the experiment.

property search_space¶

The search space for this experiment.

When setting a new search space, all parameter names and types must be preserved. However, if no trials have been created, all modifications are allowed.

property status_quo¶

The existing arm that new arms will be compared against.

property sum_trial_sizes¶

Sum of numbers of arms attached to each trial in this experiment.

supports_trial_type(trial_type: Optional[str]) → bool[source]¶

Whether this experiment allows trials of the given type.

The base experiment class only supports None. For experiments with multiple trial types, use the MultiTypeExperiment class.

property time_created¶

Creation time of the experiment.

property trial_indices_by_status¶

Indices of trials associated with the experiment, grouped by trial status.

property trials¶

The trials associated with the experiment.

NOTE: If some trials on this experiment specify their TTL, RUNNING trials will be checked for whether their TTL elapsed during this call. Found past- TTL trials will be marked as FAILED.

property trials_by_status¶

Trials associated with the experiment, grouped by trial status.

property trials_expecting_data¶

the list of all trials for which data has arrived or is expected to arrive.

Type

List[BaseTrial]

update_tracking_metric(metric: ax.core.metric.Metric) → Experiment[source]¶

Redefine a metric that already exists on the experiment.

Parameters

metric – New metric definition.

warm_start_from_old_experiment(old_experiment: ax.core.experiment.Experiment, copy_run_metadata: bool = False, trial_statuses_to_copy: Optional[List[ax.core.base_trial.TrialStatus]] = None) → List[ax.core.trial.Trial][source]¶

Copy all completed trials with data from an old Ax expeirment to this one. This function checks that the parameters of each trial are members of the current experiment’s search_space.

NOTE: Currently only handles experiments with 1-arm Trial-s, not BatchTrial-s as there has not yet been need for support of the latter.

Parameters

• old_experiment – The experiment from which to transfer trials and data

• copy_run_metadata – whether to copy the run_metadata from the old experiment

• trial_statuses_to_copy – All trials with a status in this list will be copied. By default, copies all COMPLETED and ABANDONED trials.

Returns

List of trials successfully copied from old_experiment to this one

GeneratorRun¶

class ax.core.generator_run.ArmWeight(arm: ax.core.arm.Arm, weight: float)[source]¶

Bases: ax.utils.common.base.Base

NamedTuple for tying together arms and weights.

arm: ax.core.arm.Arm¶

weight: float¶

class ax.core.generator_run.GeneratorRun(arms: List[ax.core.arm.Arm], weights: Optional[List[float]] = None, optimization_config: Optional[ax.core.optimization_config.OptimizationConfig] = None, search_space: Optional[ax.core.search_space.SearchSpace] = None, model_predictions: Optional[Tuple[Dict[str, List[float]], Dict[str, Dict[str, List[float]]]]] = None, best_arm_predictions: Optional[Tuple[ax.core.arm.Arm, Optional[Tuple[Dict[str, float], Optional[Dict[str, Dict[str, float]]]]]]] = None, type: Optional[str] = None, fit_time: Optional[float] = None, gen_time: Optional[float] = None, model_key: Optional[str] = None, model_kwargs: Optional[Dict[str, Any]] = None, bridge_kwargs: Optional[Dict[str, Any]] = None, gen_metadata: Optional[Dict[str, Any]] = None, model_state_after_gen: Optional[Dict[str, Any]] = None, generation_step_index: Optional[int] = None, candidate_metadata_by_arm_signature: Optional[Dict[str, Optional[Dict[str, Any]]]] = None)[source]¶

Bases: ax.utils.common.base.SortableBase

An object that represents a single run of a generator.

This object is created each time the gen method of a generator is called. It stores the arms and (optionally) weights that were generated by the run. When we add a generator run to a trial, its arms and weights will be merged with those from previous generator runs that were already attached to the trial.

property arm_signatures¶

Returns signatures of arms generated by this run.

property arm_weights¶

Mapping from arms to weights (order matches order in arms property).

property arms¶

Returns arms generated by this run.

property best_arm_predictions¶

property candidate_metadata_by_arm_signature¶

Retrieves model-produced candidate metadata as a mapping from arm name (for the arm the candidate became when added to experiment) to the metadata dict.

clone() → ax.core.generator_run.GeneratorRun[source]¶

Return a deep copy of a GeneratorRun.

property fit_time¶

property gen_metadata¶

Returns metadata generated by this run.

property gen_time¶

property generator_run_type¶

The type of the generator run.

property index¶

The index of this generator run within a trial’s list of generator run structs. This field is set when the generator run is added to a trial.

property model_predictions¶

property model_predictions_by_arm¶

property optimization_config¶

The optimization config used during generation of this run.

property param_df¶

Constructs a Pandas dataframe with the parameter values for each arm.

Useful for inspecting the contents of a generator run.

Returns

a dataframe with the generator run’s arms.

Return type

pd.DataFrame

property search_space¶

The search used during generation of this run.

property time_created¶

Creation time of the batch.

property weights¶

Returns weights associated with arms generated by this run.

class ax.core.generator_run.GeneratorRunType(value)[source]¶

Bases: enum.Enum

Class for enumerating generator run types.

MANUAL = 1¶

STATUS_QUO = 0¶

ax.core.generator_run.extract_arm_predictions(model_predictions: Tuple[Dict[str, List[float]], Dict[str, Dict[str, List[float]]]], arm_idx: int) → Tuple[Dict[str, float], Optional[Dict[str, Dict[str, float]]]][source]¶

Extract a particular arm from model_predictions.

Parameters

• model_predictions – Mean and Cov for all arms.

• arm_idx – Index of arm in prediction list.

Returns

(mean, cov) for specified arm.

MapData¶

class ax.core.map_data.MapData(df: Optional[pandas.DataFrame] = None, map_keys: Optional[List[str]] = None, description: Optional[str] = None)[source]

Bases: ax.core.abstract_data.AbstractDataFrameData

Class storing mapping-like results for an experiment.

Mapping-like results occur whenever a metric is reported as a collection of results, each element corresponding to a tuple of values.

The simplest case is a sequence. For instance a time series is a mapping from the 1-tuple (timestamp) to (mean, sem) results.

Another example: MultiFidelity results. This is a mapping from (fidelity_feature_1, …, fidelity_feature_n) to (mean, sem) results.

The dataframe is retrieved via the df property. The data can be stored to an external store for future use by attaching it to an experiment using experiment.attach_data() (this requires a description to be set.)

DEDUPLICATE_BY_COLUMNS = ['arm_name', 'metric_name']

REQUIRED_COLUMNS = {'arm_name', 'mean', 'metric_name', 'sem'}

deduplicate_data(keep: str = 'last') → ax.core.map_data.MapData[source]

Deduplicate by arm_name and metric_name, and then drop the map_keys columns entirely.

Parameters

keep – Determines which duplicates (rows that differ by map key) to keep. - first: Drop duplicates except for the first occurrence. - last: Drop duplicates except for the last occurrence. - False: Drop all duplicates.

Returns

Deduplicated MapData object.

static from_map_evaluations(evaluations: Dict[str, List[Tuple[Dict[str, Hashable], Dict[str, Union[float, numpy.floating, numpy.integer, Tuple[Union[float, numpy.floating, numpy.integer], Optional[Union[float, numpy.floating, numpy.integer]]]]]]]], trial_index: int, map_keys: Optional[List[str]] = None) → ax.core.map_data.MapData[source]

Convert dict of mapped evaluations to an Ax MapData object

Parameters

• evaluations – Map from arm name to metric outcomes (itself a mapping of metric names to tuples of mean and optionally a SEM).

• trial_index – Trial index to which this data belongs.

• map_keys – List of all elements of the Tuple that makes up the key in MapData.

Returns

Ax MapData object.

static from_multiple_data(data: Iterable[ax.core.map_data.MapData], subset_metrics: Optional[Iterable[str]] = None) → ax.core.map_data.MapData[source]

Combines multiple data objects into one (with the concatenated underlying dataframe).

NOTE: if one or more data objects in the iterable is of a custom subclass of MapData, object of that class will be returned. If the iterable contains multiple types of Data, an error will be raised.

Parameters

• data – Iterable of Ax MapData objects to combine.

• subset_metrics – If specified, combined MapData will only contain metrics, names of which appear in this iterable, in the underlying dataframe.

property map_key_types

property map_keys

Return the names of fields that together make a map key.

E.g. [“timestamp”] for a timeseries, [“fidelity_param_1”, “fidelity_param_2”] for a multi-fidelity set of results.

update(new_data: ax.core.map_data.MapData) → None[source]

MapMetric¶

class ax.core.map_metric.MapMetric(name: str, lower_is_better: Optional[bool] = None, properties: Optional[Dict[str, Any]] = None)[source]

Bases: ax.core.metric.Metric

Base class for representing metrics that return MapData.

The fetch_trial_data method is the essential method to override when subclassing, which specifies how to retrieve a Metric, for a given trial.

A MapMetric must return a MapData object, which requires (at minimum) the following:

https://ax.dev/api/_modules/ax/core/abstract_data.html#AbstractDataFrameData.required_columns

lower_is_better

Flag for metrics which should be minimized.

properties

Properties specific to a particular metric.

data_constructor

alias of ax.core.map_data.MapData

Metric¶

class ax.core.metric.Metric(name: str, lower_is_better: Optional[bool] = None, properties: Optional[Dict[str, Any]] = None)[source]

Bases: ax.utils.common.base.SortableBase

Base class for representing metrics.

The fetch_trial_data method is the essential method to override when subclassing, which specifies how to retrieve a Metric, for a given trial.

A Metric must return a Data object, which requires (at minimum) the following:

https://ax.dev/api/_modules/ax/core/data.html#Data.required_columns

lower_is_better

Flag for metrics which should be minimized.

properties

Properties specific to a particular metric.

clone() → Metric[source]

Create a copy of this Metric.

classmethod combine_with_last_data() → bool[source]

Indicates whether, when attaching data, we should merge the new dataframe into the most recently attached dataframe.

data_constructor

alias of ax.core.data.Data

classmethod deserialize_init_args(args: Dict[str, Any]) → Dict[str, Any][source]

Given a dictionary, extract the properties needed to initialize the metric. Used for storage.

fetch_experiment_data(experiment: core.experiment.Experiment, **kwargs: Any) → AbstractDataFrameData[source]

Fetch this metric’s data for an experiment.

Default behavior is to fetch data from all trials expecting data and concatenate the results.

classmethod fetch_experiment_data_multi(experiment: core.experiment.Experiment, metrics: Iterable[Metric], trials: Optional[Iterable[core.base_trial.BaseTrial]] = None, **kwargs: Any) → AbstractDataFrameData[source]

Fetch multiple metrics data for an experiment.

Default behavior calls fetch_trial_data_multi for each trial. Subclasses should override to batch data computation across trials + metrics.

property fetch_multi_group_by_metric

Metric class, with which to group this metric in Experiment._metrics_by_class, which is used to combine metrics on experiment into groups and then fetch their data via Metric.fetch_trial_data_multi for each group.

NOTE: By default, this property will just return the class on which it is defined; however, in some cases it is useful to group metrics by their superclass, in which case this property should return that superclass.

fetch_trial_data(trial: core.base_trial.BaseTrial, **kwargs: Any) → AbstractDataFrameData[source]

Fetch data for one trial.

classmethod fetch_trial_data_multi(trial: core.base_trial.BaseTrial, metrics: Iterable[Metric], **kwargs: Any) → AbstractDataFrameData[source]

Fetch multiple metrics data for one trial.

Default behavior calls fetch_trial_data for each metric. Subclasses should override this to trial data computation for multiple metrics.

classmethod is_available_while_running() → bool[source]

Whether metrics of this class are available while the trial is running. Metrics that are not available while the trial is running are assumed to be available only upon trial completion. For such metrics, data is assumed to never change once the trial is completed.

NOTE: If this method returns False, data-fetching via experiment.fetch_data will return the data cached on the experiment (for the metrics of the given class) whenever its available. Data is cached on experiment when attached via experiment.attach_data.

classmethod lookup_or_fetch_experiment_data_multi(experiment: core.experiment.Experiment, metrics: Iterable[Metric], trials: Optional[Iterable[core.base_trial.BaseTrial]] = None, **kwargs: Any) → AbstractDataFrameData[source]

Fetch or lookup (with fallback to fetching) data for given metrics, depending on whether they are available while running.

If metric is available while running, its data can change (and therefore we should always re-fetch it). If metric is available only upon trial completion, its data does not change, so we can look up that data on the experiment and only fetch the data that is not already attached to the experiment.

NOTE: If fetching data for a metrics class that is only available upon trial completion, data fetched in this function (data that was not yet available on experiment) will be attached to experiment.

property name

Get name of metric.

classmethod overwrite_existing_data() → bool[source]

Indicates whether, when attaching data, we should overwrite all previously attached data with the new dataframe.

classmethod serialize_init_args(metric: ax.core.metric.Metric) → Dict[str, Any][source]

Serialize the properties needed to initialize the metric. Used for storage.

MultiTypeExperiment¶

class ax.core.multi_type_experiment.MultiTypeExperiment(name: str, search_space: ax.core.search_space.SearchSpace, default_trial_type: str, default_runner: ax.core.runner.Runner, optimization_config: Optional[ax.core.optimization_config.OptimizationConfig] = None, status_quo: Optional[ax.core.arm.Arm] = None, description: Optional[str] = None, is_test: bool = False, experiment_type: Optional[str] = None, properties: Optional[Dict[str, Any]] = None, default_data_type: Optional[ax.core.experiment.DataType] = None)[source]¶

Bases: ax.core.experiment.Experiment

Class for experiment with multiple trial types.

A canonical use case for this is tuning a large production system with limited evaluation budget and a simulator which approximates evaluations on the main system. Trial deployment and data fetching is separate for the two systems, but the final data is combined and fed into multi-task models.

See the Multi-Task Modeling tutorial for more details.

name¶

Name of the experiment.

description¶

Description of the experiment.

add_tracking_metric(metric: ax.core.metric.Metric, trial_type: str, canonical_name: Optional[str] = None) → ax.core.multi_type_experiment.MultiTypeExperiment[source]¶

Add a new metric to the experiment.

Parameters

• metric – The metric to add.

• trial_type – The trial type for which this metric is used.

• canonical_name – The default metric for which this metric is a proxy.

add_trial_type(trial_type: str, runner: ax.core.runner.Runner) → ax.core.multi_type_experiment.MultiTypeExperiment[source]¶

Add a new trial_type to be supported by this experiment.

Parameters

• trial_type – The new trial_type to be added.

• runner – The default runner for trials of this type.

property default_trial_type¶

Default trial type assigned to trials in this experiment.

property default_trials¶

Return the indicies for trials of the default type.

fetch_data(metrics: Optional[List[ax.core.metric.Metric]] = None, **kwargs: Any) → ax.core.abstract_data.AbstractDataFrameData[source]¶

Fetches data for all trials on this experiment and for either the specified metrics or all metrics currently on the experiment, if metrics argument is not specified.

NOTE: For metrics that are not available while trial is running, the data may be retrieved from cache on the experiment. Data is cached on the experiment via calls to experiment.attach_data and whetner a given metric class is available while trial is running is determined by the boolean returned from its is_available_while_running class method.

Parameters

• metrics – If provided, fetch data for these metrics instead of the ones defined on the experiment.

• kwargs – keyword args to pass to underlying metrics’ fetch data functions.

Returns

Data for the experiment.

property metric_to_trial_type¶

Map metrics to trial types.

Adds in default trial type for OC metrics to custom defined trial types..

remove_tracking_metric(metric_name: str) → ax.core.multi_type_experiment.MultiTypeExperiment[source]¶

Remove a metric that already exists on the experiment.

Parameters

metric_name – Unique name of metric to remove.

reset_runners(runner: ax.core.runner.Runner) → None[source]¶

Replace all candidate trials runners.

Parameters

runner – New runner to replace with.

runner_for_trial(trial: ax.core.base_trial.BaseTrial) → Optional[ax.core.runner.Runner][source]¶

The default runner to use for a given trial.

Looks up the appropriate runner for this trial type in the trial_type_to_runner.

supports_trial_type(trial_type: Optional[str]) → bool[source]¶

Whether this experiment allows trials of the given type.

Only trial types defined in the trial_type_to_runner are allowed.

update_runner(trial_type: str, runner: ax.core.runner.Runner) → ax.core.multi_type_experiment.MultiTypeExperiment[source]¶

Update the default runner for an existing trial_type.

Parameters

• trial_type – The new trial_type to be added.

• runner – The new runner for trials of this type.

update_tracking_metric(metric: ax.core.metric.Metric, trial_type: str, canonical_name: Optional[str] = None) → ax.core.multi_type_experiment.MultiTypeExperiment[source]¶

Update an existing metric on the experiment.

Parameters

• metric – The metric to add.

• trial_type – The trial type for which this metric is used.

• canonical_name – The default metric for which this metric is a proxy.

Objective¶

class ax.core.objective.MultiObjective(objectives: Optional[List[ax.core.objective.Objective]] = None, **extra_kwargs: Any)[source]

Bases: ax.core.objective.Objective

Class for an objective composed of a multiple component objectives.

The Acquisition function determines how the objectives are weighted.

objectives

List of objectives.

clone() → ax.core.objective.Objective[source]

Create a copy of the objective.

get_unconstrainable_metrics() → List[ax.core.metric.Metric][source]

Return a list of metrics that are incompatible with OutcomeConstraints.

property metric

Override base method to error.

property metrics

Get the objective metrics.

property objective_weights

Get the objectives and weights.

property objectives

Get the objectives.

weights: List[float]

class ax.core.objective.Objective(metric: ax.core.metric.Metric, minimize: Optional[bool] = None)[source]

Bases: ax.utils.common.base.SortableBase

Base class for representing an objective.

minimize

If True, minimize metric.

clone() → ax.core.objective.Objective[source]

Create a copy of the objective.

get_unconstrainable_metrics() → List[ax.core.metric.Metric][source]

Return a list of metrics that are incompatible with OutcomeConstraints.

property metric

Get the objective metric.

property metric_names

Get a list of objective metric names.

property metrics

Get a list of objective metrics.

class ax.core.objective.ScalarizedObjective(metrics: List[ax.core.metric.Metric], weights: Optional[List[float]] = None, minimize: bool = False)[source]

Bases: ax.core.objective.Objective

Class for an objective composed of a linear scalarization of metrics.

metrics

List of metrics.

weights

Weights for scalarization; default to 1.

Type

List[float]

clone() → ax.core.objective.Objective[source]

Create a copy of the objective.

get_unconstrainable_metrics() → List[ax.core.metric.Metric][source]

Return a list of metrics that are incompatible with OutcomeConstraints.

property metric

Override base method to error.

property metric_weights

Get the metrics and weights.

property metrics

Get the metrics.

weights: List[float]

Observation¶

class ax.core.observation.Observation(features: ax.core.observation.ObservationFeatures, data: ax.core.observation.ObservationData, arm_name: Optional[str] = None)[source]¶

Bases: ax.utils.common.base.Base

Represents an observation.

A set of features (ObservationFeatures) and corresponding measurements (ObservationData). Optionally, an arm name associated with the features.

features¶

Type

ObservationFeatures

data¶

Type

ObservationData

arm_name¶

Type

Optional[str]

class ax.core.observation.ObservationData(metric_names: List[str], means: numpy.ndarray, covariance: numpy.ndarray)[source]¶

Bases: ax.utils.common.base.Base

Outcomes observed at a point.

The “point” corresponding to this ObservationData would be an ObservationFeatures object.

metric_names¶

A list of k metric names that were observed

means¶

a k-array of observed means

covariance¶

a (k x k) array of observed covariances

property covariance_matrix¶

Extract covariance matric from this observation data as mapping from metric name (m1) to mapping of another metric name (m2) to the covariance of the two metrics (m1 and m2).

property means_dict¶

Extract means from this observation data as mapping from metric name to mean.

class ax.core.observation.ObservationFeatures(parameters: Dict[str, Optional[Union[str, bool, float, int]]], trial_index: Optional[numpy.int64] = None, start_time: Optional[pandas.Timestamp] = None, end_time: Optional[pandas.Timestamp] = None, random_split: Optional[numpy.int64] = None, metadata: Optional[Dict[str, Any]] = None)[source]¶

Bases: ax.utils.common.base.Base

The features of an observation.

These include both the arm parameters and the features of the observation found in the Data object: trial index, times, and random split. This object is meant to contain everything needed to represent this observation in a model feature space. It is essentially a row of Data joined with the arm parameters.

An ObservationFeatures object would typically have a corresponding ObservationData object that provides the observed outcomes.

parameters¶

arm parameters

trial_index¶

trial index

start_time¶

batch start time

end_time¶

batch end time

random_split¶

random split

static from_arm(arm: ax.core.arm.Arm, trial_index: Optional[numpy.int64] = None, start_time: Optional[pandas.Timestamp] = None, end_time: Optional[pandas.Timestamp] = None, random_split: Optional[numpy.int64] = None, metadata: Optional[Dict[str, Any]] = None) → ax.core.observation.ObservationFeatures[source]¶

Convert a Arm to an ObservationFeatures, including additional data as specified.

update_features(new_features: ax.core.observation.ObservationFeatures) → ax.core.observation.ObservationFeatures[source]¶

Updates the existing ObservationFeatures with the fields of the the input.

Adds all of the new parameters to the existing parameters and overwrites any other fields that are not None on the new input features.

ax.core.observation.get_map_keys_and_feature_cols(data: ax.core.data.Data) → Tuple[List[str], List[str]][source]¶

ax.core.observation.observations_from_data(experiment: ax.core.experiment.Experiment, data: ax.core.data.Data, include_abandoned: bool = False) → List[ax.core.observation.Observation][source]¶

Convert Data to observations.

Converts a Data object to a list of Observation objects. Pulls arm parameters from from experiment. Overrides fidelity parameters in the arm with those found in the Data object.

Uses a diagonal covariance matrix across metric_names.

Parameters

• experiment – Experiment with arm parameters.

• data – Data of observations.

• include_abandoned – Whether data for abandoned trials and arms should be included in the observations, returned from this function.

Returns

List of Observation objects.