ax¶

class ax.Arm(parameters: Dict[str, Optional[Union[str, bool, float, int]]], name: Optional[str] = None)[source]

Base class for defining arms.

Randomization in experiments assigns units to a given arm. Thus, the arm encapsulates the parametrization needed by the unit.

clone(clear_name: bool = False) → ax.core.arm.Arm [source]

Create a copy of this arm.

Parameters: clear_name – whether this cloned copy should set its name to None instead of the name of the arm being cloned. Defaults to False.

property has_name: Return true if arm’s name is not None.

static md5hash(parameters: Dict[str, Optional[Union[str, bool, float, int]]]) → str [source]

Return unique identifier for arm’s parameters.

Parameters: parameters – Parameterization; mapping of param name to value.
Returns: Hash of arm’s parameters.

property name: Get arm name. Throws if name is None.

property name_or_short_signature

Returns arm name if exists; else last 4 characters of the hash.

Used for presentation of candidates (e.g. plotting and tables), where the candidates do not yet have names (since names are automatically set upon addition to a trial).

property parameters: Get mapping from parameter names to values.

property signature: Get unique representation of a arm.

class ax.BatchTrial(experiment: core.experiment.Experiment, generator_run: Optional[GeneratorRun] = None, trial_type: Optional[str] = None, optimize_for_power: Optional[bool] = False, ttl_seconds: Optional[int] = None, index: Optional[int] = None)[source]

Batched trial that has multiple attached arms, meant to be deployed and evaluated together, and possibly arm weights, which are a measure of how much of the total resources allocated to evaluating a batch should go towards evaluating the specific arm. For instance, for field experiments the weights could describe the fraction of the total experiment population assigned to the different treatment arms. Interpretation of the weights is defined in Runner.

NOTE: A BatchTrial is not just a trial with many arms; it is a trial, for which it is important that the arms are evaluated simultaneously, e.g. in an A/B test where the evaluation results are subject to nonstationarity. For cases where multiple arms are evaluated separately and independently of each other, use multiple Trial objects with a single arm each.

Parameters

experiment – Experiment, to which this trial is attached
generator_run – GeneratorRun, associated with this trial. This can a also be set later through add_arm or add_generator_run, but a trial’s associated generator run is immutable once set.
trial_type – Type of this trial, if used in MultiTypeExperiment.
optimize_for_power – Whether to optimize the weights of arms in this trial such that the experiment’s power to detect effects of certain size is as high as possible. Refer to documentation of BatchTrial.set_status_quo_and_optimize_power for more detail.
ttl_seconds – If specified, trials will be considered failed after this many seconds since the time the trial was ran, unless the trial is completed before then. Meant to be used to detect ‘dead’ trials, for which the evaluation process might have crashed etc., and which should be considered failed after their ‘time to live’ has passed.
index – If specified, the trial’s index will be set accordingly. This should generally not be specified, as in the index will be automatically determined based on the number of existing trials. This is only used for the purpose of loading from storage.

property abandoned_arm_names: Set of names of arms that have been abandoned within this trial.

property abandoned_arms: List of arms that have been abandoned within this trial.

property arm_weights

The set of arms and associated weights for the trial.

These are constructed by merging the arms and weights from each generator run that is attached to the trial.

property arms: All arms contained in the trial.

property arms_by_name: Map from arm name to object for all arms in trial.

clone() → ax.core.batch_trial.BatchTrial [source]

Clone the trial.

Returns: A new instance of the trial.

property experiment: The experiment this batch belongs to.

property generator_run_structs

List of generator run structs attached to this trial.

Struct holds generator_run object and the weight with which it was added.

property generator_runs: All generator runs associated with this trial.

property index: The index of this batch within the experiment’s batch list.

property is_factorial: Return true if the trial’s arms are a factorial design with no linked factors.

mark_arm_abandoned(arm_name: str, reason: Optional[str] = None) → ax.core.batch_trial.BatchTrial [source]

Mark a arm abandoned.

Usually done after deployment when one arm causes issues but user wants to continue running other arms in the batch.

NOTE: Abandoned arms are considered to be ‘pending points’ in experiment after their abandonment to avoid Ax models suggesting the same arm again as a new candidate. Abandoned arms are also excluded from model training data unless fit_abandoned option is specified to model bridge.

Parameters

arm_name – The name of the arm to abandon.
reason – The reason for abandoning the arm.

Returns

The batch instance.

normalized_arm_weights(total: float = 1, trunc_digits: Optional[int] = None) → MutableMapping[ax.core.arm.Arm, float][source]

Returns arms with a new set of weights normalized to the given total.

This method is useful for many runners where we need to normalize weights to a certain total without mutating the weights attached to a trial.

Parameters

total – The total weight to which to normalize. Default is 1, in which case arm weights can be interpreted as probabilities.
trunc_digits – The number of digits to keep. If the resulting total weight is not equal to total, re-allocate weight in such a way to maintain relative weights as best as possible.

Returns

Mapping from arms to the new set of weights.

run() → ax.core.batch_trial.BatchTrial [source]

Deploys the trial according to the behavior on the runner.

The runner returns a run_metadata dict containining metadata of the deployment process. It also returns a deployed_name of the trial within the system to which it was deployed. Both these fields are set on the trial.

Returns: The trial instance.

property status_quo: The control arm for this batch.

unset_status_quo() → None [source]: Set the status quo to None.

property weights: Weights corresponding to arms contained in the trial.

class ax.ChoiceParameter(name: str, parameter_type: ax.core.parameter.ParameterType, values: List[Optional[Union[str, bool, float, int]]], is_ordered: Optional[bool] = None, is_task: bool = False, is_fidelity: bool = False, target_value: Optional[Union[str, bool, float, int]] = None, sort_values: Optional[bool] = None, dependents: Optional[Dict[Optional[Union[str, bool, float, int]], List[str]]] = None)[source]

Parameter object that specifies a discrete set of values.

Parameters

name – Name of the parameter.
parameter_type – Enum indicating the type of parameter value (e.g. string, int).
values – List of allowed values for the parameter.
is_ordered – If False, the parameter is a categorical variable. Defaults to False if parameter_type is STRING and values is longer than 2, else True.
is_task – Treat the parameter as a task parameter for modeling.
is_fidelity – Whether this parameter is a fidelity parameter.
target_value – Target value of this parameter if it’s fidelity.
sort_values – Whether to sort values before encoding. Defaults to False if parameter_type is STRING, else True.
dependents – Optional mapping for parameters in hierarchical search spaces; format is { value -> list of dependent parameter names }.

add_values(values: List[Optional[Union[str, bool, float, int]]]) → ax.core.parameter.ChoiceParameter [source]

Add input list to the set of allowed values for parameter.

Cast all input values to the parameter type.

Parameters: values – Values being added to the allowed list.

set_values(values: List[Optional[Union[str, bool, float, int]]]) → ax.core.parameter.ChoiceParameter [source]

Set the list of allowed values for parameter.

Cast all input values to the parameter type.

Parameters: values – New list of allowed values.

validate(value: Optional[Union[str, bool, float, int]]) → bool [source]

Checks that the input is in the list of allowed values.

Parameters: value – Value being checked.
Returns: True if valid, False otherwise.

class ax.ComparisonOp(value)[source]: Class for enumerating comparison operations.

class ax.Data(df: Optional[pandas.DataFrame] = None, description: Optional[str] = None)[source]

Class storing data for an experiment.

The dataframe is retrieved via the df property. The data can be stored to an external store for future use by attaching it to an experiment using experiment.attach_data() (this requires a description to be set.)

df: DataFrame with underlying data, and required columns.

description: Human-readable description of data.

classmethod column_data_types(extra_column_types: Optional[Dict[str, Type]] = None, excluded_columns: Optional[Iterable[str]] = None) → Dict[str, Type][source]: Type specification for all supported columns.

copy_structure_with_df(df: pandas.DataFrame) → ax.core.data.Data[source]: Serialize the structural properties needed to initialize this Data. Used for storage and to help construct new similar Data. All kwargs other than df and description are considered structural.

classmethod deserialize_init_args(args: Dict[str, Any]) → Dict[str, Any][source]: Given a dictionary, extract the properties needed to initialize the metric. Used for storage.

property df_hash

Compute hash of pandas DataFrame.

This first serializes the DataFrame and computes the md5 hash on the resulting string. Note that this may cause performance issue for very large DataFrames.

Parameters: df – The DataFrame for which to compute the hash.

Returns: str: The hash of the DataFrame.

static from_evaluations(evaluations: Dict[str, Dict[str, Union[float, numpy.floating, numpy.integer, Tuple[Union[float, numpy.floating, numpy.integer], Optional[Union[float, numpy.floating, numpy.integer]]]]]], trial_index: int, sample_sizes: Optional[Dict[str, int]] = None, start_time: Optional[int] = None, end_time: Optional[int] = None) → ax.core.data.Data[source]

Convert dict of evaluations to Ax data object.

Parameters

evaluations – Map from arm name to metric outcomes (itself a mapping of metric names to tuples of mean and optionally a SEM).
trial_index – Trial index to which this data belongs.
sample_sizes – Number of samples collected for each arm.
start_time – Optional start time of run of the trial that produced this data, in milliseconds.
end_time – Optional end time of run of the trial that produced this data, in milliseconds.

Returns

Ax Data object.

static from_fidelity_evaluations(evaluations: Dict[str, List[Tuple[Dict[str, Optional[Union[str, bool, float, int]]], Dict[str, Union[float, numpy.floating, numpy.integer, Tuple[Union[float, numpy.floating, numpy.integer], Optional[Union[float, numpy.floating, numpy.integer]]]]]]]], trial_index: int, sample_sizes: Optional[Dict[str, int]] = None, start_time: Optional[int] = None, end_time: Optional[int] = None) → ax.core.data.Data[source]

Convert dict of fidelity evaluations to Ax data object.

Parameters

evaluations – Map from arm name to list of (fidelity, metric outcomes) (where metric outcomes is itself a mapping of metric names to tuples of mean and SEM).
trial_index – Trial index to which this data belongs.
sample_sizes – Number of samples collected for each arm.
start_time – Optional start time of run of the trial that produced this data, in milliseconds.
end_time – Optional end time of run of the trial that produced this data, in milliseconds.

Returns

Ax Data object.

static from_multiple_data(data: Iterable[ax.core.data.Data], subset_metrics: Optional[Iterable[str]] = None) → ax.core.data.Data[source]

Combines multiple data objects into one (with the concatenated underlying dataframe).

Parameters

data – Iterable of Ax Data objects to combine.
subset_metrics – If specified, combined Data will only contain metrics, names of which appear in this iterable, in the underlying dataframe.

get_filtered_results(**filters: Dict[str, Any]) → pandas.DataFrame[source]

Return filtered subset of data.

Parameters: filter – Column names and values they must match.

Returns: df: The filtered DataFrame.

property metric_names: Set of metric names that appear in the underlying dataframe of this object.

classmethod required_columns() → Set[str][source]: Names of columns that must be present in the underlying DataFrame.

classmethod serialize_init_args(data: ax.core.data.Data) → Dict[str, Any][source]: Serialize the class-dependent properties needed to initialize this Data. Used for storage and to help construct new similar Data. All kwargs other than “dataframe” and “description” are considered structural.

classmethod supported_columns(extra_column_names: Optional[Iterable[str]] = None) → Set[str][source]: Names of columns supported (but not necessarily required) by this class.

property true_df: Return the DataFrame being used as the source of truth (avoid using except for caching).

class ax.Experiment(search_space: ax.core.search_space.SearchSpace, name: Optional[str] = None, optimization_config: Optional[ax.core.optimization_config.OptimizationConfig] = None, tracking_metrics: Optional[List[ax.core.metric.Metric]] = None, runner: Optional[ax.core.runner.Runner] = None, status_quo: Optional[ax.core.arm.Arm] = None, description: Optional[str] = None, is_test: bool = False, experiment_type: Optional[str] = None, properties: Optional[Dict[str, Any]] = None, default_data_type: Optional[ax.core.experiment.DataType] = None)[source]

Base class for defining an experiment.

add_tracking_metric(metric: ax.core.metric.Metric) → ax.core.experiment.Experiment [source]

Add a new metric to the experiment.

Parameters: metric – Metric to be added.

add_tracking_metrics(metrics: List[ax.core.metric.Metric]) → ax.core.experiment.Experiment [source]

Add a list of new metrics to the experiment.

If any of the metrics are already defined on the experiment, we raise an error and don’t add any of them to the experiment

Parameters: metrics – Metrics to be added.

property arms_by_name: The arms belonging to this experiment, by their name.

property arms_by_signature: The arms belonging to this experiment, by their signature.

attach_data(data: ax.core.data.Data, combine_with_last_data: bool = False, overwrite_existing_data: bool = False) → int [source]

Attach data to experiment. Stores data in experiment._data_by_trial, to be looked up via experiment.lookup_data_for_trial.

Parameters

data – Data object to store.
combine_with_last_data –
By default, when attaching data, it’s identified by its timestamp, and experiment.lookup_data_for_trial returns data by most recent timestamp. Sometimes, however, we want to combine the data from multiple calls to attach_data into one dataframe. This might be because:
- We attached data for some metrics at one point and data for
the rest of the metrics later on. - We attached data for some fidelity at one point and data for another fidelity later one.
To achieve that goal, set combine_with_last_data to True. In this case, we will take the most recent previously attached data, append the newly attached data to it, attach a new Data object with the merged result, and delete the old one. Afterwards, calls to lookup_data_for_trial will return this new combined data object. This operation will also validate that the newly added data does not contain observations for metrics that already have observations at the same fidelity in the most recent data.
overwrite_existing_data – By default, we keep around all data that has ever been attached to the experiment. However, if we know that the incoming data contains all the information we need for a given trial, we can replace the existing data for that trial, thereby reducing the amount we need to store in the database.

Returns

Timestamp of storage in millis.

property data_by_trial

Data stored on the experiment, indexed by trial index and storage time.

First key is trial index and second key is storage time in milliseconds. For a given trial, data is ordered by storage time, so first added data will appear first in the list.

property default_trial_type

Default trial type assigned to trials in this experiment.

In the base experiment class this is always None. For experiments with multiple trial types, use the MultiTypeExperiment class.

property experiment_type: The type of the experiment.

fetch_data(metrics: Optional[List[ax.core.metric.Metric]] = None, **kwargs: Any) → ax.core.data.Data[source]

Fetches data for all trials on this experiment and for either the specified metrics or all metrics currently on the experiment, if metrics argument is not specified.

NOTE: For metrics that are not available while trial is running, the data may be retrieved from cache on the experiment. Data is cached on the experiment via calls to experiment.attach_data and whetner a given metric class is available while trial is running is determined by the boolean returned from its is_available_while_running class method.

Parameters

metrics – If provided, fetch data for these metrics instead of the ones defined on the experiment.
kwargs – keyword args to pass to underlying metrics’ fetch data functions.

Returns

Data for the experiment.

fetch_trials_data(trial_indices: Iterable[int], metrics: Optional[List[ax.core.metric.Metric]] = None, **kwargs: Any) → ax.core.data.Data[source]

Fetches data for specific trials on the experiment.

NOTE: For metrics that are not available while trial is running, the data may be retrieved from cache on the experiment. Data is cached on the experiment via calls to experiment.attach_data and whetner a given metric class is available while trial is running is determined by the boolean returned from its is_available_while_running class method.

Parameters

trial_indices – Indices of trials, for which to fetch data.
metrics – If provided, fetch data for these metrics instead of the ones defined on the experiment.
kwargs – Keyword args to pass to underlying metrics’ fetch data functions.

Returns

Data for the specific trials on the experiment.

get_trials_by_indices(trial_indices: Iterable[int]) → List[ax.core.base_trial.BaseTrial][source]: Grabs trials on this experiment by their indices.

property has_name: Return true if experiment’s name is not None.

property immutable_search_space_and_opt_config

Boolean representing whether search space and metrics on this experiment are mutable (by default they are).

NOTE: For experiments with immutable search spaces and metrics, generator runs will not store copies of search space and metrics, which improves storage layer performance. Not keeping copies of those on generator runs also disables keeping track of changes to search space and metrics, thereby necessitating that those attributes be immutable on experiment.

property is_moo_problem: Whether the experiment’s optimization config contains multiple objectives.

property is_simple_experiment: Whether this experiment is a regular Experiment or the subclassing SimpleExperiment.

property is_test: Get whether the experiment is a test.

lookup_data(trial_indices: Optional[Iterable[int]] = None) → ax.core.data.Data[source]

Lookup data for all trials on this experiment and for either the specified metrics or all metrics currently on the experiment, if metrics argument is not specified.

Parameters: trial_indices – Indices of trials, for which to fetch data.
Returns: Data for the experiment.

lookup_data_for_trial(trial_index: int) → Tuple[ax.core.data.Data, int][source]

Lookup stored data for a specific trial.

Returns latest data object, and its storage timestamp, present for this trial. Returns empty data and -1 if no data present.

Parameters: trial_index – The index of the trial to lookup data for.
Returns: The requested data object, and its storage timestamp in milliseconds.

lookup_data_for_ts(timestamp: int) → ax.core.data.Data[source]

Collect data for all trials stored at this timestamp.

Useful when many trials’ data was fetched and stored simultaneously and user wants to retrieve same collection of data later.

Can also be used to lookup specific data for a single trial when storage time is known.

Parameters: timestamp – Timestamp in millis at which data was stored.
Returns: Data object with all data stored at the timestamp.

property metrics: The metrics attached to the experiment.

property name: Get experiment name. Throws if name is None.

new_batch_trial(generator_run: Optional[ax.core.generator_run.GeneratorRun] = None, trial_type: Optional[str] = None, optimize_for_power: Optional[bool] = False, ttl_seconds: Optional[int] = None) → ax.core.batch_trial.BatchTrial [source]

Create a new batch trial associated with this experiment.

Parameters

generator_run – GeneratorRun, associated with this trial. This can a also be set later through add_arm or add_generator_run, but a trial’s associated generator run is immutable once set.
trial_type – Type of this trial, if used in MultiTypeExperiment.
optimize_for_power – Whether to optimize the weights of arms in this trial such that the experiment’s power to detect effects of certain size is as high as possible. Refer to documentation of BatchTrial.set_status_quo_and_optimize_power for more detail.
ttl_seconds – If specified, trials will be considered failed after this many seconds since the time the trial was ran, unless the trial is completed before then. Meant to be used to detect ‘dead’ trials, for which the evaluation process might have crashed etc., and which should be considered failed after their ‘time to live’ has passed.

new_trial(generator_run: Optional[ax.core.generator_run.GeneratorRun] = None, trial_type: Optional[str] = None, ttl_seconds: Optional[int] = None) → ax.core.trial.Trial [source]

Create a new trial associated with this experiment.

Parameters

generator_run – GeneratorRun, associated with this trial. Trial has only one arm attached to it and this generator_run must therefore contain one arm. This arm can also be set later through add_arm or add_generator_run, but a trial’s associated generator run is immutable once set.
trial_type – Type of this trial, if used in MultiTypeExperiment.
ttl_seconds – If specified, trials will be considered failed after this many seconds since the time the trial was ran, unless the trial is completed before then. Meant to be used to detect ‘dead’ trials, for which the evaluation process might have crashed etc., and which should be considered failed after their ‘time to live’ has passed.

property num_abandoned_arms: How many arms attached to this experiment are abandoned.

property num_trials: How many trials are associated with this experiment.

property optimization_config: The experiment’s optimization config.

property parameters: The parameters in the experiment’s search space.

remove_tracking_metric(metric_name: str) → ax.core.experiment.Experiment [source]

Remove a metric that already exists on the experiment.

Parameters: metric_name – Unique name of metric to remove.

reset_runners(runner: ax.core.runner.Runner) → None [source]

Replace all candidate trials runners.

Parameters: runner – New runner to replace with.

runner_for_trial(trial: ax.core.base_trial.BaseTrial) → Optional[ax.core.runner.Runner][source]

The default runner to use for a given trial.

In the base experiment class, this is always the default experiment runner. For experiments with multiple trial types, use the MultiTypeExperiment class.

property running_trial_indices: Indices of running trials, associated with the experiment.

property search_space

The search space for this experiment.

When setting a new search space, all parameter names and types must be preserved. However, if no trials have been created, all modifications are allowed.

property status_quo: The existing arm that new arms will be compared against.

property sum_trial_sizes: Sum of numbers of arms attached to each trial in this experiment.

supports_trial_type(trial_type: Optional[str]) → bool [source]

Whether this experiment allows trials of the given type.

The base experiment class only supports None. For experiments with multiple trial types, use the MultiTypeExperiment class.

property time_created: Creation time of the experiment.

property trial_indices_by_status: Indices of trials associated with the experiment, grouped by trial status.

property trials

The trials associated with the experiment.

NOTE: If some trials on this experiment specify their TTL, RUNNING trials will be checked for whether their TTL elapsed during this call. Found past- TTL trials will be marked as FAILED.

property trials_by_status: Trials associated with the experiment, grouped by trial status.

property trials_expecting_data

the list of all trials for which data has arrived or is expected to arrive.

Type: List[BaseTrial]

update_tracking_metric(metric: ax.core.metric.Metric) → ax.core.experiment.Experiment [source]

Redefine a metric that already exists on the experiment.

Parameters: metric – New metric definition.

warm_start_from_old_experiment(old_experiment: ax.core.experiment.Experiment, copy_run_metadata: bool = False, trial_statuses_to_copy: Optional[List[ax.core.base_trial.TrialStatus]] = None) → List[ax.core.trial.Trial][source]

Copy all completed trials with data from an old Ax expeirment to this one. This function checks that the parameters of each trial are members of the current experiment’s search_space.

NOTE: Currently only handles experiments with 1-arm Trial-s, not BatchTrial-s as there has not yet been need for support of the latter.

Parameters

old_experiment – The experiment from which to transfer trials and data
copy_run_metadata – whether to copy the run_metadata from the old experiment
trial_statuses_to_copy – All trials with a status in this list will be copied. By default, copies all COMPLETED and ABANDONED trials.

Returns

List of trials successfully copied from old_experiment to this one

class ax.FixedParameter(name: str, parameter_type: ax.core.parameter.ParameterType, value: Optional[Union[str, bool, float, int]], is_fidelity: bool = False, target_value: Optional[Union[str, bool, float, int]] = None, dependents: Optional[Dict[Optional[Union[str, bool, float, int]], List[str]]] = None)[source]

Parameter object that specifies a single fixed value.

validate(value: Optional[Union[str, bool, float, int]]) → bool [source]

Checks that the input is equal to the fixed value.

Parameters: value – Value being checked.
Returns: True if valid, False otherwise.

class ax.GeneratorRun(arms: List[ax.core.arm.Arm], weights: Optional[List[float]] = None, optimization_config: Optional[ax.core.optimization_config.OptimizationConfig] = None, search_space: Optional[ax.core.search_space.SearchSpace] = None, model_predictions: Optional[Tuple[Dict[str, List[float]], Dict[str, Dict[str, List[float]]]]] = None, best_arm_predictions: Optional[Tuple[ax.core.arm.Arm, Optional[Tuple[Dict[str, float], Optional[Dict[str, Dict[str, float]]]]]]] = None, type: Optional[str] = None, fit_time: Optional[float] = None, gen_time: Optional[float] = None, model_key: Optional[str] = None, model_kwargs: Optional[Dict[str, Any]] = None, bridge_kwargs: Optional[Dict[str, Any]] = None, gen_metadata: Optional[Dict[str, Any]] = None, model_state_after_gen: Optional[Dict[str, Any]] = None, generation_step_index: Optional[int] = None, candidate_metadata_by_arm_signature: Optional[Dict[str, Optional[Dict[str, Any]]]] = None)[source]

An object that represents a single run of a generator.

This object is created each time the gen method of a generator is called. It stores the arms and (optionally) weights that were generated by the run. When we add a generator run to a trial, its arms and weights will be merged with those from previous generator runs that were already attached to the trial.

property arm_signatures: Returns signatures of arms generated by this run.

property arm_weights: Mapping from arms to weights (order matches order in arms property).

property arms: Returns arms generated by this run.

property candidate_metadata_by_arm_signature: Retrieves model-produced candidate metadata as a mapping from arm name (for the arm the candidate became when added to experiment) to the metadata dict.

clone() → ax.core.generator_run.GeneratorRun [source]: Return a deep copy of a GeneratorRun.

property gen_metadata: Returns metadata generated by this run.

property generator_run_type: The type of the generator run.

property index: The index of this generator run within a trial’s list of generator run structs. This field is set when the generator run is added to a trial.

property optimization_config: The optimization config used during generation of this run.

property param_df

Constructs a Pandas dataframe with the parameter values for each arm.

Useful for inspecting the contents of a generator run.

Returns: a dataframe with the generator run’s arms.
Return type: pd.DataFrame

property search_space: The search used during generation of this run.

property time_created: Creation time of the batch.

property weights: Returns weights associated with arms generated by this run.

class ax.Metric(name: str, lower_is_better: Optional[bool] = None, properties: Optional[Dict[str, Any]] = None)[source]

Base class for representing metrics.

The fetch_trial_data method is the essential method to override when subclassing, which specifies how to retrieve a Metric, for a given trial.

A Metric must return a Data object, which requires (at minimum) the following:: https://ax.dev/api/_modules/ax/core/data.html#Data.required_columns

lower_is_better: Flag for metrics which should be minimized.

properties: Properties specific to a particular metric.

clone() → Metric[source]: Create a copy of this Metric.

classmethod combine_with_last_data() → bool [source]: Indicates whether, when attaching data, we should merge the new dataframe into the most recently attached dataframe.

data_constructor: alias of ax.core.data.Data

classmethod deserialize_init_args(args: Dict[str, Any]) → Dict[str, Any][source]: Given a dictionary, extract the properties needed to initialize the metric. Used for storage.

fetch_experiment_data(experiment: core.experiment.Experiment, **kwargs: Any) → Data[source]

Fetch this metric’s data for an experiment.

Default behavior is to fetch data from all trials expecting data and concatenate the results.

classmethod fetch_experiment_data_multi(experiment: core.experiment.Experiment, metrics: Iterable[Metric], trials: Optional[Iterable[core.base_trial.BaseTrial]] = None, **kwargs: Any) → Data[source]

Fetch multiple metrics data for an experiment.

Default behavior calls fetch_trial_data_multi for each trial. Subclasses should override to batch data computation across trials + metrics.

property fetch_multi_group_by_metric

Metric class, with which to group this metric in Experiment._metrics_by_class, which is used to combine metrics on experiment into groups and then fetch their data via Metric.fetch_trial_data_multi for each group.

NOTE: By default, this property will just return the class on which it is defined; however, in some cases it is useful to group metrics by their superclass, in which case this property should return that superclass.

fetch_trial_data(trial: core.base_trial.BaseTrial, **kwargs: Any) → Data[source]: Fetch data for one trial.

classmethod fetch_trial_data_multi(trial: core.base_trial.BaseTrial, metrics: Iterable[Metric], **kwargs: Any) → Data[source]

Fetch multiple metrics data for one trial.

Default behavior calls fetch_trial_data for each metric. Subclasses should override this to trial data computation for multiple metrics.

classmethod is_available_while_running() → bool [source]

Whether metrics of this class are available while the trial is running. Metrics that are not available while the trial is running are assumed to be available only upon trial completion. For such metrics, data is assumed to never change once the trial is completed.

NOTE: If this method returns False, data-fetching via experiment.fetch_data will return the data cached on the experiment (for the metrics of the given class) whenever its available. Data is cached on experiment when attached via experiment.attach_data.

classmethod lookup_or_fetch_experiment_data_multi(experiment: core.experiment.Experiment, metrics: Iterable[Metric], trials: Optional[Iterable[core.base_trial.BaseTrial]] = None, **kwargs: Any) → Data[source]

Fetch or lookup (with fallback to fetching) data for given metrics, depending on whether they are available while running.

If metric is available while running, its data can change (and therefore we should always re-fetch it). If metric is available only upon trial completion, its data does not change, so we can look up that data on the experiment and only fetch the data that is not already attached to the experiment.

NOTE: If fetching data for a metrics class that is only available upon trial completion, data fetched in this function (data that was not yet available on experiment) will be attached to experiment.

property name: Get name of metric.

classmethod overwrite_existing_data() → bool [source]: Indicates whether, when attaching data, we should overwrite all previously attached data with the new dataframe.

classmethod serialize_init_args(metric: ax.core.metric.Metric) → Dict[str, Any][source]: Serialize the properties needed to initialize the metric. Used for storage.

class ax.Models(value)[source]

Registry of available models.

Uses MODEL_KEY_TO_MODEL_SETUP to retrieve settings for model and model bridge, by the key stored in the enum value.

To instantiate a model in this enum, simply call an enum member like so: Models.SOBOL(search_space=search_space) or Models.GPEI(experiment=experiment, data=data). Keyword arguments specified to the call will be passed into the model or the model bridge constructors according to their keyword.

For instance, Models.SOBOL(search_space=search_space, scramble=False) will instantiate a RandomModelBridge(search_space=search_space) with a SobolGenerator(scramble=False) underlying model.

class ax.MultiObjective(objectives: Optional[List[ax.core.objective.Objective]] = None, **extra_kwargs: Any)[source]

Class for an objective composed of a multiple component objectives.

The Acquisition function determines how the objectives are weighted.

objectives: List of objectives.

clone() → ax.core.objective.Objective[source]: Create a copy of the objective.

property metric: Override base method to error.

property metrics: Get the objective metrics.

property objective_weights: Get the objectives and weights.

property objectives: Get the objectives.

class ax.MultiObjectiveOptimizationConfig(objective: ax.core.objective.Objective, outcome_constraints: Optional[List[ax.core.outcome_constraint.OutcomeConstraint]] = None, objective_thresholds: Optional[List[ax.core.outcome_constraint.ObjectiveThreshold]] = None)[source]

An optimization configuration for multi-objective optimization, which comprises multiple objective, outcome constraints, and objective thresholds.

There is no minimum or maximum number of outcome constraints, but an individual metric can have at most two constraints–which is how we represent metrics with both upper and lower bounds.

ObjectiveThresholds should be present for every objective. A good rule of thumb is to set them 10% below the minimum acceptable value for each metric.

property all_constraints: Get all constraints and thresholds.

clone_with_args(objective: Optional[ax.core.objective.Objective] = None, outcome_constraints: Optional[List[ax.core.outcome_constraint.OutcomeConstraint]] = None, objective_thresholds: Optional[List[ax.core.outcome_constraint.ObjectiveThreshold]] = None) → ax.core.optimization_config.MultiObjectiveOptimizationConfig [source]: Make a copy of this optimization config.

property objective: Get objective.

property objective_thresholds: Get objective thresholds.

property objective_thresholds_dict: Get a mapping from objective metric name to the corresponding threshold.

class ax.Objective(metric: ax.core.metric.Metric, minimize: Optional[bool] = None)[source]

Base class for representing an objective.

minimize: If True, minimize metric.

clone() → ax.core.objective.Objective[source]: Create a copy of the objective.

get_unconstrainable_metrics() → List[ax.core.metric.Metric][source]: Return a list of metrics that are incompatible with OutcomeConstraints.

property metric: Get the objective metric.

property metric_names: Get a list of objective metric names.

property metrics: Get a list of objective metrics.

class ax.ObjectiveThreshold(metric: ax.core.metric.Metric, bound: float, relative: bool = True, op: Optional[ax.core.types.ComparisonOp] = None)[source]

Class for representing Objective Thresholds.

An objective threshold represents the threshold for an objective metric to contribute to hypervolume calculations. A list containing the objective threshold for each metric collectively form a reference point.

Objective thresholds may bound the metric from above or from below. The bound can be expressed as an absolute measurement or relative to the status quo (if applicable).

The direction of the bound is inferred from the Metric’s lower_is_better attribute.

metric: Metric to constrain.

bound: The bound in the constraint.

relative: Whether you want to bound on an absolute or relative scale. If relative, bound is the acceptable percent change.

op: automatically inferred, but manually overwritable. specifies whether metric should be greater or equal to, or less than or equal to, some bound.

clone() → ax.core.outcome_constraint.ObjectiveThreshold[source]: Create a copy of this ObjectiveThreshold.

class ax.OptimizationConfig(objective: ax.core.objective.Objective, outcome_constraints: Optional[List[ax.core.outcome_constraint.OutcomeConstraint]] = None)[source]

An optimization configuration, which comprises an objective and outcome constraints.

There is no minimum or maximum number of outcome constraints, but an individual metric can have at most two constraints–which is how we represent metrics with both upper and lower bounds.

property all_constraints: Get outcome constraints.

clone() → ax.core.optimization_config.OptimizationConfig [source]: Make a copy of this optimization config.

clone_with_args(objective: Optional[ax.core.objective.Objective] = None, outcome_constraints: Optional[List[ax.core.outcome_constraint.OutcomeConstraint]] = None) → ax.core.optimization_config.OptimizationConfig [source]: Make a copy of this optimization config.

property objective: Get objective.

property outcome_constraints: Get outcome constraints.

class ax.OptimizationLoop(experiment: ax.core.experiment.Experiment, evaluation_function: Callable[[Dict[str, Optional[Union[str, bool, float, int]]], Optional[float]], Union[Dict[str, Union[float, numpy.floating, numpy.integer, Tuple[Union[float, numpy.floating, numpy.integer], Optional[Union[float, numpy.floating, numpy.integer]]]]], float, numpy.floating, numpy.integer, Tuple[Union[float, numpy.floating, numpy.integer], Optional[Union[float, numpy.floating, numpy.integer]]], List[Tuple[Dict[str, Optional[Union[str, bool, float, int]]], Dict[str, Union[float, numpy.floating, numpy.integer, Tuple[Union[float, numpy.floating, numpy.integer], Optional[Union[float, numpy.floating, numpy.integer]]]]]]], List[Tuple[Dict[str, Hashable], Dict[str, Union[float, numpy.floating, numpy.integer, Tuple[Union[float, numpy.floating, numpy.integer], Optional[Union[float, numpy.floating, numpy.integer]]]]]]]]], total_trials: int = 20, arms_per_trial: int = 1, random_seed: Optional[int] = None, wait_time: int = 0, run_async: bool = False, generation_strategy: Optional[ax.modelbridge.generation_strategy.GenerationStrategy] = None)[source]

Managed optimization loop, in which Ax oversees deployment of trials and gathering data.

full_run() → ax.service.managed_loop.OptimizationLoop [source]: Runs full optimization loop as defined in the provided optimization plan.

get_best_point() → Tuple[Dict[str, Optional[Union[str, bool, float, int]]], Optional[Tuple[Dict[str, float], Optional[Dict[str, Dict[str, float]]]]]][source]: Obtains the best point encountered in the course of this optimization.

get_current_model() → Optional[ax.modelbridge.base.ModelBridge][source]: Obtain the most recently used model in optimization.

run_trial() → None [source]: Run a single step of the optimization plan.

static with_evaluation_function(parameters: List[Dict[str, Union[str, bool, float, int, None, List[Optional[Union[str, bool, float, int]]], Dict[str, List[str]]]]], evaluation_function: Callable[[Dict[str, Optional[Union[str, bool, float, int]]], Optional[float]], Union[Dict[str, Union[float, numpy.floating, numpy.integer, Tuple[Union[float, numpy.floating, numpy.integer], Optional[Union[float, numpy.floating, numpy.integer]]]]], float, numpy.floating, numpy.integer, Tuple[Union[float, numpy.floating, numpy.integer], Optional[Union[float, numpy.floating, numpy.integer]]], List[Tuple[Dict[str, Optional[Union[str, bool, float, int]]], Dict[str, Union[float, numpy.floating, numpy.integer, Tuple[Union[float, numpy.floating, numpy.integer], Optional[Union[float, numpy.floating, numpy.integer]]]]]]], List[Tuple[Dict[str, Hashable], Dict[str, Union[float, numpy.floating, numpy.integer, Tuple[Union[float, numpy.floating, numpy.integer], Optional[Union[float, numpy.floating, numpy.integer]]]]]]]]], experiment_name: Optional[str] = None, objective_name: Optional[str] = None, minimize: bool = False, parameter_constraints: Optional[List[str]] = None, outcome_constraints: Optional[List[str]] = None, total_trials: int = 20, arms_per_trial: int = 1, wait_time: int = 0, random_seed: Optional[int] = None, generation_strategy: Optional[ax.modelbridge.generation_strategy.GenerationStrategy] = None) → OptimizationLoop[source]: Constructs a synchronous OptimizationLoop using an evaluation function.

classmethod with_runners_and_metrics(parameters: List[Dict[str, Union[str, bool, float, int, None, List[Optional[Union[str, bool, float, int]]], Dict[str, List[str]]]]], path_to_runner: str, paths_to_metrics: List[str], experiment_name: Optional[str] = None, objective_name: Optional[str] = None, minimize: bool = False, parameter_constraints: Optional[List[str]] = None, outcome_constraints: Optional[List[str]] = None, total_trials: int = 20, arms_per_trial: int = 1, wait_time: int = 0, random_seed: Optional[int] = None) → OptimizationLoop[source]: Constructs an asynchronous OptimizationLoop using Ax runners and metrics.

class ax.OrderConstraint(lower_parameter: ax.core.parameter.Parameter, upper_parameter: ax.core.parameter.Parameter)[source]

Constraint object for specifying one parameter to be smaller than another.

clone() → ax.core.parameter_constraint.OrderConstraint [source]: Clone.

clone_with_transformed_parameters(transformed_parameters: Dict[str, ax.core.parameter.Parameter]) → ax.core.parameter_constraint.OrderConstraint [source]: Clone, but replace parameters with transformed versions.

property constraint_dict: Weights on parameters for linear constraint representation.

property lower_parameter: Parameter with lower value.

property parameters: Parameters.

property upper_parameter: Parameter with higher value.

class ax.OutcomeConstraint(metric: ax.core.metric.Metric, op: ax.core.types.ComparisonOp, bound: float, relative: bool = True)[source]

Base class for representing outcome constraints.

Outcome constraints may of the form metric >= bound or metric <= bound, where the bound can be expressed as an absolute measurement or relative to the status quo (if applicable).

metric: Metric to constrain.

op: Specifies whether metric should be greater or equal to, or less than or equal to, some bound.

bound: The bound in the constraint.

relative: Whether you want to bound on an absolute or relative scale. If relative, bound is the acceptable percent change.

clone() → ax.core.outcome_constraint.OutcomeConstraint[source]: Create a copy of this OutcomeConstraint.

class ax.Parameter[source]

is_valid_type(value: Optional[Union[str, bool, float, int]]) → bool [source]: Whether a given value’s type is allowed by this parameter.

property python_type

The python type for the corresponding ParameterType enum.

Used primarily for casting values of unknown type to conform to that of the parameter.

class ax.ParameterConstraint(constraint_dict: Dict[str, float], bound: float)[source]

Base class for linear parameter constraints.

Constraints are expressed using a map from parameter name to weight followed by a bound.

The constraint is satisfied if w * v <= b where:: w is the vector of parameter weights. v is a vector of parameter values. b is the specified bound. * is the dot product operator.

property bound: Get bound of the inequality of the constraint.

check(parameter_dict: Dict[str, Union[int, float]]) → bool [source]

Whether or not the set of parameter values satisfies the constraint.

Does a weighted sum of the parameter values based on the constraint_dict and checks that the sum is less than the bound.

Parameters: parameter_dict – Map from parameter name to parameter value.
Returns: Whether the constraint is satisfied.

clone() → ax.core.parameter_constraint.ParameterConstraint [source]: Clone.

clone_with_transformed_parameters(transformed_parameters: Dict[str, ax.core.parameter.Parameter]) → ax.core.parameter_constraint.ParameterConstraint [source]: Clone, but replaced parameters with transformed versions.

property constraint_dict: Get mapping from parameter names to weights.

class ax.ParameterType(value)[source]: An enumeration.

class ax.RangeParameter(name: str, parameter_type: ax.core.parameter.ParameterType, lower: float, upper: float, log_scale: bool = False, logit_scale: bool = False, digits: Optional[int] = None, is_fidelity: bool = False, target_value: Optional[Union[str, bool, float, int]] = None)[source]

Parameter object that specifies a range of values.

property digits

Number of digits to round values to for float type.

Upper and lower bound are re-cast after this property is changed.

is_valid_type(value: Optional[Union[str, bool, float, int]]) → bool [source]: Same as default except allows floats whose value is an int for Int parameters.

property log_scale: Whether the parameter’s random values should be sampled from log space.

property logit_scale: Whether the parameter’s random values should be sampled from logit space.

property lower

Lower bound of the parameter range.

Value is cast to parameter type upon set and also validated to ensure the bound is strictly less than upper bound.

update_range(lower: Optional[float] = None, upper: Optional[float] = None) → ax.core.parameter.RangeParameter [source]

Set the range to the given values.

If lower or upper is not provided, it will be left at its current value.

Parameters

lower – New value for the lower bound.
upper – New value for the upper bound.

property upper

Upper bound of the parameter range.

Value is cast to parameter type upon set and also validated to ensure the bound is strictly greater than lower bound.

validate(value: Optional[Union[str, bool, float, int]]) → bool [source]

Returns True if input is a valid value for the parameter.

Checks that value is of the right type and within the valid range for the parameter. Returns False if value is None.

Parameters: value – Value being checked.
Returns: True if valid, False otherwise.

class ax.Runner[source]

Abstract base class for custom runner classes

clone() → ax.core.runner.Runner [source]: Create a copy of this Runner.

classmethod deserialize_init_args(args: Dict[str, Any]) → Dict[str, Any][source]: Given a dictionary, deserialize the properties needed to initialize the runner. Used for storage.

poll_available_capacity() → int [source]

Checks how much available capacity there is to schedule trial evaluations. Required for runners used with Ax Scheduler.

NOTE: This method might be difficult to implement in some systems. Returns -1 if capacity of the system is “unlimited” or “unknown” (meaning that the Scheduler should be trying to schedule as many trials as is possible without violating scheduler settings). There is no need to artificially force this method to limit capacity; Scheduler has other limitations in place to limit number of trials running at once, like the SchedulerOptions.max_pending_trials setting, or more granular control in the form of the max_parallelism setting in each of the GenerationStep`s of a `GenerationStrategy).

Returns: An integer, representing how many trials there is available capacity for; -1 if capacity is “unlimited” or not possible to know in advance.

poll_trial_status(trials: Iterable[core.base_trial.BaseTrial]) → Dict[core.base_trial.TrialStatus, Set[int]][source]

Checks the status of any non-terminal trials and returns their indices as a mapping from TrialStatus to a list of indices. Required for runners used with Ax Scheduler.

NOTE: Does not need to handle waiting between polling calls while trials are running; this function should just perform a single poll.

Parameters: trials – Trials to poll.
Returns: A dictionary mapping TrialStatus to a list of trial indices that have the respective status at the time of the polling. This does not need to include trials that at the time of polling already have a terminal (ABANDONED, FAILED, COMPLETED) status (but it may).

abstract run(trial: core.base_trial.BaseTrial) → Dict[str, Any][source]

Deploys a trial based on custom runner subclass implementation.

Parameters: trial – The trial to deploy.
Returns: Dict of run metadata from the deployment process.

run_multiple(trials: Iterable[core.base_trial.BaseTrial]) → Dict[int, Dict[str, Any]][source]

Runs a single evaluation for each of the given trials. Useful when deploying multiple trials at once is more efficient than deploying them one-by-one. Used in Ax Scheduler.

NOTE: By default simply loops over run_trial. Should be overwritten if deploying multiple trials in batch is preferable.

Parameters: trials – Iterable of trials to be deployed, each containing arms with parameterizations to be evaluated. Can be a Trial if contains only one arm or a BatchTrial if contains multiple arms.
Returns: Dict of trial index to the run metadata of that trial from the deployment process.

classmethod serialize_init_args(runner: ax.core.runner.Runner) → Dict[str, Any][source]: Serialize the properties needed to initialize the runner. Used for storage.

property staging_required: Whether the trial goes to staged or running state once deployed.

stop(trial: core.base_trial.BaseTrial, reason: Optional[str] = None) → Dict[str, Any][source]

Stop a trial based on custom runner subclass implementation.

Optional method.

Parameters

trial – The trial to stop.
reason – A message containing information why the trial is to be stopped.

Returns

A dictionary of run metadata from the stopping process.

class ax.SearchSpace(parameters: List[ax.core.parameter.Parameter], parameter_constraints: Optional[List[ax.core.parameter_constraint.ParameterConstraint]] = None)[source]

Base object for SearchSpace object.

Contains a set of Parameter objects, each of which have a name, type, and set of valid values. The search space also contains a set of ParameterConstraint objects, which can be used to define restrictions across parameters (e.g. p_a < p_b).

cast_arm(arm: ax.core.arm.Arm) → ax.core.arm.Arm [source]

Cast parameterization of given arm to the types in this SearchSpace.

For each parameter in given arm, cast it to the proper type specified in this search space. Throws if there is a mismatch in parameter names. This is mostly useful for int/float, which user can be sloppy with when hand written.

Parameters: arm – Arm to cast.
Returns: New casted arm.

check_all_parameters_present(parameterization: Dict[str, Optional[Union[str, bool, float, int]]], raise_error: bool = False) → bool [source]

Whether a given parameterization contains all the parameters in the search space.

Parameters

parameterization – Dict from parameter name to value to validate.
raise_error – If true parameterization does not belong, raises an error with detailed explanation of why.

Returns

Whether the parameterization is contained in the search space.

check_membership(parameterization: Dict[str, Optional[Union[str, bool, float, int]]], raise_error: bool = False, check_all_parameters_present: bool = True) → bool [source]

Whether the given parameterization belongs in the search space.

Checks that the given parameter values have the same name/type as search space parameters, are contained in the search space domain, and satisfy the parameter constraints.

Parameters

parameterization – Dict from parameter name to value to validate.
raise_error – If true parameterization does not belong, raises an error with detailed explanation of why.
check_all_parameters_present – Ensure that parameterization specifies values for all parameters as expected by the search space.

Returns

Whether the parameterization is contained in the search space.

check_types(parameterization: Dict[str, Optional[Union[str, bool, float, int]]], allow_none: bool = True, raise_error: bool = False) → bool [source]

Checks that the given parameterization’s types match the search space.

Parameters

parameterization – Dict from parameter name to value to validate.
allow_none – Whether None is a valid parameter value.
raise_error – If true and parameterization does not belong, raises an error with detailed explanation of why.

Returns

Whether the parameterization has valid types.

construct_arm(parameters: Optional[Dict[str, Optional[Union[str, bool, float, int]]]] = None, name: Optional[str] = None) → ax.core.arm.Arm [source]: Construct new arm using given parameters and name. Any missing parameters fallback to the experiment defaults, represented as None

out_of_design_arm() → ax.core.arm.Arm [source]

Create a default out-of-design arm.

An out of design arm contains values for some parameters which are outside of the search space. In the modeling conversion, these parameters are all stripped down to an empty dictionary, since the point is already outside of the modeled space.

Returns: New arm w/ null parameter values.

class ax.SimpleExperiment(search_space: ax.core.search_space.SearchSpace, name: Optional[str] = None, objective_name: Optional[str] = None, evaluation_function: Callable[[Dict[str, Optional[Union[str, bool, float, int]]], Optional[float]], Union[Dict[str, Union[float, numpy.floating, numpy.integer, Tuple[Union[float, numpy.floating, numpy.integer], Optional[Union[float, numpy.floating, numpy.integer]]]]], float, numpy.floating, numpy.integer, Tuple[Union[float, numpy.floating, numpy.integer], Optional[Union[float, numpy.floating, numpy.integer]]], List[Tuple[Dict[str, Optional[Union[str, bool, float, int]]], Dict[str, Union[float, numpy.floating, numpy.integer, Tuple[Union[float, numpy.floating, numpy.integer], Optional[Union[float, numpy.floating, numpy.integer]]]]]]], List[Tuple[Dict[str, Hashable], Dict[str, Union[float, numpy.floating, numpy.integer, Tuple[Union[float, numpy.floating, numpy.integer], Optional[Union[float, numpy.floating, numpy.integer]]]]]]]]] = <function unimplemented_evaluation_function>, minimize: bool = False, outcome_constraints: Optional[List[ax.core.outcome_constraint.OutcomeConstraint]] = None, status_quo: Optional[ax.core.arm.Arm] = None, properties: Optional[Dict[str, Any]] = None, default_data_type: Optional[ax.core.experiment.DataType] = None)[source]

[DEPRECATED] Simplified experiment class with defaults.

Parameters

search_space – Parameter space.
name – Name of this experiment.
objective_name – Which of the metrics computed by the evaluation function is the objective.
evaluation_function – Function that evaluates mean and standard error for a parameter configuration. This function should accept a dictionary of parameter names to parameter values (TParametrization) and optionally a weight, and return a dictionary of metric names to a tuple of means and standard errors (TEvaluationOutcome). The function can also return a single tuple, in which case we assume the metric is the objective.
minimize – Whether the objective should be minimized, defaults to False.
outcome_constraints – constraints on the outcome, if any.
status_quo – Arm representing existing “control” arm.
properties – Dictionary of this experiment’s properties.
default_data_type – Enum representing the data type this experiment uses.

add_tracking_metric(metric: ax.core.metric.Metric) → ax.core.simple_experiment.SimpleExperiment [source]

Add a new metric to the experiment.

Parameters: metric – Metric to be added.

eval() → ax.core.data.Data[source]: Evaluate all arms in the experiment with the evaluation function passed as argument to this SimpleExperiment.

eval_trial(trial: ax.core.base_trial.BaseTrial) → ax.core.data.Data[source]

Evaluate trial arms with the evaluation function of this experiment.

Parameters: trial – trial, whose arms to evaluate.

property evaluation_function: Get the evaluation function.

fetch_data(metrics: Optional[List[ax.core.metric.Metric]] = None, **kwargs: Any) → ax.core.data.Data[source]

Fetches data for all trials on this experiment and for either the specified metrics or all metrics currently on the experiment, if metrics argument is not specified.

NOTE: For metrics that are not available while trial is running, the data may be retrieved from cache on the experiment. Data is cached on the experiment via calls to experiment.attach_data and whetner a given metric class is available while trial is running is determined by the boolean returned from its is_available_while_running class method.

Parameters

metrics – If provided, fetch data for these metrics instead of the ones defined on the experiment.
kwargs – keyword args to pass to underlying metrics’ fetch data functions.

Returns

Data for the experiment.

property has_evaluation_function: Whether this SimpleExperiment has a valid evaluation function attached.

property is_simple_experiment: Whether this experiment is a regular Experiment or the subclassing SimpleExperiment.

update_tracking_metric(metric: ax.core.metric.Metric) → ax.core.simple_experiment.SimpleExperiment [source]

Redefine a metric that already exists on the experiment.

Parameters: metric – New metric definition.

class ax.SumConstraint(parameters: List[ax.core.parameter.Parameter], is_upper_bound: bool, bound: float)[source]

Constraint on the sum of parameters being greater or less than a bound.

clone() → ax.core.parameter_constraint.SumConstraint [source]

Clone.

To use the same constraint, we need to reconstruct the original bound. We do this by re-applying the original bound weighting.

clone_with_transformed_parameters(transformed_parameters: Dict[str, ax.core.parameter.Parameter]) → ax.core.parameter_constraint.SumConstraint [source]: Clone, but replace parameters with transformed versions.

property constraint_dict: Weights on parameters for linear constraint representation.

property op: Whether the sum is constrained by a <= or >= inequality.

property parameters: Parameters.

class ax.Trial(experiment: core.experiment.Experiment, generator_run: Optional[GeneratorRun] = None, trial_type: Optional[str] = None, ttl_seconds: Optional[int] = None, index: Optional[int] = None)[source]

Trial that only has one attached arm and no arm weights.

Parameters

experiment – Experiment, to which this trial is attached.
generator_run – GeneratorRun, associated with this trial. Trial has only one generator run (of just one arm) attached to it. This can also be set later through add_arm or add_generator_run, but a trial’s associated genetor run is immutable once set.
trial_type – Type of this trial, if used in MultiTypeExperiment.
ttl_seconds – If specified, trials will be considered failed after this many seconds since the time the trial was ran, unless the trial is completed before then. Meant to be used to detect ‘dead’ trials, for which the evaluation process might have crashed etc., and which should be considered failed after their ‘time to live’ has passed.
index – If specified, the trial’s index will be set accordingly. This should generally not be specified, as in the index will be automatically determined based on the number of existing trials. This is only used for the purpose of loading from storage.

property abandoned_arms: Abandoned arms attached to this trial.

property arm: The arm associated with this batch.

property arms

All arms attached to this trial.

Returns

list of a single arm: attached to this trial if there is one, else None.

Return type

arms

property arms_by_name

Dictionary of all arms attached to this trial with their names as keys.

Returns

dictionary of a single: arm name to arm if one is attached to this trial, else None.

Return type

arms

property generator_run: Generator run attached to this trial.

property generator_runs: All generator runs associated with this trial.

get_metric_mean(metric_name: str) → float [source]: Metric mean for the arm attached to this trial, retrieved from the latest data available for the metric for the trial.

property objective_mean

Objective mean for the arm attached to this trial, retrieved from the latest data available for the objective for the trial.

Note: the retrieved objective is the experiment-level objective at the time of the call to objective_mean, which is not necessarily the objective that was set at the time the trial was created or ran.

ax.optimize(parameters: List[Dict[str, Union[str, bool, float, int, None, List[Optional[Union[str, bool, float, int]]], Dict[str, List[str]]]]], evaluation_function: Callable[[Dict[str, Optional[Union[str, bool, float, int]]], Optional[float]], Union[Dict[str, Union[float, numpy.floating, numpy.integer, Tuple[Union[float, numpy.floating, numpy.integer], Optional[Union[float, numpy.floating, numpy.integer]]]]], float, numpy.floating, numpy.integer, Tuple[Union[float, numpy.floating, numpy.integer], Optional[Union[float, numpy.floating, numpy.integer]]], List[Tuple[Dict[str, Optional[Union[str, bool, float, int]]], Dict[str, Union[float, numpy.floating, numpy.integer, Tuple[Union[float, numpy.floating, numpy.integer], Optional[Union[float, numpy.floating, numpy.integer]]]]]]], List[Tuple[Dict[str, Hashable], Dict[str, Union[float, numpy.floating, numpy.integer, Tuple[Union[float, numpy.floating, numpy.integer], Optional[Union[float, numpy.floating, numpy.integer]]]]]]]]], experiment_name: Optional[str] = None, objective_name: Optional[str] = None, minimize: bool = False, parameter_constraints: Optional[List[str]] = None, outcome_constraints: Optional[List[str]] = None, total_trials: int = 20, arms_per_trial: int = 1, random_seed: Optional[int] = None, generation_strategy: Optional[ax.modelbridge.generation_strategy.GenerationStrategy] = None) → Tuple[Dict[str, Optional[Union[str, bool, float, int]]], Optional[Tuple[Dict[str, float], Optional[Dict[str, Dict[str, float]]]]], ax.core.experiment.Experiment, Optional[ax.modelbridge.base.ModelBridge]][source]: Construct and run a full optimization loop.

Ax

0.2.3

ax¶

Ax

Navigation

Related Topics