ax.metrics¶
Branin¶
-
class
ax.metrics.branin.AugmentedBraninMetric(name: str, param_names: List[str], noise_sd: Optional[float] = 0.0, lower_is_better: Optional[bool] = None)[source]¶
Branin Map¶
-
class
ax.metrics.branin_map.BraninFidelityMapMetric(name: str, param_names: Iterable[str], noise_sd: float = 0.0, lower_is_better: Optional[bool] = None)[source]¶ Bases:
ax.metrics.noisy_function_map.NoisyFunctionMapMetric-
f(x: numpy.ndarray) → Mapping[str, Any][source]¶ The deterministic function that produces the metric outcomes.
-
fetch_trial_data(trial: ax.core.base_trial.BaseTrial, noisy: bool = True, **kwargs: Any) → ax.core.map_data.MapData[source]¶ Fetch data for one trial.
-
-
class
ax.metrics.branin_map.BraninTimestampMapMetric(name: str, param_names: Iterable[str], map_key_infos: Optional[Iterable[ax.core.map_data.MapKeyInfo]] = None, noise_sd: float = 0.0, lower_is_better: Optional[bool] = None, rate: Optional[float] = None, cache_evaluations: bool = True)[source]¶ Bases:
ax.metrics.noisy_function_map.NoisyFunctionMapMetric-
f(x: numpy.ndarray, timestamp: int) → Mapping[str, Any][source]¶ The deterministic function that produces the metric outcomes.
-
fetch_trial_data(trial: ax.core.base_trial.BaseTrial, noisy: bool = True, **kwargs: Any) → ax.core.map_data.MapData[source]¶ Fetch data for one trial.
-
-
ax.metrics.branin_map.random() → x in the interval [0, 1).¶
Chemistry¶
Classes for optimizing yields from chemical reactions.
References
- Perera2018
D. Perera, J. W. Tucker, S. Brahmbhatt, C. Helal, A. Chong, W. Farrell, P. Richardson, N. W. Sach. A platform for automated nanomole-scale reaction screening and micromole-scale synthesis in flow. Science, 26. 2018.
- Shields2021
B. J. Shields, J. Stevens, J. Li, et al. Bayesian reaction optimization as a tool for chemical synthesis. Nature 590, 89–96 (2021).
“SUZUKI” involves optimization solvent, ligand, and base combinations in a Suzuki-Miyaura coupling to optimize carbon-carbon bond formation. See _[Perera2018] for details.
“DIRECT_ARYLATION” involves optimizing the solvent, base, and ligand chemicals as well as the temperature and concentration for a direct arylation reaction. See _[Shields2021] for details.
-
class
ax.metrics.chemistry.ChemistryData(param_names: ‘List[str]’, objective_dict: ‘Dict[Tuple[TParamValue, …], float]’)[source]¶ Bases:
object
-
class
ax.metrics.chemistry.ChemistryMetric(name: str, noiseless: bool = False, problem_type: ax.metrics.chemistry.ChemistryProblemType = <ChemistryProblemType.SUZUKI: 'suzuki'>, lower_is_better: bool = False)[source]¶ Bases:
ax.core.metric.MetricMetric for modeling chemical reactions.
Metric describing the outcomes of chemical reactions. Based on tabulate data. Problems typically contain many discrete and categorical parameters.
- Parameters
name – The name of the metric.
noiseless – If True, consider observations noiseless, otherwise assume unknown Gaussian observation noise.
problem_type – The problem type.
-
noiseless¶ If True, consider observations noiseless, otherwise assume unknown Gaussian observation noise.
-
lower_is_better¶ If True, the metric should be minimized.
-
clone() → ax.metrics.chemistry.ChemistryMetric[source]¶ Create a copy of this Metric.
-
fetch_trial_data(trial: ax.core.base_trial.BaseTrial, **kwargs: Any) → ax.core.data.Data[source]¶ Fetch data for one trial.
Curve¶
Metrics that allow to retrieve curves of partial results. Typically used to retrieve partial learning curves of ML training jobs.
-
class
ax.metrics.curve.AbstractCurveMetric(name: str, curve_name: str, lower_is_better: bool = True)[source]¶ Bases:
ax.core.map_metric.MapMetric,abc.ABCMetric representing (partial) learning curves of ML model training jobs.
-
MAP_KEY= <ax.core.map_data.MapKeyInfo object>¶
-
classmethod
fetch_experiment_data_multi(experiment: ax.core.experiment.Experiment, metrics: Iterable[ax.core.metric.Metric], trials: Optional[Iterable[ax.core.base_trial.BaseTrial]] = None, **kwargs: Any) → ax.core.data.Data[source]¶ Fetch multiple metrics data for an experiment.
-
fetch_trial_data(trial: ax.core.base_trial.BaseTrial, **kwargs: Any) → ax.core.data.Data[source]¶ Fetch data for one trial.
-
classmethod
fetch_trial_data_multi(trial: ax.core.base_trial.BaseTrial, metrics: Iterable[ax.core.metric.Metric], **kwargs: Any) → ax.core.data.Data[source]¶ Fetch multiple metrics data for one trial.
-
abstract classmethod
get_curves_from_ids(ids: Iterable[Union[int, str]]) → Dict[Union[int, str], Dict[str, pandas.Series]][source]¶ Get partial result curves from backend ids.
- Parameters
ids – The ids of the backend runs for which to fetch the partial result curves.
- Returns
A dictionary mapping the backend id to the partial result curves, each of which is represented as a mapping from the metric name to a pandas Series indexed by the progression (which will be mapped to the MAP_KEY of the metric class). E.g. if curve_name=loss and MAP_KEY=training_rows, then a Series should look like:
training_rows (index) | loss
- ———————–|——
100 | 0.5 200 | 0.2
-
abstract classmethod
get_ids_from_trials(trials: Iterable[ax.core.base_trial.BaseTrial]) → List[Optional[Union[int, str]]][source]¶ Get backend run ids associated with trials.
- Parameters
trials – The trials for which to retrieve the associated ids that can be used to to identify the corresponding runs on the backend.
- Returns
A list of identifiers (ints or strings) corresponding to the backend runs associated with the trials, in the same order as the trials input.
-
classmethod
is_available_while_running() → bool[source]¶ Whether metrics of this class are available while the trial is running. Metrics that are not available while the trial is running are assumed to be available only upon trial completion. For such metrics, data is assumed to never change once the trial is completed.
NOTE: If this method returns False, data-fetching via experiment.fetch_data will return the data cached on the experiment (for the metrics of the given class) whenever its available. Data is cached on experiment when attached via experiment.attach_data.
-
Factorial¶
-
class
ax.metrics.factorial.FactorialMetric(name: str, coefficients: Dict[str, Dict[Optional[Union[str, bool, float, int]], float]], batch_size: int = 10000, noise_var: float = 0.0)[source]¶ Bases:
ax.core.metric.MetricMetric for testing factorial designs assuming a main effects only logit model.
-
fetch_trial_data(trial: ax.core.base_trial.BaseTrial, **kwargs: Any) → ax.core.data.Data[source]¶ Fetch data for one trial.
-
classmethod
is_available_while_running() → bool[source]¶ Whether metrics of this class are available while the trial is running. Metrics that are not available while the trial is running are assumed to be available only upon trial completion. For such metrics, data is assumed to never change once the trial is completed.
NOTE: If this method returns False, data-fetching via experiment.fetch_data will return the data cached on the experiment (for the metrics of the given class) whenever its available. Data is cached on experiment when attached via experiment.attach_data.
-
Hartmann6¶
-
class
ax.metrics.hartmann6.AugmentedHartmann6Metric(name: str, param_names: List[str], noise_sd: Optional[float] = 0.0, lower_is_better: Optional[bool] = None)[source]¶
L2 Norm¶
Noisy Functions¶
-
class
ax.metrics.noisy_function.GenericNoisyFunctionMetric(name: str, f: Callable[[Dict[str, Optional[Union[str, bool, float, int]]]], float], noise_sd: Optional[float] = 0.0, lower_is_better: Optional[bool] = None)[source]¶ Bases:
ax.metrics.noisy_function.NoisyFunctionMetric-
clone() → ax.metrics.noisy_function.GenericNoisyFunctionMetric[source]¶ Create a copy of this Metric.
-
property
param_names¶
-
-
class
ax.metrics.noisy_function.NoisyFunctionMetric(name: str, param_names: List[str], noise_sd: Optional[float] = 0.0, lower_is_better: Optional[bool] = None)[source]¶ Bases:
ax.core.metric.MetricA metric defined by a generic deterministic function, with normal noise with mean 0 and mean_sd scale added to the result.
-
clone() → ax.metrics.noisy_function.NoisyFunctionMetric[source]¶ Create a copy of this Metric.
-
fetch_trial_data(trial: ax.core.base_trial.BaseTrial, noisy: bool = True, **kwargs: Any) → ax.core.data.Data[source]¶ Fetch data for one trial.
-
classmethod
is_available_while_running() → bool[source]¶ Whether metrics of this class are available while the trial is running. Metrics that are not available while the trial is running are assumed to be available only upon trial completion. For such metrics, data is assumed to never change once the trial is completed.
NOTE: If this method returns False, data-fetching via experiment.fetch_data will return the data cached on the experiment (for the metrics of the given class) whenever its available. Data is cached on experiment when attached via experiment.attach_data.
-
Noisy Function Map¶
-
class
ax.metrics.noisy_function_map.NoisyFunctionMapMetric(name: str, param_names: Iterable[str], map_key_infos: Iterable[ax.core.map_data.MapKeyInfo], noise_sd: float = 0.0, lower_is_better: Optional[bool] = None, cache_evaluations: bool = True)[source]¶ Bases:
ax.core.map_metric.MapMetricA metric defined by a generic deterministic function, with normal noise with mean 0 and mean_sd scale added to the result.
-
clone() → ax.metrics.noisy_function_map.NoisyFunctionMapMetric[source]¶ Create a copy of this Metric.
-
f(x: numpy.ndarray) → Mapping[str, Any][source]¶ The deterministic function that produces the metric outcomes.
-
fetch_trial_data(trial: ax.core.base_trial.BaseTrial, noisy: bool = True, **kwargs: Any) → ax.core.map_data.MapData[source]¶ Fetch data for one trial.
-
classmethod
is_available_while_running() → bool[source]¶ Whether metrics of this class are available while the trial is running. Metrics that are not available while the trial is running are assumed to be available only upon trial completion. For such metrics, data is assumed to never change once the trial is completed.
NOTE: If this method returns False, data-fetching via experiment.fetch_data will return the data cached on the experiment (for the metrics of the given class) whenever its available. Data is cached on experiment when attached via experiment.attach_data.
-
Sklearn¶
-
class
ax.metrics.sklearn.SklearnMetric(name: str, lower_is_better: bool = False, model_type: ax.metrics.sklearn.SklearnModelType = <SklearnModelType.RF: 'rf'>, dataset: ax.metrics.sklearn.SklearnDataset = <SklearnDataset.DIGITS: 'digits'>, observed_noise: bool = False, num_folds: int = 5)[source]¶ Bases:
ax.core.metric.MetricA metric that trains and evaluates an sklearn model.
The evaluation metric is the k-fold “score”. The scoring function depends on the model type and task type (e.g. classification/regression), but higher scores are better.
See sklearn documentation for supported parameters.
In addition, this metric supports tuning the hidden_layer_size and the number of hidden layers (num_hidden_layers) of a NN model.
-
clone() → ax.metrics.sklearn.SklearnMetric[source]¶ Create a copy of this Metric.
-
fetch_trial_data(trial: ax.core.base_trial.BaseTrial, noisy: bool = True, **kwargs: Any) → ax.core.data.Data[source]¶ Fetch data for one trial.
-
train_eval(arm: ax.core.arm.Arm) → Tuple[float, float][source]¶ Train and evaluate model.
- Parameters
arm – An arm specifying the parameters to evaluate.
- Returns
The average k-fold CV score
- The SE of the mean k-fold CV score if observed_noise is True
and ‘nan’ otherwise
- Return type
A two-element tuple containing
-
Tensorboard¶
-
class
ax.metrics.tensorboard.TensorboardCurveMetric(name: str, curve_name: str, lower_is_better: bool = True)[source]¶ Bases:
ax.metrics.curve.AbstractCurveMetricA CurveMetric for getting Tensorboard curves.
-
MAP_KEY= <ax.core.map_data.MapKeyInfo object>¶
-
classmethod
get_curves_from_ids(ids: Iterable[Union[int, str]]) → Dict[Union[int, str], Dict[str, pandas.Series]][source]¶ Get curve data from tensorboard logs.
NOTE: If the ids are not simple paths/posix locations, subclass this metric and replace this method with an appropriate one that retrieves the log results.
- Parameters
ids – A list of string paths to tensorboard log directories.
- Returns
A dictionary mapping metric names to pandas Series of data.
-