ax.benchmark¶

Benchmark Method¶

class ax.benchmark.benchmark_method.BenchmarkMethod(name: str, generation_strategy: GenerationStrategy, scheduler_options: SchedulerOptions, distribute_replications: bool = False, use_model_predictions_for_best_point: bool = False)[source]¶

Bases: Base

Benchmark method, represented in terms of Ax generation strategy (which tells us which models to use when) and scheduler options (which tell us extra execution information like maximum parallelism, early stopping configuration, etc.).

Note: If BenchmarkMethod.scheduler_options.total_trials is less than BenchmarkProblem.num_trials then only the number of trials specified in the former will be run.

Parameters:

• name – String description.

• generation_strategy – The GenerationStrategy to use.

• scheduler_options – SchedulerOptions that specify options such as max_pending_trials, timeout_hours, and batch_size. Can be generated with sensible defaults for benchmarking with get_benchmark_scheduler_options.

• distribute_replications – Indicates whether the replications should be run in a distributed manner. Ax itself does not use this attribute.

• use_model_predictions_for_best_point – Whether to use model predictions with get_pareto_optimal_parameters (if multi-objective) or BestPointMixin._get_best_trial (if single-objective). However, note that if multi-objective, best-point selection is not currently supported and get_pareto_optimal_parameters will raise a NotImplementedError.

distribute_replications: bool = False¶

generation_strategy: GenerationStrategy¶

get_best_parameters(experiment: Experiment, optimization_config: OptimizationConfig, n_points: int) → list[dict[str, None | str | bool | float | int]][source]¶

Get n_points promising points. NOTE: Only SOO with n_points = 1 is supported.

The expected use case is that these points will be evaluated against an oracle for hypervolume (if multi-objective) or for the value of the best parameter (if single-objective).

For multi-objective cases, n_points > 1 is needed. For SOO, n_points > 1 reflects setups where we can choose some points which will then be evaluated noiselessly or at high fidelity and then use the best one.

Parameters:

• experiment – The experiment to get the data from. This should contain values that would be observed in a realistic setting and not contain oracle values.

• optimization_config – The optimization_config for the corresponding BenchmarkProblem.

• n_points – The number of points to return.

name: str¶

scheduler_options: SchedulerOptions¶

use_model_predictions_for_best_point: bool = False¶

ax.benchmark.benchmark_method.get_benchmark_scheduler_options(timeout_hours: int = 4, batch_size: int = 1) → SchedulerOptions[source]¶

The typical SchedulerOptions used in benchmarking.

Currently, regardless of batch size, all pending trials must complete before new ones are generated. That is, when batch_size > 1, the design is “batch sequential”, and when batch_size = 1, the design is “fully sequential.”

Parameters:

• timeout_hours – The maximum amount of time (in hours) to run each benchmark replication. Defaults to 4 hours.

• batch_size – Number of trials to generate at once.

Benchmark Metric¶

class ax.benchmark.benchmark_metric.BenchmarkMetric(name: str, lower_is_better: bool, observe_noise_sd: bool = True, outcome_index: int | None = None)[source]¶

Bases: Metric

A generic metric used for observed values produced by Ax Benchmarks.

Compatible with results generated by BenchmarkRunner.

fetch_trial_data(trial: BaseTrial, **kwargs: Any) → Result[Data, MetricFetchE][source]¶

Parameters:

• trial – The trial from which to fetch data.

• kwargs – Unsupported and will raise an exception.

Returns:

A MetricFetchResult containing the data for the requested metric.

Benchmark Problem¶

class ax.benchmark.benchmark_problem.BenchmarkProblem(*, name: str, optimization_config: ~ax.core.optimization_config.OptimizationConfig, num_trials: int, test_function: ~ax.benchmark.benchmark_test_function.BenchmarkTestFunction, noise_std: float | list[float] | dict[str, float] = 0.0, optimal_value: float, search_space: ~ax.core.search_space.SearchSpace, report_inference_value_as_trace: bool = False, n_best_points: int = 1, target_fidelity_and_task: ~collections.abc.Mapping[str, None | str | bool | float | int] = <factory>)[source]¶

Bases: Base

Problem against which different methods can be benchmarked.

Defines how data is generated, the objective (via the OptimizationConfig), and the SearchSpace.

Parameters:

• name – Can be generated programmatically with _get_name.

• optimization_config – Defines the objective of optimization. Metrics must be `BenchmarkMetric`s.

• num_trials – Number of optimization iterations to run. BatchTrials count as one trial.

• optimal_value – The best ground-truth objective value. Hypervolume for multi-objective problems. If the best value is not known, it is conventional to set it to a value that is almost certainly better than the best value, so that a benchmark’s score will not exceed 100%.

• search_space – The search space.

• test_function – A BenchmarkTestFunction, which will generate noiseless data. This will be used by a BenchmarkRunner.

• noise_std – Describes how noise is added to the output of the test_function. If a float, IID random normal noise with that standard deviation is added. A list of floats, or a dict whose keys match test_functions.outcome_names, sets different noise standard deviations for the different outcomes produced by the test_function. This will be used by a BenchmarkRunner.

• report_inference_value_as_trace – Whether the optimization_trace on a BenchmarkResult should use the oracle_trace (if False, default) or the inference_trace. See BenchmarkResult for more information. Currently, this is only supported for single-objective problems.

• n_best_points – Number of points for a best-point selector to recommend. Currently, only n_best_points=1 is supported.

evaluate_oracle(parameters: Mapping[str, None | str | bool | float | int]) → ndarray[Any, dtype[_ScalarType_co]][source]¶

Evaluate oracle metric values at a parameterization. In the base class, oracle values are underlying noiseless function values evaluated at the target task and fidelity (if applicable).

This method can be customized for more complex setups based on different notions of what the “oracle” value should be. For example, with a preference-learned objective, the values might be true metrics evaluated at the true utility function (which would be unobserved in reality).

get_oracle_experiment_from_experiment(experiment: Experiment) → Experiment[source]¶

get_oracle_experiment_from_params(dict_of_dict_of_params: Mapping[int, Mapping[str, Mapping[str, None | str | bool | float | int]]]) → Experiment[source]¶

Get a new experiment with the same search space and optimization config as those belonging to this problem, but with parameterizations evaluated at oracle values.

Parameters:

dict_of_dict_of_params – Keys are trial indices, values are Mappings (e.g. dicts) that map arm names to parameterizations.

Example

>>> problem.get_oracle_experiment_from_params(
...     {
...         0: {
...            "0_0": {"x0": 0.0, "x1": 0.0},
...            "0_1": {"x0": 0.3, "x1": 0.4},
...         },
...         1: {"1_0": {"x0": 0.0, "x1": 0.0}},
...     }
... )

property is_moo: bool¶

Whether the problem is multi-objective.

n_best_points: int = 1¶

name: str¶

noise_std: float | list[float] | dict[str, float] = 0.0¶

num_trials: int¶

optimal_value: float¶

optimization_config: OptimizationConfig¶

report_inference_value_as_trace: bool = False¶

search_space: SearchSpace¶

target_fidelity_and_task: Mapping[str, None | str | bool | float | int]¶

test_function: BenchmarkTestFunction¶

ax.benchmark.benchmark_problem.create_problem_from_botorch(*, test_problem_class: type[BaseTestProblem], test_problem_kwargs: dict[str, Any], noise_std: float | list[float] = 0.0, num_trials: int, lower_is_better: bool = True, observe_noise_sd: bool = False, search_space: SearchSpace | None = None, report_inference_value_as_trace: bool = False) → BenchmarkProblem[source]¶

Create a BenchmarkProblem from a BoTorch BaseTestProblem.

Uses specialized Metrics and Runners for benchmarking. The test problem’s result will be computed by the Runner, BenchmarkRunner, and retrieved by the Metric(s), which are `BenchmarkMetric`s.

Parameters:

• test_problem_class – The BoTorch test problem class which will be used to define the search_space, optimization_config, and runner.

• test_problem_kwargs – Keyword arguments used to instantiate the test_problem_class.

• noise_std – Standard deviation of synthetic noise added to outcomes. If a float, the same noise level is used for all objectives.

• lower_is_better – Whether this is a minimization problem. For MOO, this applies to all objectives.

• num_trials – Simply the num_trials of the BenchmarkProblem created.

• observe_noise_sd – Whether the standard deviation of the observation noise is observed or not (in which case it must be inferred by the model). This is separate from whether synthetic noise is added to the problem, which is controlled by the noise_std of the test problem.

• search_space – If provided, the search_space of the BenchmarkProblem. Otherwise, a SearchSpace with all `RangeParameter`s is created from the bounds of the test problem.

• report_inference_value_as_trace – If True, indicates that the optimization_trace on a BenchmarkResult ought to be the inference_trace; otherwise, it will be the oracle_trace. See BenchmarkResult for more information.

ax.benchmark.benchmark_problem.get_continuous_search_space(bounds: list[tuple[float, float]]) → SearchSpace[source]¶

ax.benchmark.benchmark_problem.get_moo_opt_config_and_outcome_names(*, num_constraints: int, lower_is_better: bool, observe_noise_sd: bool, objective_names: list[str], ref_point: list[float]) → tuple[MultiObjectiveOptimizationConfig, list[str]][source]¶

ax.benchmark.benchmark_problem.get_soo_config_and_outcome_names(*, num_constraints: int, lower_is_better: bool, observe_noise_sd: bool, objective_name: str) → tuple[OptimizationConfig, list[str]][source]¶

Benchmark Result¶

class ax.benchmark.benchmark_result.AggregatedBenchmarkResult(name: str, results: list[BenchmarkResult], optimization_trace: DataFrame, score_trace: DataFrame, fit_time: list[float], gen_time: list[float])[source]¶

Bases: Base

The result of a benchmark test, or series of replications. Scalar data present in the BenchmarkResult is here represented as (mean, sem) pairs.

fit_time: list[float]¶

classmethod from_benchmark_results(results: list[BenchmarkResult]) → AggregatedBenchmarkResult[source]¶

Aggregrates a list of BenchmarkResults. For various reasons (timeout, errors, etc.) each BenchmarkResult may have a different number of trials; aggregated traces and statistics are computed with and truncated to the minimum trial count to ensure each replication is included.

gen_time: list[float]¶

name: str¶

optimization_trace: DataFrame¶

results: list[BenchmarkResult]¶

score_trace: DataFrame¶

class ax.benchmark.benchmark_result.BenchmarkResult(name: str, seed: int, oracle_trace: ndarray[Any, dtype[_ScalarType_co]], inference_trace: ndarray[Any, dtype[_ScalarType_co]], optimization_trace: ndarray[Any, dtype[_ScalarType_co]], score_trace: ndarray[Any, dtype[_ScalarType_co]], fit_time: float, gen_time: float, experiment: Experiment | None = None, experiment_storage_id: str | None = None)[source]¶

Bases: Base

The result of a single optimization loop from one (BenchmarkProblem, BenchmarkMethod) pair.

Parameters:

• name – Name of the benchmark. Should make it possible to determine the problem and the method.

• seed – Seed used for determinism.

• oracle_trace – For single-objective problems, element i of the optimization trace is the best oracle value of the arms evaluated after the first i trials. For multi-objective problems, element i of the optimization trace is the hypervolume of the oracle values of the arms in the first i trials (which may be ``BatchTrial``s). Oracle values are typically ground-truth (rather than noisy) and evaluated at the target task and fidelity.

• inference_trace –

  Inference trace comes from choosing a “best” point based only on data that would be observable in realistic settings and then evaluating the oracle value of that point. For multi-objective problems, we find a Pareto set and evaluate its hypervolume.

  There are several ways of specifying the “best” point: One could pick the point with the best observed value, or the point with the best model prediction, and could consider the whole search space, the set of trials completed so far, etc. How the inference trace is computed is specified by a best-point selector, which is an attribute of the BenchmarkMethod.

  Note: This is not “inference regret”, which is a lower-is-better value that is relative to the best possible value. The inference value trace is higher-is-better if the problem is a maximization problem or if the problem is multi-objective (in which case hypervolume is used). Hence, it is signed the same as oracle_trace and optimization_trace. score_trace is higher-is-better and relative to the optimum.

• optimization_trace – Either the oracle_trace or the inference_trace, depending on whether the BenchmarkProblem specifies report_inference_value. Having optimization_trace specified separately is useful when we need just one value to evaluate how well the benchmark went.

• score_trace – The scores associated with the problem, typically either the optimization_trace or inference_value_trace normalized to a 0-100 scale for comparability between problems.

• fit_time – Total time spent fitting models.

• gen_time – Total time spent generating candidates.

• experiment – If not None, the Ax experiment associated with the optimization that generated this data. Either experiment or experiment_storage_id must be provided.

• experiment_storage_id – Pointer to location where experiment data can be read.

experiment: Experiment | None = None¶

experiment_storage_id: str | None = None¶

fit_time: float¶

gen_time: float¶

inference_trace: ndarray[Any, dtype[_ScalarType_co]]¶

name: str¶

optimization_trace: ndarray[Any, dtype[_ScalarType_co]]¶

oracle_trace: ndarray[Any, dtype[_ScalarType_co]]¶

score_trace: ndarray[Any, dtype[_ScalarType_co]]¶

seed: int¶

Benchmark¶

Module for benchmarking Ax algorithms.

Key terms used:

• Replication: 1 run of an optimization loop; (BenchmarkProblem, BenchmarkMethod) pair.

• Test: multiple replications, ran for statistical significance.

• Full run: multiple tests on many (BenchmarkProblem, BenchmarkMethod) pairs.

• Method: (one of) the algorithm(s) being benchmarked.

• Problem: a synthetic function, a surrogate surface, or an ML model, on which to assess the performance of algorithms.

ax.benchmark.benchmark.benchmark_multiple_problems_methods(problems: Iterable[BenchmarkProblem], methods: Iterable[BenchmarkMethod], seeds: Iterable[int]) → list[AggregatedBenchmarkResult][source]¶

For each problem and method in the Cartesian product of problems and methods, run the replication on each seed in seeds and get the results as an AggregatedBenchmarkResult, then return a list of each AggregatedBenchmarkResult.

ax.benchmark.benchmark.benchmark_one_method_problem(problem: BenchmarkProblem, method: BenchmarkMethod, seeds: Iterable[int]) → AggregatedBenchmarkResult[source]¶

ax.benchmark.benchmark.benchmark_replication(problem: BenchmarkProblem, method: BenchmarkMethod, seed: int) → BenchmarkResult[source]¶

Run one benchmarking replication (equivalent to one optimization loop).

After each trial, the method gets the best parameter(s) found so far, as evaluated based on empirical data. After all trials are run, the problem gets the oracle values of each “best” parameter; this yields the inference trace. The cumulative maximum of the oracle value of each parameterization tested is the oracle_trace.

Parameters:

• problem – The BenchmarkProblem to test against (can be synthetic or real)

• method – The BenchmarkMethod to test

• seed – The seed to use for this replication.

Returns:

BenchmarkResult object.

ax.benchmark.benchmark.compute_score_trace(optimization_trace: ndarray[Any, dtype[_ScalarType_co]], num_baseline_trials: int, problem: BenchmarkProblem) → ndarray[Any, dtype[_ScalarType_co]][source]¶

Computes a score trace from the optimization trace.

ax.benchmark.benchmark.get_benchmark_runner(problem: BenchmarkProblem) → BenchmarkRunner[source]¶

Construct a BenchmarkRunner for the given problem.

Benchmark Runner¶

class ax.benchmark.benchmark_runner.BenchmarkRunner(*, test_function: BenchmarkTestFunction, noise_std: float | list[float] | dict[str, float] = 0.0)[source]¶

Bases: Runner

A Runner that produces both observed and ground-truth values.

Observed values equal ground-truth values plus noise, with the noise added according to the standard deviations returned by get_noise_stds().

This runner does require that every benchmark has a ground truth, which won’t necessarily be true for real-world problems. Such problems fall into two categories:

• If they are deterministic, they can be used with this runner by viewing them as noiseless problems where the observed values are the ground truth. The observed values will be used for tracking the progress of optimization.

• If they are not deterministc, they are not supported. It is not conceptually clear how to benchmark such problems, so we decided to not over-engineer for that before such a use case arrives.

Parameters:

• test_function – A BenchmarkTestFunction from which to generate deterministic data before adding noise.

• noise_std – The standard deviation of the noise added to the data. Can be a list or dict to be per-metric.

• search_space – Used to extract target fidelity and task.

classmethod deserialize_init_args(args: dict[str, Any], decoder_registry: dict[str, type[T] | Callable[[...], T]] | None = None, class_decoder_registry: dict[str, Callable[[dict[str, Any]], Any]] | None = None) → dict[str, Any][source]¶

It is tricky to use SerializationMixin with instances that have Ax objects as attributes, as BenchmarkRunners do. Therefore, serialization is not supported.

get_Y_true(params: Mapping[str, None | str | bool | float | int]) → Tensor[source]¶

Evaluates the test problem.

Returns:

An m-dim tensor of ground truth (noiseless) evaluations.

get_noise_stds() → dict[str, float][source]¶

noise_std: float | list[float] | dict[str, float] = 0.0¶

property outcome_names: list[str]¶

The names of the outcomes.

poll_trial_status(trials: Iterable[BaseTrial]) → dict[TrialStatus, set[int]][source]¶

Checks the status of any non-terminal trials and returns their indices as a mapping from TrialStatus to a list of indices. Required for runners used with Ax Scheduler.

NOTE: Does not need to handle waiting between polling calls while trials are running; this function should just perform a single poll.

Parameters:

trials – Trials to poll.

Returns:

A dictionary mapping TrialStatus to a list of trial indices that have the respective status at the time of the polling. This does not need to include trials that at the time of polling already have a terminal (ABANDONED, FAILED, COMPLETED) status (but it may).

run(trial: BaseTrial) → dict[str, Any][source]¶

Run the trial by evaluating its parameterization(s).

Parameters:

trial – The trial to evaluate.

Returns:

• Ys: A dict mapping arm names to lists of corresponding outcomes,

  where the order of the outcomes is the same as in outcome_names.

• Ystds: A dict mapping arm names to lists of corresponding outcome

  noise standard deviations (possibly nan if the noise level is unobserved), where the order of the outcomes is the same as in outcome_names.

• ”outcome_names”: A list of metric names.

Return type:

A dictionary with the following keys

classmethod serialize_init_args(obj: Any) → dict[str, Any][source]¶

It is tricky to use SerializationMixin with instances that have Ax objects as attributes, as BenchmarkRunners do. Therefore, serialization is not supported.

test_function: BenchmarkTestFunction¶

Benchmark Test Function¶

class ax.benchmark.benchmark_test_function.BenchmarkTestFunction(*, outcome_names: list[str])[source]¶

Bases: ABC

The basic Ax class for generating deterministic data to benchmark against.

(Noise - if desired - is added by the runner.)

abstract evaluate_true(params: Mapping[str, None | str | bool | float | int]) → Tensor[source]¶

Evaluate noiselessly.

Returns:

1d tensor of shape (len(outcome_names),).

outcome_names: list[str]¶

Benchmark Methods Modular BoTorch¶

ax.benchmark.methods.modular_botorch.get_sobol_botorch_modular_acquisition(model_cls: type[Model], acquisition_cls: type[AcquisitionFunction], distribute_replications: bool, scheduler_options: SchedulerOptions | None = None, name: str | None = None, num_sobol_trials: int = 5, model_gen_kwargs: dict[str, Any] | None = None, use_model_predictions_for_best_point: bool = False) → BenchmarkMethod[source]¶

Get a BenchmarkMethod that uses Sobol followed by MBM.

Parameters:

• model_cls – BoTorch model class, e.g. SingleTaskGP

• acquisition_cls – Acquisition function class, e.g. qLogNoisyExpectedImprovement.

• distribute_replications – Whether to use multiple machines

• scheduler_options – Passed as-is to scheduler. Default: get_benchmark_scheduler_options().

• name – Name that will be attached to the GenerationStrategy.

• num_sobol_trials – Number of Sobol trials; if the scheduler_options specify to use `BatchTrial`s, then this refers to the number of `BatchTrial`s.

• model_gen_kwargs – Passed to the BoTorch GenerationStep and ultimately to the BoTorch Model.

• use_model_predictions_for_best_point – Passed to the created BenchmarkMethod.

Example

>>> # A simple example
>>> from ax.benchmark.methods.sobol_botorch_modular import (
...     get_sobol_botorch_modular_acquisition
... )
>>> from ax.benchmark.benchmark_method import get_benchmark_scheduler_options
>>>
>>> method = get_sobol_botorch_modular_acquisition(
...     model_cls=SingleTaskGP,
...     acquisition_cls=qLogNoisyExpectedImprovement,
...     distribute_replications=False,
... )
>>> # Pass sequential=False to BoTorch's optimize_acqf
>>> batch_method = get_sobol_botorch_modular_acquisition(
...     model_cls=SingleTaskGP,
...     acquisition_cls=qLogNoisyExpectedImprovement,
...     distribute_replications=False,
...     scheduler_options=get_benchmark_scheduler_options(
...         batch_size=5,
...     ),
...     model_gen_kwargs={
...         "model_gen_options": {
...             "optimizer_kwargs": {"sequential": False}
...         }
...     },
...     num_sobol_trials=1,
... )

Benchmark Methods Sobol¶

ax.benchmark.methods.sobol.get_sobol_benchmark_method(distribute_replications: bool, scheduler_options: SchedulerOptions | None = None) → BenchmarkMethod[source]¶

Benchmark Problems Registry¶

class ax.benchmark.problems.registry.BenchmarkProblemRegistryEntry(factory_fn: collections.abc.Callable[..., ax.benchmark.benchmark_problem.BenchmarkProblem], factory_kwargs: dict[str, Any])[source]¶

Bases: object

factory_fn: Callable[[...], BenchmarkProblem]¶

factory_kwargs: dict[str, Any]¶

ax.benchmark.problems.registry.get_problem(problem_key: str, registry: Mapping[str, BenchmarkProblemRegistryEntry] | None = None, **additional_kwargs: Any) → BenchmarkProblem[source]¶

Generate a benchmark problem from a key, registry, and additional arguments.

Parameters:

• problem_key – The key by which a BenchmarkProblemRegistryEntry is looked up in the registry; a problem will then be generated from that entry and additional_kwargs. Note that this is not necessarily the same as the name attribute of the problem, and that one problem_key can generate several different BenchmarkProblem`s by passing `additional_kwargs. However, it is a good practice to maintain a 1:1 mapping between problem_key and the name.

• registry – If not provided, uses BENCHMARK_PROBLEM_REGISTRY to use problems defined within Ax.

• additional_kwargs – Additional kwargs to pass to the factory function of the BenchmarkProblemRegistryEntry.

Benchmark Problems High Dimensional Embedding¶

ax.benchmark.problems.hd_embedding.embed_higher_dimension(problem: TProblem, total_dimensionality: int) → TProblem[source]¶

Return a new BenchmarkProblem with enough RangeParameter`s added to the search space to make its total dimensionality equal to `total_dimensionality and add total_dimensionality to its name.

The search space of the original problem is within the search space of the new problem, and the constraints are copied from the original problem.

Benchmark Problems Mixed Integer Synthetic¶

Mixed integer extensions of some common synthetic test functions. These are adapted from [Daulton2022bopr].

References

ax.benchmark.problems.synthetic.discretized.mixed_integer.get_discrete_ackley(num_trials: int = 50, observe_noise_sd: bool = False, bounds: list[tuple[float, float]] | None = None) → BenchmarkProblem[source]¶

13D Ackley problem where first 10 dimensions are discretized.

This also restricts Ackley evaluation bounds to [0, 1].

ax.benchmark.problems.synthetic.discretized.mixed_integer.get_discrete_hartmann(num_trials: int = 50, observe_noise_sd: bool = False, bounds: list[tuple[float, float]] | None = None) → BenchmarkProblem[source]¶

6D Hartmann problem where first 4 dimensions are discretized.

ax.benchmark.problems.synthetic.discretized.mixed_integer.get_discrete_rosenbrock(num_trials: int = 50, observe_noise_sd: bool = False, bounds: list[tuple[float, float]] | None = None) → BenchmarkProblem[source]¶

10D Rosenbrock problem where first 6 dimensions are discretized.

Benchmark Problems Jenatton¶

class ax.benchmark.problems.synthetic.hss.jenatton.Jenatton(*, outcome_names: list[str])[source]¶

Bases: BenchmarkTestFunction

Jenatton test function for hierarchical search spaces.

evaluate_true(params: Mapping[str, float | int | None]) → Tensor[source]¶

Evaluate noiselessly.

Returns:

1d tensor of shape (len(outcome_names),).

ax.benchmark.problems.synthetic.hss.jenatton.get_jenatton_benchmark_problem(num_trials: int = 50, observe_noise_sd: bool = False, noise_std: float = 0.0) → BenchmarkProblem[source]¶

ax.benchmark.problems.synthetic.hss.jenatton.jenatton_test_function(x1: int | None = None, x2: int | None = None, x3: int | None = None, x4: float | None = None, x5: float | None = None, x6: float | None = None, x7: float | None = None, r8: float | None = None, r9: float | None = None) → float[source]¶

Jenatton test function for hierarchical search spaces.

This function is taken from:

R. Jenatton, C. Archambeau, J. González, and M. Seeger. Bayesian optimization with tree-structured dependencies. ICML 2017.

Benchmark Problems PyTorchCNN¶

Benchmark Problems PyTorchCNN TorchVision¶

class ax.benchmark.problems.hpo.torchvision.CNN[source]¶

Bases: Module

forward(x: Tensor) → Tensor[source]¶

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class ax.benchmark.problems.hpo.torchvision.PyTorchCNNTorchvisionBenchmarkTestFunction(*, outcome_names: list[str] = <factory>, name: str, device: torch.device = <factory>, train_loader: dataclasses.InitVar[torch.utils.data.dataloader.DataLoader | None] = None, test_loader: dataclasses.InitVar[torch.utils.data.dataloader.DataLoader | None] = None)[source]¶

Bases: BenchmarkTestFunction

device: device¶

evaluate_true(params: Mapping[str, int | float]) → Tensor[source]¶

Evaluate noiselessly.

Returns:

1d tensor of shape (len(outcome_names),).

name: str¶

outcome_names: list[str]¶

test_loader: dataclasses.InitVar[torch.utils.data.dataloader.DataLoader | None] = None¶

train_loader: dataclasses.InitVar[torch.utils.data.dataloader.DataLoader | None] = None¶

ax.benchmark.problems.hpo.torchvision.get_pytorch_cnn_torchvision_benchmark_problem(name: str, num_trials: int) → BenchmarkProblem[source]¶

ax.benchmark.problems.hpo.torchvision.train_and_evaluate(lr: float, momentum: float, weight_decay: float, step_size: int, gamma: float, device: device, train_loader: DataLoader, test_loader: DataLoader) → float[source]¶

Return the fraction of correctly classified test examples.

Benchmark Test Functions: BoTorch Test¶

class ax.benchmark.benchmark_test_functions.botorch_test.BoTorchTestFunction(*, outcome_names: list[str], botorch_problem: BaseTestProblem, modified_bounds: list[tuple[float, float]] | None = None)[source]¶

Bases: BenchmarkTestFunction

Class for generating data from a BoTorch BaseTestProblem.

Parameters:

• outcome_names – Names of outcomes. Should have the same length as the dimension of the test function, including constraints.

• botorch_problem – The BoTorch BaseTestProblem.

• modified_bounds – The bounds that are used by the Ax search space while optimizing the problem. If different from the bounds of the test problem, we project the parameters into the test problem bounds before evaluating the test problem. For example, if the test problem is defined on [0, 1] but the Ax search space is integers in [0, 10], an Ax parameter value of 5 will correspond to 0.5 while evaluating the test problem. If modified bounds are not provided, the test problem will be evaluated using the raw parameter values.

botorch_problem: BaseTestProblem¶

evaluate_true(params: Mapping[str, float | int]) → Tensor[source]¶

Evaluate noiselessly.

Returns:

1d tensor of shape (len(outcome_names),).

modified_bounds: list[tuple[float, float]] | None = None¶

outcome_names: list[str]¶

tensorize_params(params: Mapping[str, int | float]) → Tensor[source]¶

Benchmark Test Functions: Surrogate¶

class ax.benchmark.benchmark_test_functions.surrogate.SurrogateTestFunction(*, outcome_names: list[str], name: str, _surrogate: TorchModelBridge | None = None, _datasets: list[SupervisedDataset] | None = None, get_surrogate_and_datasets: None | Callable[[], tuple[TorchModelBridge, list[SupervisedDataset]]] = None)[source]¶

Bases: BenchmarkTestFunction

Data-generating function for surrogate benchmark problems.

Parameters:

• name – The name of the runner.

• outcome_names – Names of outcomes to return in evaluate_true, if the surrogate produces more outcomes than are needed.

• _surrogate – Either None, or a TorchModelBridge surrogate to use for generating observations. If None, get_surrogate_and_datasets must not be None and will be used to generate the surrogate when it is needed.

• _datasets – Either None, or the SupervisedDataset`s used to fit the surrogate model. If `None, get_surrogate_and_datasets must not be None and will be used to generate the datasets when they are needed.

• get_surrogate_and_datasets – Function that returns the surrogate and datasets, to allow for lazy construction. If get_surrogate_and_datasets is not provided, surrogate and datasets must be provided, and vice versa.

property datasets: list[SupervisedDataset]¶

evaluate_true(params: Mapping[str, None | str | bool | float | int]) → Tensor[source]¶

Evaluate noiselessly.

Returns:

1d tensor of shape (len(outcome_names),).

get_surrogate_and_datasets: None | Callable[[], tuple[TorchModelBridge, list[SupervisedDataset]]] = None¶

name: str¶

outcome_names: list[str]¶

set_surrogate_and_datasets() → None[source]¶

property surrogate: TorchModelBridge¶