Ax Service API with RayTune on PyTorch CNN¶

Ax integrates easily with different scheduling frameworks and distributed training frameworks. In this example, Ax-driven optimization is executed in a distributed fashion using RayTune.

RayTune is a scalable framework for hyperparameter tuning that provides many state-of-the-art hyperparameter tuning algorithms and seamlessly scales from laptop to distributed cluster with fault tolerance. RayTune leverages Ray's Actor API to provide asynchronous parallel and distributed execution.

Ray 'Actors' are a simple and clean abstraction for replicating your Python classes across multiple workers and nodes. Each hyperparameter evaluation is asynchronously executed on a separate Ray actor and reports intermediate training progress back to RayTune. Upon reporting, RayTune then uses this information to performs actions such as early termination, re-prioritization, or checkpointing.

In [1]:

import logging

from ray import tune
from ray.tune import report
from ray.tune.suggest.ax import AxSearch

logger = logging.getLogger(tune.__name__)
logger.setLevel(
    level=logging.CRITICAL
)  # Reduce the number of Ray warnings that are not relevant here.

/tmp/ipykernel_3063/1865771151.py:5: DeprecationWarning: The module `ray.tune.suggest` has been moved to `ray.tune.search` and the old location will be deprecated soon. Please adjust your imports to point to the new location. Example: Do a global search and replace `ray.tune.suggest` with `ray.tune.search`.
  from ray.tune.suggest.ax import AxSearch
/tmp/ipykernel_3063/1865771151.py:5: DeprecationWarning: The module `ray.tune.suggest.ax` has been moved to `ray.tune.search.ax` and the old location will be deprecated soon. Please adjust your imports to point to the new location. Example: Do a global search and replace `ray.tune.suggest.ax` with `ray.tune.search.ax`.
  from ray.tune.suggest.ax import AxSearch

In [2]:

import numpy as np
import torch
from ax.plot.contour import plot_contour
from ax.plot.trace import optimization_trace_single_method
from ax.service.ax_client import AxClient
from ax.utils.notebook.plotting import init_notebook_plotting, render
from ax.utils.tutorials.cnn_utils import CNN, evaluate, load_mnist, train

init_notebook_plotting()

[INFO 12-29 21:34:50] ax.utils.notebook.plotting: Injecting Plotly library into cell. Do not overwrite or delete cell.

1. Initialize client¶

We specify enforce_sequential_optimization as False, because Ray runs many trials in parallel. With the sequential optimization enforcement, AxClient would expect the first few trials to be completed with data before generating more trials.

When high parallelism is not required, it is best to enforce sequential optimization, as it allows for achieving optimal results in fewer (but sequential) trials. In cases where parallelism is important, such as with distributed training using Ray, we choose to forego minimizing resource utilization and run more trials in parallel.

In [3]:

ax = AxClient(enforce_sequential_optimization=False)

[INFO 12-29 21:34:50] ax.service.ax_client: Starting optimization with verbose logging. To disable logging, set the `verbose_logging` argument to `False`. Note that float values in the logs are rounded to 6 decimal points.

2. Set up experiment¶

Here we set up the search space and specify the objective; refer to the Ax API tutorials for more detail.

In [4]:

MINIMIZE = False  # Whether we should be minimizing or maximizing the objective

In [5]:

ax.create_experiment(
    name="mnist_experiment",
    parameters=[
        {"name": "lr", "type": "range", "bounds": [1e-6, 0.4], "log_scale": True},
        {"name": "momentum", "type": "range", "bounds": [0.0, 1.0]},
    ],
    objective_name="mean_accuracy",
    minimize=MINIMIZE,
)

[INFO 12-29 21:34:50] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter lr. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 12-29 21:34:50] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter momentum. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 12-29 21:34:50] ax.service.utils.instantiation: Created search space: SearchSpace(parameters=[RangeParameter(name='lr', parameter_type=FLOAT, range=[1e-06, 0.4], log_scale=True), RangeParameter(name='momentum', parameter_type=FLOAT, range=[0.0, 1.0])], parameter_constraints=[]).
[INFO 12-29 21:34:50] ax.modelbridge.dispatch_utils: Using Bayesian optimization since there are more ordered parameters than there are categories for the unordered categorical parameters.
[INFO 12-29 21:34:50] ax.modelbridge.dispatch_utils: Calculating the number of remaining initialization trials based on num_initialization_trials=None max_initialization_trials=None num_tunable_parameters=2 num_trials=None use_batch_trials=False
[INFO 12-29 21:34:50] ax.modelbridge.dispatch_utils: calculated num_initialization_trials=5
[INFO 12-29 21:34:50] ax.modelbridge.dispatch_utils: num_completed_initialization_trials=0 num_remaining_initialization_trials=5
[INFO 12-29 21:34:50] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+GPEI', steps=[Sobol for 5 trials, GPEI for subsequent trials]). Iterations after 5 will take longer to generate due to model-fitting.

In [6]:

ax.experiment.optimization_config.objective.minimize

Out[6]:

False

In [7]:

load_mnist(data_path="~/.data")  # Pre-load the dataset before the initial evaluations are executed.

Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to /home/runner/.data/MNIST/raw/train-images-idx3-ubyte.gz

  0%|          | 0/9912422 [00:00<?, ?it/s]

Extracting /home/runner/.data/MNIST/raw/train-images-idx3-ubyte.gz to /home/runner/.data/MNIST/raw

Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to /home/runner/.data/MNIST/raw/train-labels-idx1-ubyte.gz

  0%|          | 0/28881 [00:00<?, ?it/s]

Extracting /home/runner/.data/MNIST/raw/train-labels-idx1-ubyte.gz to /home/runner/.data/MNIST/raw

Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to /home/runner/.data/MNIST/raw/t10k-images-idx3-ubyte.gz

  0%|          | 0/1648877 [00:00<?, ?it/s]

Extracting /home/runner/.data/MNIST/raw/t10k-images-idx3-ubyte.gz to /home/runner/.data/MNIST/raw

Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to /home/runner/.data/MNIST/raw/t10k-labels-idx1-ubyte.gz

  0%|          | 0/4542 [00:00<?, ?it/s]

Extracting /home/runner/.data/MNIST/raw/t10k-labels-idx1-ubyte.gz to /home/runner/.data/MNIST/raw

Out[7]:

(<torch.utils.data.dataloader.DataLoader at 0x7fc4f8099dc0>,
 <torch.utils.data.dataloader.DataLoader at 0x7fc440fbb520>,
 <torch.utils.data.dataloader.DataLoader at 0x7fc440fbb640>)

3. Define how to evaluate trials¶

Since we use the Ax Service API here, we evaluate the parameterizations that Ax suggests, using RayTune. The evaluation function follows its usual pattern, taking in a parameterization and outputting an objective value. For detail on evaluation functions, see Trial Evaluation.

In [8]:

def train_evaluate(parameterization):
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    train_loader, valid_loader, test_loader = load_mnist(data_path="~/.data")
    net = train(
        net=CNN(),
        train_loader=train_loader,
        parameters=parameterization,
        dtype=torch.float,
        device=device,
    )
    report(
        mean_accuracy=evaluate(
            net=net,
            data_loader=valid_loader,
            dtype=torch.float,
            device=device,
        )
    )

4. Run optimization¶

Execute the Ax optimization and trial evaluation in RayTune using AxSearch algorithm:

In [9]:

# Set up AxSearcher in RayTune
algo = AxSearch(ax_client=ax)
# Wrap AxSearcher in a concurrently limiter, to ensure that Bayesian optimization receives the
# data for completed trials before creating more trials
algo = tune.suggest.ConcurrencyLimiter(algo, max_concurrent=3)
tune.run(
    train_evaluate,
    num_samples=30,
    search_alg=algo,
    verbose=0,  # Set this level to 1 to see status updates and to 2 to also see trial results.
    # To use GPU, specify: resources_per_trial={"gpu": 1}.
)

/home/runner/work/Ax/Ax/ax/core/observation.py:274: FutureWarning:

In a future version of pandas, a length 1 tuple will be returned when iterating over a groupby with a grouper equal to a list of length 1. Don't supply a list with a single grouper to avoid this warning.

[INFO 12-29 21:34:54] ax.service.ax_client: Generated new trial 0 with parameters {'lr': 0.000448, 'momentum': 0.15958}.
/home/runner/work/Ax/Ax/ax/core/observation.py:274: FutureWarning:

In a future version of pandas, a length 1 tuple will be returned when iterating over a groupby with a grouper equal to a list of length 1. Don't supply a list with a single grouper to avoid this warning.

[INFO 12-29 21:34:54] ax.service.ax_client: Generated new trial 1 with parameters {'lr': 3e-06, 'momentum': 0.796108}.
/home/runner/work/Ax/Ax/ax/core/observation.py:274: FutureWarning:

In a future version of pandas, a length 1 tuple will be returned when iterating over a groupby with a grouper equal to a list of length 1. Don't supply a list with a single grouper to avoid this warning.

[INFO 12-29 21:34:54] ax.service.ax_client: Generated new trial 2 with parameters {'lr': 1.2e-05, 'momentum': 0.538357}.
[INFO 12-29 21:35:07] ax.service.ax_client: Completed trial 1 with data: {'mean_accuracy': (0.5675, None)}.
[INFO 12-29 21:35:07] ax.service.ax_client: Generated new trial 3 with parameters {'lr': 0.015297, 'momentum': 0.677089}.
[INFO 12-29 21:35:07] ax.service.ax_client: Completed trial 0 with data: {'mean_accuracy': (0.929667, None)}.
[INFO 12-29 21:35:07] ax.service.ax_client: Generated new trial 4 with parameters {'lr': 0.121624, 'momentum': 0.407587}.
[INFO 12-29 21:35:16] ax.service.ax_client: Completed trial 2 with data: {'mean_accuracy': (0.746, None)}.
[INFO 12-29 21:35:16] ax.service.ax_client: Generated new trial 5 with parameters {'lr': 1.2e-05, 'momentum': 0.0}.
[INFO 12-29 21:35:17] ax.service.ax_client: Completed trial 3 with data: {'mean_accuracy': (0.112667, None)}.
[INFO 12-29 21:35:17] ax.service.ax_client: Generated new trial 6 with parameters {'lr': 0.006591, 'momentum': 0.0}.
[INFO 12-29 21:35:24] ax.service.ax_client: Completed trial 4 with data: {'mean_accuracy': (0.105333, None)}.
[INFO 12-29 21:35:25] ax.service.ax_client: Generated new trial 7 with parameters {'lr': 7e-05, 'momentum': 0.227281}.
[INFO 12-29 21:35:26] ax.service.ax_client: Completed trial 5 with data: {'mean_accuracy': (0.487333, None)}.
[INFO 12-29 21:35:27] ax.service.ax_client: Generated new trial 8 with parameters {'lr': 0.000214, 'momentum': 0.375103}.
[INFO 12-29 21:35:34] ax.service.ax_client: Completed trial 6 with data: {'mean_accuracy': (0.115667, None)}.
[INFO 12-29 21:35:35] ax.service.ax_client: Generated new trial 9 with parameters {'lr': 0.00021, 'momentum': 0.0}.
[INFO 12-29 21:35:36] ax.service.ax_client: Completed trial 7 with data: {'mean_accuracy': (0.869, None)}.
[INFO 12-29 21:35:37] ax.service.ax_client: Generated new trial 10 with parameters {'lr': 0.000118, 'momentum': 0.697606}.
[INFO 12-29 21:35:43] ax.service.ax_client: Completed trial 8 with data: {'mean_accuracy': (0.918167, None)}.
[INFO 12-29 21:35:45] ax.service.ax_client: Generated new trial 11 with parameters {'lr': 0.000223, 'momentum': 0.216812}.
[INFO 12-29 21:35:46] ax.service.ax_client: Completed trial 9 with data: {'mean_accuracy': (0.906667, None)}.
[INFO 12-29 21:35:47] ax.service.ax_client: Generated new trial 12 with parameters {'lr': 5e-05, 'momentum': 1.0}.
[INFO 12-29 21:35:53] ax.service.ax_client: Completed trial 10 with data: {'mean_accuracy': (0.931167, None)}.
[INFO 12-29 21:35:55] ax.service.ax_client: Generated new trial 13 with parameters {'lr': 0.000275, 'momentum': 1.0}.
[INFO 12-29 21:35:58] ax.service.ax_client: Completed trial 11 with data: {'mean_accuracy': (0.909167, None)}.
[INFO 12-29 21:35:59] ax.service.ax_client: Generated new trial 14 with parameters {'lr': 6.3e-05, 'momentum': 0.611518}.
[INFO 12-29 21:36:04] ax.service.ax_client: Completed trial 12 with data: {'mean_accuracy': (0.849667, None)}.
[INFO 12-29 21:36:06] ax.service.ax_client: Generated new trial 15 with parameters {'lr': 0.000407, 'momentum': 0.66411}.
[INFO 12-29 21:36:08] ax.service.ax_client: Completed trial 13 with data: {'mean_accuracy': (0.268333, None)}.
[INFO 12-29 21:36:10] ax.service.ax_client: Generated new trial 16 with parameters {'lr': 4.1e-05, 'momentum': 0.735952}.
[INFO 12-29 21:36:15] ax.service.ax_client: Completed trial 14 with data: {'mean_accuracy': (0.9135, None)}.
[INFO 12-29 21:36:17] ax.service.ax_client: Generated new trial 17 with parameters {'lr': 1.6e-05, 'momentum': 1.0}.
[INFO 12-29 21:36:19] ax.service.ax_client: Completed trial 15 with data: {'mean_accuracy': (0.93, None)}.
[INFO 12-29 21:36:21] ax.service.ax_client: Generated new trial 18 with parameters {'lr': 0.00029, 'momentum': 0.556927}.
[INFO 12-29 21:36:26] ax.service.ax_client: Completed trial 16 with data: {'mean_accuracy': (0.899, None)}.
[INFO 12-29 21:36:28] ax.service.ax_client: Generated new trial 19 with parameters {'lr': 0.000624, 'momentum': 0.476343}.
[INFO 12-29 21:36:32] ax.service.ax_client: Completed trial 17 with data: {'mean_accuracy': (0.875667, None)}.
[INFO 12-29 21:36:33] ax.service.ax_client: Generated new trial 20 with parameters {'lr': 0.000202, 'momentum': 0.584793}.
[INFO 12-29 21:36:38] ax.service.ax_client: Completed trial 18 with data: {'mean_accuracy': (0.945167, None)}.
[INFO 12-29 21:36:39] ax.service.ax_client: Generated new trial 21 with parameters {'lr': 0.000462, 'momentum': 0.0}.
[INFO 12-29 21:36:42] ax.service.ax_client: Completed trial 19 with data: {'mean_accuracy': (0.934667, None)}.
[INFO 12-29 21:36:43] ax.service.ax_client: Generated new trial 22 with parameters {'lr': 0.000451, 'momentum': 0.564712}.
[INFO 12-29 21:36:50] ax.service.ax_client: Completed trial 20 with data: {'mean_accuracy': (0.932333, None)}.
[INFO 12-29 21:36:52] ax.service.ax_client: Generated new trial 23 with parameters {'lr': 0.000496, 'momentum': 0.327785}.
[INFO 12-29 21:36:54] ax.service.ax_client: Completed trial 21 with data: {'mean_accuracy': (0.935333, None)}.
[INFO 12-29 21:36:55] ax.modelbridge.base: Untransformed parameter 0.40000000000000013 greater than upper bound 0.4, clamping
[INFO 12-29 21:36:55] ax.service.ax_client: Generated new trial 24 with parameters {'lr': 0.4, 'momentum': 1.0}.
[INFO 12-29 21:37:02] ax.service.ax_client: Completed trial 22 with data: {'mean_accuracy': (0.957, None)}.
[INFO 12-29 21:37:03] ax.service.ax_client: Generated new trial 25 with parameters {'lr': 1e-06, 'momentum': 0.249746}.
[INFO 12-29 21:37:06] ax.service.ax_client: Completed trial 23 with data: {'mean_accuracy': (0.946, None)}.
[INFO 12-29 21:37:08] ax.service.ax_client: Generated new trial 26 with parameters {'lr': 0.000413, 'momentum': 0.432858}.
[INFO 12-29 21:37:14] ax.service.ax_client: Completed trial 24 with data: {'mean_accuracy': (0.101167, None)}.
[INFO 12-29 21:37:15] ax.service.ax_client: Generated new trial 27 with parameters {'lr': 3e-05, 'momentum': 0.905257}.
[INFO 12-29 21:37:19] ax.service.ax_client: Completed trial 25 with data: {'mean_accuracy': (0.1205, None)}.
[INFO 12-29 21:37:21] ax.service.ax_client: Generated new trial 28 with parameters {'lr': 4.8e-05, 'momentum': 0.411281}.
[INFO 12-29 21:37:25] ax.service.ax_client: Completed trial 26 with data: {'mean_accuracy': (0.948833, None)}.
[INFO 12-29 21:37:26] ax.service.ax_client: Generated new trial 29 with parameters {'lr': 0.000384, 'momentum': 0.586623}.
[INFO 12-29 21:37:30] ax.service.ax_client: Completed trial 27 with data: {'mean_accuracy': (0.918333, None)}.
[INFO 12-29 21:37:35] ax.service.ax_client: Completed trial 28 with data: {'mean_accuracy': (0.876833, None)}.
[INFO 12-29 21:37:39] ax.service.ax_client: Completed trial 29 with data: {'mean_accuracy': (0.942, None)}.

Out[9]:

<ray.tune.analysis.experiment_analysis.ExperimentAnalysis at 0x7fc440e39c10>

5. Retrieve the optimization results¶

In [10]:

best_parameters, values = ax.get_best_parameters()
best_parameters

Out[10]:

{'lr': 0.00045105615350493, 'momentum': 0.5647121361805426}

In [11]:

means, covariances = values
means

Out[11]:

{'mean_accuracy': 0.9519302757733763}

6. Plot the response surface and optimization trace¶

In [12]:

render(
    plot_contour(
        model=ax.generation_strategy.model,
        param_x="lr",
        param_y="momentum",
        metric_name="mean_accuracy",
    )
)

In [13]:

# `plot_single_method` expects a 2-d array of means, because it expects to average means from multiple
# optimization runs, so we wrap out best objectives array in another array.
best_objectives = np.array(
    [[trial.objective_mean * 100 for trial in ax.experiment.trials.values()]]
)
best_objective_plot = optimization_trace_single_method(
    y=np.maximum.accumulate(best_objectives, axis=1),
    title="Model performance vs. # of iterations",
    ylabel="Accuracy",
)
render(best_objective_plot)

Download Tutorial Jupyter Notebook

Download Tutorial Source Code

Total runtime of script: 2 minutes, 58.04 seconds.