Ax integrates easily with different scheduling frameworks and distributed training frameworks. In this example, Ax-driven optimization is executed in a distributed fashion using RayTune.
RayTune is a scalable framework for hyperparameter tuning that provides many state-of-the-art hyperparameter tuning algorithms and seamlessly scales from laptop to distributed cluster with fault tolerance. RayTune leverages Ray's Actor API to provide asynchronous parallel and distributed execution.
Ray 'Actors' are a simple and clean abstraction for replicating your Python classes across multiple workers and nodes. Each hyperparameter evaluation is asynchronously executed on a separate Ray actor and reports intermediate training progress back to RayTune. Upon reporting, RayTune then uses this information to performs actions such as early termination, re-prioritization, or checkpointing.
import logging
from ray import tune
from ray.tune import track
from ray.tune.suggest.ax import AxSearch
logger = logging.getLogger(tune.__name__)
logger.setLevel(level=logging.CRITICAL) # Reduce the number of Ray warnings that are not relevant here.
import torch
import numpy as np
from ax.plot.contour import plot_contour
from ax.plot.trace import optimization_trace_single_method
from ax.service.ax_client import AxClient
from ax.utils.notebook.plotting import render, init_notebook_plotting
from ax.utils.tutorials.cnn_utils import load_mnist, train, evaluate
init_notebook_plotting()
[INFO 03-20 17:06:45] ipy_plotting: Injecting Plotly library into cell. Do not overwrite or delete cell.
We specify enforce_sequential_optimization as False, because Ray runs many trials in parallel. With the sequential optimization enforcement, AxClient would expect the first few trials to be completed with data before generating more trials.
When high parallelism is not required, it is best to enforce sequential optimization, as it allows for achieving optimal results in fewer (but sequential) trials. In cases where parallelism is important, such as with distributed training using Ray, we choose to forego minimizing resource utilization and run more trials in parallel.
ax = AxClient(enforce_sequential_optimization=False)
[INFO 03-20 17:06:46] ax.service.ax_client: Starting optimization with verbose logging. To disable logging, set the `verbose_logging` argument to `False`. Note that float values in the logs are rounded to 2 decimal points.
Here we set up the search space and specify the objective; refer to the Ax API tutorials for more detail.
ax.create_experiment(
name="mnist_experiment",
parameters=[
{"name": "lr", "type": "range", "bounds": [1e-6, 0.4], "log_scale": True},
{"name": "momentum", "type": "range", "bounds": [0.0, 1.0]},
],
objective_name="mean_accuracy",
)
[INFO 03-20 17:06:46] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+GPEI', steps=[Sobol for 5 trials, GPEI for subsequent trials]). Iterations after 5 will take longer to generate due to model-fitting.
Since we use the Ax Service API here, we evaluate the parameterizations that Ax suggests, using RayTune. The evaluation function follows its usual pattern, taking in a parameterization and outputting an objective value. For detail on evaluation functions, see Trial Evaluation.
def train_evaluate(parameterization):
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
train_loader, valid_loader, test_loader = load_mnist(data_path='~/.data')
net = train(train_loader=train_loader, parameters=parameterization, dtype=torch.float, device=device)
track.log(
mean_accuracy=evaluate(
net=net,
data_loader=valid_loader,
dtype=torch.float,
device=device,
)
)
Execute the Ax optimization and trial evaluation in RayTune using AxSearch algorithm:
tune.run(
train_evaluate,
num_samples=30,
search_alg=AxSearch(ax), # Note that the argument here is the `AxClient`.
verbose=0, # Set this level to 1 to see status updates and to 2 to also see trial results.
# To use GPU, specify: resources_per_trial={"gpu": 1}.
)
2020-03-20 17:06:46,086 WARNING services.py:586 -- setpgrp failed, processes may not be cleaned up properly: [Errno 1] Operation not permitted.
2020-03-20 17:06:46,088 INFO resource_spec.py:212 -- Starting Ray with 4.35 GiB memory available for workers and up to 2.18 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).
2020-03-20 17:06:46,446 INFO services.py:1078 -- View the Ray dashboard at localhost:8265
[INFO 03-20 17:06:47] ax.service.ax_client: Generated new trial 0 with parameters {'lr': 0.04, 'momentum': 0.39}.
[INFO 03-20 17:06:47] ax.service.ax_client: Generated new trial 1 with parameters {'lr': 0.22, 'momentum': 0.05}.
[INFO 03-20 17:06:47] ax.service.ax_client: Generated new trial 2 with parameters {'lr': 0.0, 'momentum': 0.08}.
[INFO 03-20 17:06:47] ax.service.ax_client: Generated new trial 3 with parameters {'lr': 0.37, 'momentum': 0.77}.
[INFO 03-20 17:06:47] ax.service.ax_client: Generated new trial 4 with parameters {'lr': 0.0, 'momentum': 0.98}.
[INFO 03-20 17:06:47] ax.service.ax_client: Generated new trial 5 with parameters {'lr': 0.23, 'momentum': 0.39}.
[INFO 03-20 17:06:48] ax.service.ax_client: Generated new trial 6 with parameters {'lr': 0.11, 'momentum': 0.89}.
[INFO 03-20 17:06:48] ax.service.ax_client: Generated new trial 7 with parameters {'lr': 0.0, 'momentum': 0.32}.
[INFO 03-20 17:06:48] ax.service.ax_client: Generated new trial 8 with parameters {'lr': 0.01, 'momentum': 0.76}.
[INFO 03-20 17:06:48] ax.service.ax_client: Generated new trial 9 with parameters {'lr': 0.0, 'momentum': 0.31}.
(pid=4076) Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to /home/travis/.data/MNIST/raw/train-images-idx3-ubyte.gz (pid=4077) Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to /home/travis/.data/MNIST/raw/train-images-idx3-ubyte.gz 0.1% 0.9% 4.3% 0.1% 20.6% 20.7% 0.9% 47.1% 1.0% 49.3% 49.7% 1.2% 50.7% 51.2% 1.2% 52.1% 1.3% 52.5% 1.4% 52.9% 1.5% 53.3% 1.6% 1.7% 7.9% 78.8% 33.4% 100.1% 77.3% (pid=4076) Extracting /home/travis/.data/MNIST/raw/train-images-idx3-ubyte.gz to /home/travis/.data/MNIST/raw 100.1% (pid=4077) Extracting /home/travis/.data/MNIST/raw/train-images-idx3-ubyte.gz to /home/travis/.data/MNIST/raw (pid=4077) Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to /home/travis/.data/MNIST/raw/train-labels-idx1-ubyte.gz (pid=4076) Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to /home/travis/.data/MNIST/raw/train-labels-idx1-ubyte.gz 28.4% 28.4% 113.5% (pid=4077) Extracting /home/travis/.data/MNIST/raw/train-labels-idx1-ubyte.gz to /home/travis/.data/MNIST/raw (pid=4077) Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to /home/travis/.data/MNIST/raw/t10k-images-idx3-ubyte.gz (pid=4076) Extracting /home/travis/.data/MNIST/raw/train-labels-idx1-ubyte.gz to /home/travis/.data/MNIST/raw (pid=4076) Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to /home/travis/.data/MNIST/raw/t10k-images-idx3-ubyte.gz 113.5% 0.5% 0.5% 5.5% 25.8% 18.4% 100.4% (pid=4077) Extracting /home/travis/.data/MNIST/raw/t10k-images-idx3-ubyte.gz to /home/travis/.data/MNIST/raw (pid=4076) Extracting /home/travis/.data/MNIST/raw/t10k-images-idx3-ubyte.gz to /home/travis/.data/MNIST/raw 100.4% (pid=4077) Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to /home/travis/.data/MNIST/raw/t10k-labels-idx1-ubyte.gz (pid=4076) Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to /home/travis/.data/MNIST/raw/t10k-labels-idx1-ubyte.gz 180.4% (pid=4077) Extracting /home/travis/.data/MNIST/raw/t10k-labels-idx1-ubyte.gz to /home/travis/.data/MNIST/raw (pid=4077) Processing... (pid=4076) Extracting /home/travis/.data/MNIST/raw/t10k-labels-idx1-ubyte.gz to /home/travis/.data/MNIST/raw (pid=4076) Processing... 180.4% (pid=4077) Done! (pid=4077) 2020-03-20 17:06:53,721 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4077) Traceback (most recent call last): (pid=4077) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4077) self._entrypoint() (pid=4077) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4077) return self._trainable_func(config, self._status_reporter) (pid=4077) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4077) output = train_func(config) (pid=4077) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4077) TypeError: train() missing 1 required positional argument: 'net' (pid=4077) Exception in thread Thread-1: (pid=4077) Traceback (most recent call last): (pid=4077) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4077) self.run() (pid=4077) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4077) raise e (pid=4077) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4077) self._entrypoint() (pid=4077) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4077) return self._trainable_func(config, self._status_reporter) (pid=4077) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4077) output = train_func(config) (pid=4077) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4077) TypeError: train() missing 1 required positional argument: 'net' (pid=4077) (pid=4076) Done!
[INFO 03-20 17:06:53] ax.service.ax_client: Generated new trial 10 with parameters {'lr': 0.02, 'momentum': 0.08}.
(pid=4076) 2020-03-20 17:06:53,740 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4076) Traceback (most recent call last): (pid=4076) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4076) self._entrypoint() (pid=4076) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4076) return self._trainable_func(config, self._status_reporter) (pid=4076) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4076) output = train_func(config) (pid=4076) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4076) TypeError: train() missing 1 required positional argument: 'net' (pid=4076) Exception in thread Thread-1: (pid=4076) Traceback (most recent call last): (pid=4076) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4076) self.run() (pid=4076) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4076) raise e (pid=4076) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4076) self._entrypoint() (pid=4076) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4076) return self._trainable_func(config, self._status_reporter) (pid=4076) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4076) output = train_func(config) (pid=4076) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4076) TypeError: train() missing 1 required positional argument: 'net' (pid=4076)
[INFO 03-20 17:06:54] ax.service.ax_client: Generated new trial 11 with parameters {'lr': 0.0, 'momentum': 0.88}.
(pid=4156) 2020-03-20 17:06:57,666 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4156) Traceback (most recent call last): (pid=4156) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4156) self._entrypoint() (pid=4156) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4156) return self._trainable_func(config, self._status_reporter) (pid=4156) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4156) output = train_func(config) (pid=4156) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4156) TypeError: train() missing 1 required positional argument: 'net' (pid=4156) Exception in thread Thread-1: (pid=4156) Traceback (most recent call last): (pid=4156) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4156) self.run() (pid=4156) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4156) raise e (pid=4156) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4156) self._entrypoint() (pid=4156) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4156) return self._trainable_func(config, self._status_reporter) (pid=4156) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4156) output = train_func(config) (pid=4156) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4156) TypeError: train() missing 1 required positional argument: 'net' (pid=4156) (pid=4157) 2020-03-20 17:06:57,755 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4157) Traceback (most recent call last): (pid=4157) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4157) self._entrypoint() (pid=4157) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4157) return self._trainable_func(config, self._status_reporter) (pid=4157) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4157) output = train_func(config) (pid=4157) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4157) TypeError: train() missing 1 required positional argument: 'net' (pid=4157) Exception in thread Thread-1: (pid=4157) Traceback (most recent call last): (pid=4157) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4157) self.run() (pid=4157) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4157) raise e (pid=4157) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4157) self._entrypoint() (pid=4157) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4157) return self._trainable_func(config, self._status_reporter) (pid=4157) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4157) output = train_func(config) (pid=4157) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4157) TypeError: train() missing 1 required positional argument: 'net' (pid=4157)
[INFO 03-20 17:06:57] ax.service.ax_client: Generated new trial 12 with parameters {'lr': 0.0, 'momentum': 0.3}.
[INFO 03-20 17:06:58] ax.service.ax_client: Generated new trial 13 with parameters {'lr': 0.0, 'momentum': 0.06}.
(pid=4168) 2020-03-20 17:07:00,677 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4168) Traceback (most recent call last): (pid=4168) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4168) self._entrypoint() (pid=4168) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4168) return self._trainable_func(config, self._status_reporter) (pid=4168) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4168) output = train_func(config) (pid=4168) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4168) TypeError: train() missing 1 required positional argument: 'net' (pid=4168) Exception in thread Thread-1: (pid=4168) Traceback (most recent call last): (pid=4168) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4168) self.run() (pid=4168) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4168) raise e (pid=4168) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4168) self._entrypoint() (pid=4168) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4168) return self._trainable_func(config, self._status_reporter) (pid=4168) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4168) output = train_func(config) (pid=4168) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4168) TypeError: train() missing 1 required positional argument: 'net' (pid=4168)
[INFO 03-20 17:07:01] ax.service.ax_client: Generated new trial 14 with parameters {'lr': 0.0, 'momentum': 0.87}.
(pid=4190) 2020-03-20 17:07:01,721 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4190) Traceback (most recent call last): (pid=4190) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4190) self._entrypoint() (pid=4190) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4190) return self._trainable_func(config, self._status_reporter) (pid=4190) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4190) output = train_func(config) (pid=4190) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4190) TypeError: train() missing 1 required positional argument: 'net' (pid=4190) Exception in thread Thread-1: (pid=4190) Traceback (most recent call last): (pid=4190) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4190) self.run() (pid=4190) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4190) raise e (pid=4190) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4190) self._entrypoint() (pid=4190) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4190) return self._trainable_func(config, self._status_reporter) (pid=4190) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4190) output = train_func(config) (pid=4190) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4190) TypeError: train() missing 1 required positional argument: 'net' (pid=4190)
[INFO 03-20 17:07:02] ax.service.ax_client: Generated new trial 15 with parameters {'lr': 0.0, 'momentum': 0.67}.
(pid=4204) 2020-03-20 17:07:04,995 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4204) Traceback (most recent call last): (pid=4204) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4204) self._entrypoint() (pid=4204) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4204) return self._trainable_func(config, self._status_reporter) (pid=4204) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4204) output = train_func(config) (pid=4204) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4204) TypeError: train() missing 1 required positional argument: 'net' (pid=4204) Exception in thread Thread-1: (pid=4204) Traceback (most recent call last): (pid=4204) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4204) self.run() (pid=4204) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4204) raise e (pid=4204) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4204) self._entrypoint() (pid=4204) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4204) return self._trainable_func(config, self._status_reporter) (pid=4204) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4204) output = train_func(config) (pid=4204) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4204) TypeError: train() missing 1 required positional argument: 'net' (pid=4204) (pid=4205) 2020-03-20 17:07:05,140 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4205) Traceback (most recent call last): (pid=4205) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4205) self._entrypoint() (pid=4205) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4205) return self._trainable_func(config, self._status_reporter) (pid=4205) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4205) output = train_func(config) (pid=4205) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4205) TypeError: train() missing 1 required positional argument: 'net' (pid=4205) Exception in thread Thread-1: (pid=4205) Traceback (most recent call last): (pid=4205) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4205) self.run() (pid=4205) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4205) raise e (pid=4205) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4205) self._entrypoint() (pid=4205) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4205) return self._trainable_func(config, self._status_reporter) (pid=4205) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4205) output = train_func(config) (pid=4205) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4205) TypeError: train() missing 1 required positional argument: 'net' (pid=4205)
[INFO 03-20 17:07:05] ax.service.ax_client: Generated new trial 16 with parameters {'lr': 0.0, 'momentum': 0.19}.
[INFO 03-20 17:07:05] ax.service.ax_client: Generated new trial 17 with parameters {'lr': 0.05, 'momentum': 0.86}.
(pid=4215) 2020-03-20 17:07:07,964 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4215) Traceback (most recent call last): (pid=4215) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4215) self._entrypoint() (pid=4215) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4215) return self._trainable_func(config, self._status_reporter) (pid=4215) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4215) output = train_func(config) (pid=4215) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4215) TypeError: train() missing 1 required positional argument: 'net' (pid=4215) Exception in thread Thread-1: (pid=4215) Traceback (most recent call last): (pid=4215) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4215) self.run() (pid=4215) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4215) raise e (pid=4215) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4215) self._entrypoint() (pid=4215) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4215) return self._trainable_func(config, self._status_reporter) (pid=4215) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4215) output = train_func(config) (pid=4215) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4215) TypeError: train() missing 1 required positional argument: 'net' (pid=4215)
[INFO 03-20 17:07:08] ax.service.ax_client: Generated new trial 18 with parameters {'lr': 0.0, 'momentum': 0.6}.
(pid=4238) 2020-03-20 17:07:09,208 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4238) Traceback (most recent call last): (pid=4238) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4238) self._entrypoint() (pid=4238) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4238) return self._trainable_func(config, self._status_reporter) (pid=4238) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4238) output = train_func(config) (pid=4238) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4238) TypeError: train() missing 1 required positional argument: 'net' (pid=4238) Exception in thread Thread-1: (pid=4238) Traceback (most recent call last): (pid=4238) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4238) self.run() (pid=4238) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4238) raise e (pid=4238) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4238) self._entrypoint() (pid=4238) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4238) return self._trainable_func(config, self._status_reporter) (pid=4238) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4238) output = train_func(config) (pid=4238) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4238) TypeError: train() missing 1 required positional argument: 'net' (pid=4238)
[INFO 03-20 17:07:09] ax.service.ax_client: Generated new trial 19 with parameters {'lr': 0.02, 'momentum': 0.7}.
(pid=4252) 2020-03-20 17:07:12,226 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4252) Traceback (most recent call last): (pid=4252) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4252) self._entrypoint() (pid=4252) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4252) return self._trainable_func(config, self._status_reporter) (pid=4252) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4252) output = train_func(config) (pid=4252) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4252) TypeError: train() missing 1 required positional argument: 'net' (pid=4252) Exception in thread Thread-1: (pid=4252) Traceback (most recent call last): (pid=4252) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4252) self.run() (pid=4252) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4252) raise e (pid=4252) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4252) self._entrypoint() (pid=4252) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4252) return self._trainable_func(config, self._status_reporter) (pid=4252) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4252) output = train_func(config) (pid=4252) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4252) TypeError: train() missing 1 required positional argument: 'net' (pid=4252)
[INFO 03-20 17:07:12] ax.service.ax_client: Generated new trial 20 with parameters {'lr': 0.1, 'momentum': 0.12}.
(pid=4257) 2020-03-20 17:07:12,664 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4257) Traceback (most recent call last): (pid=4257) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4257) self._entrypoint() (pid=4257) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4257) return self._trainable_func(config, self._status_reporter) (pid=4257) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4257) output = train_func(config) (pid=4257) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4257) TypeError: train() missing 1 required positional argument: 'net' (pid=4257) Exception in thread Thread-1: (pid=4257) Traceback (most recent call last): (pid=4257) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4257) self.run() (pid=4257) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4257) raise e (pid=4257) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4257) self._entrypoint() (pid=4257) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4257) return self._trainable_func(config, self._status_reporter) (pid=4257) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4257) output = train_func(config) (pid=4257) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4257) TypeError: train() missing 1 required positional argument: 'net' (pid=4257)
[INFO 03-20 17:07:13] ax.service.ax_client: Generated new trial 21 with parameters {'lr': 0.0, 'momentum': 0.91}.
(pid=4263) 2020-03-20 17:07:15,430 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4263) Traceback (most recent call last): (pid=4263) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4263) self._entrypoint() (pid=4263) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4263) return self._trainable_func(config, self._status_reporter) (pid=4263) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4263) output = train_func(config) (pid=4263) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4263) TypeError: train() missing 1 required positional argument: 'net' (pid=4263) Exception in thread Thread-1: (pid=4263) Traceback (most recent call last): (pid=4263) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4263) self.run() (pid=4263) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4263) raise e (pid=4263) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4263) self._entrypoint() (pid=4263) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4263) return self._trainable_func(config, self._status_reporter) (pid=4263) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4263) output = train_func(config) (pid=4263) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4263) TypeError: train() missing 1 required positional argument: 'net' (pid=4263)
[INFO 03-20 17:07:15] ax.service.ax_client: Generated new trial 22 with parameters {'lr': 0.0, 'momentum': 0.49}.
2020-03-20 17:07:16,580 WARNING worker.py:1058 -- The actor or task with ID ffffffffffffffffa9aba8520100 is pending and cannot currently be scheduled. It requires {CPU: 1.000000} for execution and {CPU: 1.000000} for placement, but this node only has remaining {node:10.30.0.115: 1.000000}, {CPU: 2.000000}, {memory: 4.345703 GiB}, {object_store_memory: 1.464844 GiB}. In total there are 0 pending tasks and 1 pending actors on this node. This is likely due to all cluster resources being claimed by actors. To resolve the issue, consider creating fewer actors or increase the resources available to this Ray cluster. You can ignore this message if this Ray cluster is expected to auto-scale.
(pid=4286) 2020-03-20 17:07:16,444 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4286) Traceback (most recent call last): (pid=4286) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4286) self._entrypoint() (pid=4286) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4286) return self._trainable_func(config, self._status_reporter) (pid=4286) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4286) output = train_func(config) (pid=4286) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4286) TypeError: train() missing 1 required positional argument: 'net' (pid=4286) Exception in thread Thread-1: (pid=4286) Traceback (most recent call last): (pid=4286) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4286) self.run() (pid=4286) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4286) raise e (pid=4286) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4286) self._entrypoint() (pid=4286) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4286) return self._trainable_func(config, self._status_reporter) (pid=4286) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4286) output = train_func(config) (pid=4286) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4286) TypeError: train() missing 1 required positional argument: 'net' (pid=4286)
[INFO 03-20 17:07:16] ax.service.ax_client: Generated new trial 23 with parameters {'lr': 0.0, 'momentum': 0.02}.
(pid=4301) 2020-03-20 17:07:19,788 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4301) Traceback (most recent call last): (pid=4301) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4301) self._entrypoint() (pid=4301) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4301) return self._trainable_func(config, self._status_reporter) (pid=4301) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4301) output = train_func(config) (pid=4301) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4301) TypeError: train() missing 1 required positional argument: 'net' (pid=4301) Exception in thread Thread-1: (pid=4301) Traceback (most recent call last): (pid=4301) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4301) self.run() (pid=4301) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4301) raise e (pid=4301) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4301) self._entrypoint() (pid=4301) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4301) return self._trainable_func(config, self._status_reporter) (pid=4301) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4301) output = train_func(config) (pid=4301) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4301) TypeError: train() missing 1 required positional argument: 'net' (pid=4301)
[INFO 03-20 17:07:20] ax.service.ax_client: Generated new trial 24 with parameters {'lr': 0.0, 'momentum': 0.56}.
(pid=4302) 2020-03-20 17:07:19,969 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4302) Traceback (most recent call last): (pid=4302) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4302) self._entrypoint() (pid=4302) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4302) return self._trainable_func(config, self._status_reporter) (pid=4302) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4302) output = train_func(config) (pid=4302) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4302) TypeError: train() missing 1 required positional argument: 'net' (pid=4302) Exception in thread Thread-1: (pid=4302) Traceback (most recent call last): (pid=4302) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4302) self.run() (pid=4302) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4302) raise e (pid=4302) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4302) self._entrypoint() (pid=4302) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4302) return self._trainable_func(config, self._status_reporter) (pid=4302) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4302) output = train_func(config) (pid=4302) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4302) TypeError: train() missing 1 required positional argument: 'net' (pid=4302)
[INFO 03-20 17:07:20] ax.service.ax_client: Generated new trial 25 with parameters {'lr': 0.0, 'momentum': 0.68}.
(pid=4309) 2020-03-20 17:07:22,852 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4309) Traceback (most recent call last): (pid=4309) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4309) self._entrypoint() (pid=4309) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4309) return self._trainable_func(config, self._status_reporter) (pid=4309) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4309) output = train_func(config) (pid=4309) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4309) TypeError: train() missing 1 required positional argument: 'net' (pid=4309) Exception in thread Thread-1: (pid=4309) Traceback (most recent call last): (pid=4309) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4309) self.run() (pid=4309) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4309) raise e (pid=4309) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4309) self._entrypoint() (pid=4309) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4309) return self._trainable_func(config, self._status_reporter) (pid=4309) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4309) output = train_func(config) (pid=4309) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4309) TypeError: train() missing 1 required positional argument: 'net' (pid=4309)
[INFO 03-20 17:07:23] ax.service.ax_client: Generated new trial 26 with parameters {'lr': 0.0, 'momentum': 0.32}.
(pid=4336) 2020-03-20 17:07:23,971 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4336) Traceback (most recent call last): (pid=4336) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4336) self._entrypoint() (pid=4336) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4336) return self._trainable_func(config, self._status_reporter) (pid=4336) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4336) output = train_func(config) (pid=4336) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4336) TypeError: train() missing 1 required positional argument: 'net' (pid=4336) Exception in thread Thread-1: (pid=4336) Traceback (most recent call last): (pid=4336) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4336) self.run() (pid=4336) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4336) raise e (pid=4336) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4336) self._entrypoint() (pid=4336) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4336) return self._trainable_func(config, self._status_reporter) (pid=4336) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4336) output = train_func(config) (pid=4336) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4336) TypeError: train() missing 1 required positional argument: 'net' (pid=4336)
[INFO 03-20 17:07:24] ax.service.ax_client: Generated new trial 27 with parameters {'lr': 0.0, 'momentum': 0.51}.
(pid=4350) 2020-03-20 17:07:27,284 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4350) Traceback (most recent call last): (pid=4350) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4350) self._entrypoint() (pid=4350) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4350) return self._trainable_func(config, self._status_reporter) (pid=4350) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4350) output = train_func(config) (pid=4350) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4350) TypeError: train() missing 1 required positional argument: 'net' (pid=4350) Exception in thread Thread-1: (pid=4350) Traceback (most recent call last): (pid=4350) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4350) self.run() (pid=4350) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4350) raise e (pid=4350) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4350) self._entrypoint() (pid=4350) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4350) return self._trainable_func(config, self._status_reporter) (pid=4350) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4350) output = train_func(config) (pid=4350) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4350) TypeError: train() missing 1 required positional argument: 'net' (pid=4350)
[INFO 03-20 17:07:27] ax.service.ax_client: Generated new trial 28 with parameters {'lr': 0.0, 'momentum': 0.56}.
(pid=4351) 2020-03-20 17:07:27,718 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4351) Traceback (most recent call last): (pid=4351) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4351) self._entrypoint() (pid=4351) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4351) return self._trainable_func(config, self._status_reporter) (pid=4351) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4351) output = train_func(config) (pid=4351) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4351) TypeError: train() missing 1 required positional argument: 'net' (pid=4351) Exception in thread Thread-1: (pid=4351) Traceback (most recent call last): (pid=4351) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4351) self.run() (pid=4351) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4351) raise e (pid=4351) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4351) self._entrypoint() (pid=4351) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4351) return self._trainable_func(config, self._status_reporter) (pid=4351) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4351) output = train_func(config) (pid=4351) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4351) TypeError: train() missing 1 required positional argument: 'net' (pid=4351)
[INFO 03-20 17:07:28] ax.service.ax_client: Generated new trial 29 with parameters {'lr': 0.06, 'momentum': 0.36}.
(pid=4361) 2020-03-20 17:07:30,515 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4361) Traceback (most recent call last): (pid=4361) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4361) self._entrypoint() (pid=4361) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4361) return self._trainable_func(config, self._status_reporter) (pid=4361) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4361) output = train_func(config) (pid=4361) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4361) TypeError: train() missing 1 required positional argument: 'net' (pid=4361) Exception in thread Thread-1: (pid=4361) Traceback (most recent call last): (pid=4361) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4361) self.run() (pid=4361) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4361) raise e (pid=4361) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4361) self._entrypoint() (pid=4361) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4361) return self._trainable_func(config, self._status_reporter) (pid=4361) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4361) output = train_func(config) (pid=4361) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4361) TypeError: train() missing 1 required positional argument: 'net' (pid=4361) (pid=4384) 2020-03-20 17:07:31,689 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4384) Traceback (most recent call last): (pid=4384) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4384) self._entrypoint() (pid=4384) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4384) return self._trainable_func(config, self._status_reporter) (pid=4384) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4384) output = train_func(config) (pid=4384) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4384) TypeError: train() missing 1 required positional argument: 'net' (pid=4384) Exception in thread Thread-1: (pid=4384) Traceback (most recent call last): (pid=4384) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4384) self.run() (pid=4384) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4384) raise e (pid=4384) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4384) self._entrypoint() (pid=4384) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4384) return self._trainable_func(config, self._status_reporter) (pid=4384) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4384) output = train_func(config) (pid=4384) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4384) TypeError: train() missing 1 required positional argument: 'net' (pid=4384) (pid=4398) 2020-03-20 17:07:34,124 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4398) Traceback (most recent call last): (pid=4398) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4398) self._entrypoint() (pid=4398) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4398) return self._trainable_func(config, self._status_reporter) (pid=4398) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4398) output = train_func(config) (pid=4398) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4398) TypeError: train() missing 1 required positional argument: 'net' (pid=4398) Exception in thread Thread-1: (pid=4398) Traceback (most recent call last): (pid=4398) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4398) self.run() (pid=4398) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4398) raise e (pid=4398) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4398) self._entrypoint() (pid=4398) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4398) return self._trainable_func(config, self._status_reporter) (pid=4398) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4398) output = train_func(config) (pid=4398) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4398) TypeError: train() missing 1 required positional argument: 'net' (pid=4398) (pid=4408) 2020-03-20 17:07:35,286 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4408) Traceback (most recent call last): (pid=4408) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4408) self._entrypoint() (pid=4408) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4408) return self._trainable_func(config, self._status_reporter) (pid=4408) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4408) output = train_func(config) (pid=4408) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4408) TypeError: train() missing 1 required positional argument: 'net' (pid=4408) Exception in thread Thread-1: (pid=4408) Traceback (most recent call last): (pid=4408) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4408) self.run() (pid=4408) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4408) raise e (pid=4408) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4408) self._entrypoint() (pid=4408) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4408) return self._trainable_func(config, self._status_reporter) (pid=4408) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4408) output = train_func(config) (pid=4408) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4408) TypeError: train() missing 1 required positional argument: 'net' (pid=4408) (pid=4422) 2020-03-20 17:07:38,183 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4422) Traceback (most recent call last): (pid=4422) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4422) self._entrypoint() (pid=4422) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4422) return self._trainable_func(config, self._status_reporter) (pid=4422) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4422) output = train_func(config) (pid=4422) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4422) TypeError: train() missing 1 required positional argument: 'net' (pid=4422) Exception in thread Thread-1: (pid=4422) Traceback (most recent call last): (pid=4422) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4422) self.run() (pid=4422) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4422) raise e (pid=4422) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4422) self._entrypoint() (pid=4422) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4422) return self._trainable_func(config, self._status_reporter) (pid=4422) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4422) output = train_func(config) (pid=4422) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4422) TypeError: train() missing 1 required positional argument: 'net' (pid=4422) (pid=4423) 2020-03-20 17:07:39,026 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4423) Traceback (most recent call last): (pid=4423) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4423) self._entrypoint() (pid=4423) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4423) return self._trainable_func(config, self._status_reporter) (pid=4423) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4423) output = train_func(config) (pid=4423) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4423) TypeError: train() missing 1 required positional argument: 'net' (pid=4423) Exception in thread Thread-1: (pid=4423) Traceback (most recent call last): (pid=4423) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4423) self.run() (pid=4423) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4423) raise e (pid=4423) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4423) self._entrypoint() (pid=4423) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4423) return self._trainable_func(config, self._status_reporter) (pid=4423) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4423) output = train_func(config) (pid=4423) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4423) TypeError: train() missing 1 required positional argument: 'net' (pid=4423) (pid=4433) 2020-03-20 17:07:41,194 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4433) Traceback (most recent call last): (pid=4433) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4433) self._entrypoint() (pid=4433) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4433) return self._trainable_func(config, self._status_reporter) (pid=4433) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4433) output = train_func(config) (pid=4433) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4433) TypeError: train() missing 1 required positional argument: 'net' (pid=4433) Exception in thread Thread-1: (pid=4433) Traceback (most recent call last): (pid=4433) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4433) self.run() (pid=4433) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4433) raise e (pid=4433) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4433) self._entrypoint() (pid=4433) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4433) return self._trainable_func(config, self._status_reporter) (pid=4433) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4433) output = train_func(config) (pid=4433) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4433) TypeError: train() missing 1 required positional argument: 'net' (pid=4433) (pid=4458) 2020-03-20 17:07:42,471 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4458) Traceback (most recent call last): (pid=4458) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4458) self._entrypoint() (pid=4458) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4458) return self._trainable_func(config, self._status_reporter) (pid=4458) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4458) output = train_func(config) (pid=4458) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4458) TypeError: train() missing 1 required positional argument: 'net' (pid=4458) Exception in thread Thread-1: (pid=4458) Traceback (most recent call last): (pid=4458) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4458) self.run() (pid=4458) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4458) raise e (pid=4458) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4458) self._entrypoint() (pid=4458) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4458) return self._trainable_func(config, self._status_reporter) (pid=4458) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4458) output = train_func(config) (pid=4458) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4458) TypeError: train() missing 1 required positional argument: 'net' (pid=4458) (pid=4470) 2020-03-20 17:07:45,374 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4470) Traceback (most recent call last): (pid=4470) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4470) self._entrypoint() (pid=4470) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4470) return self._trainable_func(config, self._status_reporter) (pid=4470) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4470) output = train_func(config) (pid=4470) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4470) TypeError: train() missing 1 required positional argument: 'net' (pid=4470) Exception in thread Thread-1: (pid=4470) Traceback (most recent call last): (pid=4470) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4470) self.run() (pid=4470) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4470) raise e (pid=4470) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4470) self._entrypoint() (pid=4470) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4470) return self._trainable_func(config, self._status_reporter) (pid=4470) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4470) output = train_func(config) (pid=4470) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4470) TypeError: train() missing 1 required positional argument: 'net' (pid=4470) (pid=4475) 2020-03-20 17:07:45,902 ERROR function_runner.py:99 -- Runner Thread raised error. (pid=4475) Traceback (most recent call last): (pid=4475) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4475) self._entrypoint() (pid=4475) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4475) return self._trainable_func(config, self._status_reporter) (pid=4475) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4475) output = train_func(config) (pid=4475) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4475) TypeError: train() missing 1 required positional argument: 'net' (pid=4475) Exception in thread Thread-1: (pid=4475) Traceback (most recent call last): (pid=4475) File "/opt/python/3.7.1/lib/python3.7/threading.py", line 917, in _bootstrap_inner (pid=4475) self.run() (pid=4475) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 112, in run (pid=4475) raise e (pid=4475) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 93, in run (pid=4475) self._entrypoint() (pid=4475) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 141, in entrypoint (pid=4475) return self._trainable_func(config, self._status_reporter) (pid=4475) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/function_runner.py", line 267, in _trainable_func (pid=4475) output = train_func(config) (pid=4475) File "<ipython-input-5-ff8e06f677c6>", line 4, in train_evaluate (pid=4475) TypeError: train() missing 1 required positional argument: 'net' (pid=4475)
--------------------------------------------------------------------------- TuneError Traceback (most recent call last) <ipython-input-6-5f9e4648a7f2> in <module> 3 num_samples=30, 4 search_alg=AxSearch(ax), # Note that the argument here is the `AxClient`. ----> 5 verbose=0, # Set this level to 1 to see status updates and to 2 to also see trial results. 6 # To use GPU, specify: resources_per_trial={"gpu": 1}. 7 ) ~/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py in run(run_or_experiment, name, stop, config, resources_per_trial, num_samples, local_dir, upload_dir, trial_name_creator, loggers, sync_to_cloud, sync_to_driver, checkpoint_freq, checkpoint_at_end, sync_on_checkpoint, keep_checkpoints_num, checkpoint_score_attr, global_checkpoint_period, export_formats, max_failures, restore, search_alg, scheduler, with_server, server_port, verbose, progress_reporter, resume, queue_trials, reuse_actors, trial_executor, raise_on_failed_trial, return_trials, ray_auto_init, sync_function) 343 if errored_trials: 344 if raise_on_failed_trial: --> 345 raise TuneError("Trials did not complete", errored_trials) 346 else: 347 logger.error("Trials did not complete: %s", errored_trials) TuneError: ('Trials did not complete', [train_evaluate_2e8c23b2, train_evaluate_2e8c23b3, train_evaluate_2e8c23b4, train_evaluate_2e8c23b5, train_evaluate_2e8c23b6, train_evaluate_2e8c23b7, train_evaluate_2e8c23b8, train_evaluate_2f2cbf52, train_evaluate_2f2cbf53, train_evaluate_2f2cbf54, train_evaluate_2f2cbf55, train_evaluate_329f12b6, train_evaluate_329f12b7, train_evaluate_350f1c58, train_evaluate_350f1c59, train_evaluate_36ee1d80, train_evaluate_36ee1d81, train_evaluate_397384f0, train_evaluate_397384f1, train_evaluate_3b49520a, train_evaluate_3b49520b, train_evaluate_3de456ae, train_evaluate_3de456af, train_evaluate_3fcf1b3e, train_evaluate_3fcf1b3f, train_evaluate_4250f5c6, train_evaluate_4250f5c7, train_evaluate_443be8f0, train_evaluate_443be8f1, train_evaluate_46e3346e])
best_parameters, values = ax.get_best_parameters()
best_parameters
{'lr': 0.0035144522261298635, 'momentum': 7.346423042648887e-16}
means, covariances = values
means
{'mean_accuracy': 0.969666685940424}
render(
plot_contour(
model=ax.generation_strategy.model, param_x='lr', param_y='momentum', metric_name='mean_accuracy'
)
)
# `plot_single_method` expects a 2-d array of means, because it expects to average means from multiple
# optimization runs, so we wrap out best objectives array in another array.
best_objectives = np.array([[trial.objective_mean * 100 for trial in ax.experiment.trials.values()]])
best_objective_plot = optimization_trace_single_method(
y=np.maximum.accumulate(best_objectives, axis=1),
title="Model performance vs. # of iterations",
ylabel="Accuracy",
)
render(best_objective_plot)