Ax integrates easily with different scheduling frameworks and distributed training frameworks. In this example, Ax-driven optimization is executed in a distributed fashion using RayTune.
RayTune is a scalable framework for hyperparameter tuning that provides many state-of-the-art hyperparameter tuning algorithms and seamlessly scales from laptop to distributed cluster with fault tolerance. RayTune leverages Ray's Actor API to provide asynchronous parallel and distributed execution.
Ray 'Actors' are a simple and clean abstraction for replicating your Python classes across multiple workers and nodes. Each hyperparameter evaluation is asynchronously executed on a separate Ray actor and reports intermediate training progress back to RayTune. Upon reporting, RayTune then uses this information to performs actions such as early termination, re-prioritization, or checkpointing.
import logging
from ray import tune
from ray.tune import track
from ray.tune.suggest.ax import AxSearch
logger = logging.getLogger(tune.__name__)
logger.setLevel(
level=logging.CRITICAL
) # Reduce the number of Ray warnings that are not relevant here.
import numpy as np
import torch
from ax.plot.contour import plot_contour
from ax.plot.trace import optimization_trace_single_method
from ax.service.ax_client import AxClient
from ax.utils.notebook.plotting import init_notebook_plotting, render
from ax.utils.tutorials.cnn_utils import CNN, evaluate, load_mnist, train
init_notebook_plotting()
[INFO 11-04 18:40:44] ax.utils.notebook.plotting: Injecting Plotly library into cell. Do not overwrite or delete cell.
We specify enforce_sequential_optimization as False, because Ray runs many trials in parallel. With the sequential optimization enforcement, AxClient would expect the first few trials to be completed with data before generating more trials.
When high parallelism is not required, it is best to enforce sequential optimization, as it allows for achieving optimal results in fewer (but sequential) trials. In cases where parallelism is important, such as with distributed training using Ray, we choose to forego minimizing resource utilization and run more trials in parallel.
ax = AxClient(enforce_sequential_optimization=False)
[INFO 11-04 18:40:44] ax.service.ax_client: Starting optimization with verbose logging. To disable logging, set the `verbose_logging` argument to `False`. Note that float values in the logs are rounded to 2 decimal points.
Here we set up the search space and specify the objective; refer to the Ax API tutorials for more detail.
ax.create_experiment(
name="mnist_experiment",
parameters=[
{"name": "lr", "type": "range", "bounds": [1e-6, 0.4], "log_scale": True},
{"name": "momentum", "type": "range", "bounds": [0.0, 1.0]},
],
objective_name="mean_accuracy",
)
[INFO 11-04 18:40:44] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter lr. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 11-04 18:40:44] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter momentum. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 11-04 18:40:44] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+GPEI', steps=[Sobol for 5 trials, GPEI for subsequent trials]). Iterations after 5 will take longer to generate due to model-fitting.
Since we use the Ax Service API here, we evaluate the parameterizations that Ax suggests, using RayTune. The evaluation function follows its usual pattern, taking in a parameterization and outputting an objective value. For detail on evaluation functions, see Trial Evaluation.
def train_evaluate(parameterization):
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
train_loader, valid_loader, test_loader = load_mnist(data_path="~/.data")
net = train(
net=CNN(),
train_loader=train_loader,
parameters=parameterization,
dtype=torch.float,
device=device,
)
track.log(
mean_accuracy=evaluate(
net=net,
data_loader=valid_loader,
dtype=torch.float,
device=device,
)
)
Execute the Ax optimization and trial evaluation in RayTune using AxSearch algorithm:
tune.run(
train_evaluate,
num_samples=30,
search_alg=AxSearch(ax_client=ax, max_concurrent=3),
verbose=0, # Set this level to 1 to see status updates and to 2 to also see trial results.
# To use GPU, specify: resources_per_trial={"gpu": 1}.
)
2020-11-04 18:40:44,887 INFO services.py:1166 -- View the Ray dashboard at http://127.0.0.1:8265 [INFO 11-04 18:40:47] ax.service.ax_client: Generated new trial 0 with parameters {'lr': 0.0, 'momentum': 0.86}. [INFO 11-04 18:40:47] ax.service.ax_client: Generated new trial 1 with parameters {'lr': 0.0, 'momentum': 0.25}. [INFO 11-04 18:40:47] ax.service.ax_client: Generated new trial 2 with parameters {'lr': 0.0, 'momentum': 0.91}. 2020-11-04 18:40:47,431 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID df5a1a8201000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' 2020-11-04 18:40:47,477 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 7bbd902801000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4270) 2020-11-04 18:40:47,450 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4270) Traceback (most recent call last): (pid=4270) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4270) actor_class = pickle.loads(pickled_class) (pid=4270) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4270) from ray.tune.tune import run_experiments, run (pid=4270) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4270) from ray.tune.analysis import ExperimentAnalysis (pid=4270) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4270) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4270) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4270) from ray.tune.trial import Trial (pid=4270) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4270) from ray.tune.durable_trainable import DurableTrainable (pid=4270) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4270) from ray.tune.trainable import Trainable, TrainableUtil (pid=4270) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4270) from ray.tune.logger import UnifiedLogger (pid=4270) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4270) from ray.tune.syncer import get_node_syncer (pid=4270) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4270) @dataclass (pid=4270) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4270) return wrap(_cls) (pid=4270) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4270) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4270) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4270) for name, type in cls_annotations.items()] (pid=4270) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4270) for name, type in cls_annotations.items()] (pid=4270) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4270) if (_is_classvar(a_type, typing) (pid=4270) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4270) return type(a_type) is typing._ClassVar (pid=4270) AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4269) 2020-11-04 18:40:47,425 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4269) Traceback (most recent call last): (pid=4269) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4269) actor_class = pickle.loads(pickled_class) (pid=4269) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4269) from ray.tune.tune import run_experiments, run (pid=4269) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4269) from ray.tune.analysis import ExperimentAnalysis (pid=4269) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4269) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4269) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4269) from ray.tune.trial import Trial (pid=4269) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4269) from ray.tune.durable_trainable import DurableTrainable (pid=4269) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4269) from ray.tune.trainable import Trainable, TrainableUtil (pid=4269) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4269) from ray.tune.logger import UnifiedLogger (pid=4269) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4269) from ray.tune.syncer import get_node_syncer (pid=4269) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4269) @dataclass (pid=4269) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4269) return wrap(_cls) (pid=4269) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4269) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4269) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4269) for name, type in cls_annotations.items()] (pid=4269) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4269) for name, type in cls_annotations.items()] (pid=4269) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4269) if (_is_classvar(a_type, typing) (pid=4269) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4269) return type(a_type) is typing._ClassVar (pid=4269) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 11-04 18:40:47] ax.service.ax_client: Generated new trial 3 with parameters {'lr': 0.0, 'momentum': 0.24}. [INFO 11-04 18:40:47] ax.service.ax_client: Generated new trial 4 with parameters {'lr': 0.0, 'momentum': 0.99}. 2020-11-04 18:40:48,572 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 3bf0c85601000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4355) 2020-11-04 18:40:48,551 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4355) Traceback (most recent call last): (pid=4355) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4355) actor_class = pickle.loads(pickled_class) (pid=4355) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4355) from ray.tune.tune import run_experiments, run (pid=4355) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4355) from ray.tune.analysis import ExperimentAnalysis (pid=4355) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4355) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4355) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4355) from ray.tune.trial import Trial (pid=4355) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4355) from ray.tune.durable_trainable import DurableTrainable (pid=4355) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4355) from ray.tune.trainable import Trainable, TrainableUtil (pid=4355) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4355) from ray.tune.logger import UnifiedLogger (pid=4355) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4355) from ray.tune.syncer import get_node_syncer (pid=4355) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4355) @dataclass (pid=4355) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4355) return wrap(_cls) (pid=4355) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4355) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4355) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4355) for name, type in cls_annotations.items()] (pid=4355) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4355) for name, type in cls_annotations.items()] (pid=4355) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4355) if (_is_classvar(a_type, typing) (pid=4355) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4355) return type(a_type) is typing._ClassVar (pid=4355) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 11-04 18:40:48] ax.service.ax_client: Generated new trial 5 with parameters {'lr': 0.26, 'momentum': 0.17}. 2020-11-04 18:40:48,735 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID ae935fc001000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4354) 2020-11-04 18:40:48,721 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4354) Traceback (most recent call last): (pid=4354) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4354) actor_class = pickle.loads(pickled_class) (pid=4354) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4354) from ray.tune.tune import run_experiments, run (pid=4354) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4354) from ray.tune.analysis import ExperimentAnalysis (pid=4354) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4354) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4354) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4354) from ray.tune.trial import Trial (pid=4354) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4354) from ray.tune.durable_trainable import DurableTrainable (pid=4354) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4354) from ray.tune.trainable import Trainable, TrainableUtil (pid=4354) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4354) from ray.tune.logger import UnifiedLogger (pid=4354) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4354) from ray.tune.syncer import get_node_syncer (pid=4354) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4354) @dataclass (pid=4354) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4354) return wrap(_cls) (pid=4354) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4354) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4354) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4354) for name, type in cls_annotations.items()] (pid=4354) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4354) for name, type in cls_annotations.items()] (pid=4354) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4354) if (_is_classvar(a_type, typing) (pid=4354) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4354) return type(a_type) is typing._ClassVar (pid=4354) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 11-04 18:40:48] ax.service.ax_client: Generated new trial 6 with parameters {'lr': 0.0, 'momentum': 0.15}. (pid=4365) 2020-11-04 18:40:49,302 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4365) Traceback (most recent call last): (pid=4365) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4365) actor_class = pickle.loads(pickled_class) (pid=4365) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4365) from ray.tune.tune import run_experiments, run (pid=4365) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4365) from ray.tune.analysis import ExperimentAnalysis (pid=4365) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4365) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4365) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4365) from ray.tune.trial import Trial (pid=4365) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4365) from ray.tune.durable_trainable import DurableTrainable (pid=4365) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4365) from ray.tune.trainable import Trainable, TrainableUtil (pid=4365) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4365) from ray.tune.logger import UnifiedLogger (pid=4365) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4365) from ray.tune.syncer import get_node_syncer (pid=4365) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4365) @dataclass (pid=4365) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4365) return wrap(_cls) (pid=4365) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4365) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4365) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4365) for name, type in cls_annotations.items()] (pid=4365) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4365) for name, type in cls_annotations.items()] (pid=4365) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4365) if (_is_classvar(a_type, typing) (pid=4365) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4365) return type(a_type) is typing._ClassVar (pid=4365) AttributeError: module 'typing' has no attribute '_ClassVar' 2020-11-04 18:40:49,313 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID b7603b6c01000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 11-04 18:40:49] ax.service.ax_client: Generated new trial 7 with parameters {'lr': 0.16, 'momentum': 0.7}. 2020-11-04 18:40:49,916 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID be3cb80901000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4393) 2020-11-04 18:40:49,911 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4393) Traceback (most recent call last): (pid=4393) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4393) actor_class = pickle.loads(pickled_class) (pid=4393) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4393) from ray.tune.tune import run_experiments, run (pid=4393) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4393) from ray.tune.analysis import ExperimentAnalysis (pid=4393) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4393) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4393) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4393) from ray.tune.trial import Trial (pid=4393) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4393) from ray.tune.durable_trainable import DurableTrainable (pid=4393) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4393) from ray.tune.trainable import Trainable, TrainableUtil (pid=4393) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4393) from ray.tune.logger import UnifiedLogger (pid=4393) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4393) from ray.tune.syncer import get_node_syncer (pid=4393) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4393) @dataclass (pid=4393) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4393) return wrap(_cls) (pid=4393) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4393) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4393) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4393) for name, type in cls_annotations.items()] (pid=4393) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4393) for name, type in cls_annotations.items()] (pid=4393) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4393) if (_is_classvar(a_type, typing) (pid=4393) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4393) return type(a_type) is typing._ClassVar (pid=4393) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 11-04 18:40:50] ax.service.ax_client: Generated new trial 8 with parameters {'lr': 0.26, 'momentum': 0.67}. 2020-11-04 18:40:50,178 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID af23404c01000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4408) 2020-11-04 18:40:50,165 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4408) Traceback (most recent call last): (pid=4408) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4408) actor_class = pickle.loads(pickled_class) (pid=4408) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4408) from ray.tune.tune import run_experiments, run (pid=4408) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4408) from ray.tune.analysis import ExperimentAnalysis (pid=4408) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4408) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4408) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4408) from ray.tune.trial import Trial (pid=4408) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4408) from ray.tune.durable_trainable import DurableTrainable (pid=4408) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4408) from ray.tune.trainable import Trainable, TrainableUtil (pid=4408) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4408) from ray.tune.logger import UnifiedLogger (pid=4408) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4408) from ray.tune.syncer import get_node_syncer (pid=4408) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4408) @dataclass (pid=4408) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4408) return wrap(_cls) (pid=4408) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4408) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4408) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4408) for name, type in cls_annotations.items()] (pid=4408) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4408) for name, type in cls_annotations.items()] (pid=4408) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4408) if (_is_classvar(a_type, typing) (pid=4408) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4408) return type(a_type) is typing._ClassVar (pid=4408) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 11-04 18:40:50] ax.service.ax_client: Generated new trial 9 with parameters {'lr': 0.0, 'momentum': 0.66}. 2020-11-04 18:40:50,820 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 7d26912601000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4427) 2020-11-04 18:40:50,808 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4427) Traceback (most recent call last): (pid=4427) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4427) actor_class = pickle.loads(pickled_class) (pid=4427) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4427) from ray.tune.tune import run_experiments, run (pid=4427) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4427) from ray.tune.analysis import ExperimentAnalysis (pid=4427) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4427) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4427) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4427) from ray.tune.trial import Trial (pid=4427) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4427) from ray.tune.durable_trainable import DurableTrainable (pid=4427) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4427) from ray.tune.trainable import Trainable, TrainableUtil (pid=4427) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4427) from ray.tune.logger import UnifiedLogger (pid=4427) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4427) from ray.tune.syncer import get_node_syncer (pid=4427) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4427) @dataclass (pid=4427) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4427) return wrap(_cls) (pid=4427) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4427) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4427) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4427) for name, type in cls_annotations.items()] (pid=4427) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4427) for name, type in cls_annotations.items()] (pid=4427) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4427) if (_is_classvar(a_type, typing) (pid=4427) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4427) return type(a_type) is typing._ClassVar (pid=4427) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 11-04 18:40:51] ax.service.ax_client: Generated new trial 10 with parameters {'lr': 0.0, 'momentum': 0.28}. 2020-11-04 18:40:51,327 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 335d6f0801000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4448) 2020-11-04 18:40:51,315 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4448) Traceback (most recent call last): (pid=4448) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4448) actor_class = pickle.loads(pickled_class) (pid=4448) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4448) from ray.tune.tune import run_experiments, run (pid=4448) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4448) from ray.tune.analysis import ExperimentAnalysis (pid=4448) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4448) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4448) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4448) from ray.tune.trial import Trial (pid=4448) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4448) from ray.tune.durable_trainable import DurableTrainable (pid=4448) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4448) from ray.tune.trainable import Trainable, TrainableUtil (pid=4448) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4448) from ray.tune.logger import UnifiedLogger (pid=4448) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4448) from ray.tune.syncer import get_node_syncer (pid=4448) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4448) @dataclass (pid=4448) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4448) return wrap(_cls) (pid=4448) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4448) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4448) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4448) for name, type in cls_annotations.items()] (pid=4448) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4448) for name, type in cls_annotations.items()] (pid=4448) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4448) if (_is_classvar(a_type, typing) (pid=4448) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4448) return type(a_type) is typing._ClassVar (pid=4448) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 11-04 18:40:51] ax.service.ax_client: Generated new trial 11 with parameters {'lr': 0.0, 'momentum': 0.83}. 2020-11-04 18:40:51,701 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID ad5fa5cb01000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4465) 2020-11-04 18:40:51,695 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4465) Traceback (most recent call last): (pid=4465) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4465) actor_class = pickle.loads(pickled_class) (pid=4465) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4465) from ray.tune.tune import run_experiments, run (pid=4465) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4465) from ray.tune.analysis import ExperimentAnalysis (pid=4465) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4465) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4465) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4465) from ray.tune.trial import Trial (pid=4465) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4465) from ray.tune.durable_trainable import DurableTrainable (pid=4465) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4465) from ray.tune.trainable import Trainable, TrainableUtil (pid=4465) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4465) from ray.tune.logger import UnifiedLogger (pid=4465) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4465) from ray.tune.syncer import get_node_syncer (pid=4465) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4465) @dataclass (pid=4465) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4465) return wrap(_cls) (pid=4465) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4465) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4465) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4465) for name, type in cls_annotations.items()] (pid=4465) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4465) for name, type in cls_annotations.items()] (pid=4465) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4465) if (_is_classvar(a_type, typing) (pid=4465) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4465) return type(a_type) is typing._ClassVar (pid=4465) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 11-04 18:40:51] ax.service.ax_client: Generated new trial 12 with parameters {'lr': 0.0, 'momentum': 0.93}. 2020-11-04 18:40:52,452 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 35d7a3b901000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4484) 2020-11-04 18:40:52,436 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4484) Traceback (most recent call last): (pid=4484) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4484) actor_class = pickle.loads(pickled_class) (pid=4484) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4484) from ray.tune.tune import run_experiments, run (pid=4484) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4484) from ray.tune.analysis import ExperimentAnalysis (pid=4484) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4484) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4484) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4484) from ray.tune.trial import Trial (pid=4484) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4484) from ray.tune.durable_trainable import DurableTrainable (pid=4484) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4484) from ray.tune.trainable import Trainable, TrainableUtil (pid=4484) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4484) from ray.tune.logger import UnifiedLogger (pid=4484) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4484) from ray.tune.syncer import get_node_syncer (pid=4484) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4484) @dataclass (pid=4484) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4484) return wrap(_cls) (pid=4484) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4484) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4484) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4484) for name, type in cls_annotations.items()] (pid=4484) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4484) for name, type in cls_annotations.items()] (pid=4484) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4484) if (_is_classvar(a_type, typing) (pid=4484) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4484) return type(a_type) is typing._ClassVar (pid=4484) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 11-04 18:40:52] ax.service.ax_client: Generated new trial 13 with parameters {'lr': 0.04, 'momentum': 0.26}. 2020-11-04 18:40:52,830 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 91fdfca801000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4502) 2020-11-04 18:40:52,822 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4502) Traceback (most recent call last): (pid=4502) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4502) actor_class = pickle.loads(pickled_class) (pid=4502) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4502) from ray.tune.tune import run_experiments, run (pid=4502) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4502) from ray.tune.analysis import ExperimentAnalysis (pid=4502) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4502) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4502) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4502) from ray.tune.trial import Trial (pid=4502) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4502) from ray.tune.durable_trainable import DurableTrainable (pid=4502) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4502) from ray.tune.trainable import Trainable, TrainableUtil (pid=4502) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4502) from ray.tune.logger import UnifiedLogger (pid=4502) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4502) from ray.tune.syncer import get_node_syncer (pid=4502) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4502) @dataclass (pid=4502) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4502) return wrap(_cls) (pid=4502) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4502) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4502) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4502) for name, type in cls_annotations.items()] (pid=4502) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4502) for name, type in cls_annotations.items()] (pid=4502) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4502) if (_is_classvar(a_type, typing) (pid=4502) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4502) return type(a_type) is typing._ClassVar (pid=4502) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 11-04 18:40:53] ax.service.ax_client: Generated new trial 14 with parameters {'lr': 0.01, 'momentum': 0.8}. 2020-11-04 18:40:53,078 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID b9c4a7c601000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4519) 2020-11-04 18:40:53,067 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4519) Traceback (most recent call last): (pid=4519) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4519) actor_class = pickle.loads(pickled_class) (pid=4519) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4519) from ray.tune.tune import run_experiments, run (pid=4519) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4519) from ray.tune.analysis import ExperimentAnalysis (pid=4519) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4519) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4519) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4519) from ray.tune.trial import Trial (pid=4519) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4519) from ray.tune.durable_trainable import DurableTrainable (pid=4519) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4519) from ray.tune.trainable import Trainable, TrainableUtil (pid=4519) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4519) from ray.tune.logger import UnifiedLogger (pid=4519) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4519) from ray.tune.syncer import get_node_syncer (pid=4519) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4519) @dataclass (pid=4519) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4519) return wrap(_cls) (pid=4519) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4519) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4519) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4519) for name, type in cls_annotations.items()] (pid=4519) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4519) for name, type in cls_annotations.items()] (pid=4519) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4519) if (_is_classvar(a_type, typing) (pid=4519) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4519) return type(a_type) is typing._ClassVar (pid=4519) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 11-04 18:40:53] ax.service.ax_client: Generated new trial 15 with parameters {'lr': 0.14, 'momentum': 0.1}. 2020-11-04 18:40:54,146 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 090f0d8b01000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4553) 2020-11-04 18:40:54,140 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4553) Traceback (most recent call last): (pid=4553) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4553) actor_class = pickle.loads(pickled_class) (pid=4553) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4553) from ray.tune.tune import run_experiments, run (pid=4553) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4553) from ray.tune.analysis import ExperimentAnalysis (pid=4553) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4553) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4553) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4553) from ray.tune.trial import Trial (pid=4553) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4553) from ray.tune.durable_trainable import DurableTrainable (pid=4553) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4553) from ray.tune.trainable import Trainable, TrainableUtil (pid=4553) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4553) from ray.tune.logger import UnifiedLogger (pid=4553) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4553) from ray.tune.syncer import get_node_syncer (pid=4553) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4553) @dataclass (pid=4553) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4553) return wrap(_cls) (pid=4553) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4553) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4553) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4553) for name, type in cls_annotations.items()] (pid=4553) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4553) for name, type in cls_annotations.items()] (pid=4553) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4553) if (_is_classvar(a_type, typing) (pid=4553) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4553) return type(a_type) is typing._ClassVar (pid=4553) AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4554) 2020-11-04 18:40:54,232 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4554) Traceback (most recent call last): (pid=4554) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4554) actor_class = pickle.loads(pickled_class) (pid=4554) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4554) from ray.tune.tune import run_experiments, run (pid=4554) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4554) from ray.tune.analysis import ExperimentAnalysis (pid=4554) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4554) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4554) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4554) from ray.tune.trial import Trial (pid=4554) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4554) from ray.tune.durable_trainable import DurableTrainable (pid=4554) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4554) from ray.tune.trainable import Trainable, TrainableUtil (pid=4554) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4554) from ray.tune.logger import UnifiedLogger (pid=4554) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4554) from ray.tune.syncer import get_node_syncer (pid=4554) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4554) @dataclass (pid=4554) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4554) return wrap(_cls) 2020-11-04 18:40:54,244 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID c65d7ae001000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4554) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4554) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4554) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4554) for name, type in cls_annotations.items()] (pid=4554) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4554) for name, type in cls_annotations.items()] (pid=4554) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4554) if (_is_classvar(a_type, typing) (pid=4554) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4554) return type(a_type) is typing._ClassVar (pid=4554) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 11-04 18:40:54] ax.service.ax_client: Generated new trial 16 with parameters {'lr': 0.13, 'momentum': 0.26}. [INFO 11-04 18:40:54] ax.service.ax_client: Generated new trial 17 with parameters {'lr': 0.01, 'momentum': 0.91}. 2020-11-04 18:40:54,732 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 3d80c2fe01000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4569) 2020-11-04 18:40:54,727 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4569) Traceback (most recent call last): (pid=4569) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4569) actor_class = pickle.loads(pickled_class) (pid=4569) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4569) from ray.tune.tune import run_experiments, run (pid=4569) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4569) from ray.tune.analysis import ExperimentAnalysis (pid=4569) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4569) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4569) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4569) from ray.tune.trial import Trial (pid=4569) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4569) from ray.tune.durable_trainable import DurableTrainable (pid=4569) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4569) from ray.tune.trainable import Trainable, TrainableUtil (pid=4569) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4569) from ray.tune.logger import UnifiedLogger (pid=4569) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4569) from ray.tune.syncer import get_node_syncer (pid=4569) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4569) @dataclass (pid=4569) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4569) return wrap(_cls) (pid=4569) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4569) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4569) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4569) for name, type in cls_annotations.items()] (pid=4569) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4569) for name, type in cls_annotations.items()] (pid=4569) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4569) if (_is_classvar(a_type, typing) (pid=4569) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4569) return type(a_type) is typing._ClassVar (pid=4569) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 11-04 18:40:55] ax.service.ax_client: Generated new trial 18 with parameters {'lr': 0.0, 'momentum': 0.88}. 2020-11-04 18:40:55,755 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 4305340a01000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4607) 2020-11-04 18:40:55,746 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4607) Traceback (most recent call last): (pid=4607) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4607) actor_class = pickle.loads(pickled_class) (pid=4607) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4607) from ray.tune.tune import run_experiments, run (pid=4607) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4607) from ray.tune.analysis import ExperimentAnalysis (pid=4607) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4607) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4607) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4607) from ray.tune.trial import Trial (pid=4607) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4607) from ray.tune.durable_trainable import DurableTrainable (pid=4607) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4607) from ray.tune.trainable import Trainable, TrainableUtil (pid=4607) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4607) from ray.tune.logger import UnifiedLogger (pid=4607) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4607) from ray.tune.syncer import get_node_syncer (pid=4607) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4607) @dataclass (pid=4607) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4607) return wrap(_cls) (pid=4607) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4607) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4607) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4607) for name, type in cls_annotations.items()] (pid=4607) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4607) for name, type in cls_annotations.items()] (pid=4607) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4607) if (_is_classvar(a_type, typing) (pid=4607) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4607) return type(a_type) is typing._ClassVar (pid=4607) AttributeError: module 'typing' has no attribute '_ClassVar' 2020-11-04 18:40:56,017 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID a7357af301000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 11-04 18:40:56] ax.service.ax_client: Generated new trial 19 with parameters {'lr': 0.0, 'momentum': 0.29}. (pid=4610) 2020-11-04 18:40:56,002 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4610) Traceback (most recent call last): (pid=4610) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4610) actor_class = pickle.loads(pickled_class) (pid=4610) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4610) from ray.tune.tune import run_experiments, run (pid=4610) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4610) from ray.tune.analysis import ExperimentAnalysis (pid=4610) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4610) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4610) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4610) from ray.tune.trial import Trial (pid=4610) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4610) from ray.tune.durable_trainable import DurableTrainable (pid=4610) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4610) from ray.tune.trainable import Trainable, TrainableUtil (pid=4610) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4610) from ray.tune.logger import UnifiedLogger (pid=4610) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4610) from ray.tune.syncer import get_node_syncer (pid=4610) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4610) @dataclass (pid=4610) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4610) return wrap(_cls) (pid=4610) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4610) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4610) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4610) for name, type in cls_annotations.items()] (pid=4610) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4610) for name, type in cls_annotations.items()] (pid=4610) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4610) if (_is_classvar(a_type, typing) (pid=4610) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4610) return type(a_type) is typing._ClassVar (pid=4610) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 11-04 18:40:56] ax.service.ax_client: Generated new trial 20 with parameters {'lr': 0.0, 'momentum': 0.4}. 2020-11-04 18:40:56,359 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 31965ca801000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4621) 2020-11-04 18:40:56,351 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4621) Traceback (most recent call last): (pid=4621) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4621) actor_class = pickle.loads(pickled_class) (pid=4621) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4621) from ray.tune.tune import run_experiments, run (pid=4621) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4621) from ray.tune.analysis import ExperimentAnalysis (pid=4621) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4621) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4621) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4621) from ray.tune.trial import Trial (pid=4621) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4621) from ray.tune.durable_trainable import DurableTrainable (pid=4621) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4621) from ray.tune.trainable import Trainable, TrainableUtil (pid=4621) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4621) from ray.tune.logger import UnifiedLogger (pid=4621) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4621) from ray.tune.syncer import get_node_syncer (pid=4621) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4621) @dataclass (pid=4621) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4621) return wrap(_cls) (pid=4621) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4621) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4621) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4621) for name, type in cls_annotations.items()] (pid=4621) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4621) for name, type in cls_annotations.items()] (pid=4621) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4621) if (_is_classvar(a_type, typing) (pid=4621) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4621) return type(a_type) is typing._ClassVar (pid=4621) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 11-04 18:40:56] ax.service.ax_client: Generated new trial 21 with parameters {'lr': 0.0, 'momentum': 0.94}. 2020-11-04 18:40:57,512 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 8fcb3d8501000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4664) 2020-11-04 18:40:57,507 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4664) Traceback (most recent call last): (pid=4664) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4664) actor_class = pickle.loads(pickled_class) (pid=4664) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4664) from ray.tune.tune import run_experiments, run (pid=4664) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4664) from ray.tune.analysis import ExperimentAnalysis (pid=4664) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4664) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4664) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4664) from ray.tune.trial import Trial (pid=4664) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4664) from ray.tune.durable_trainable import DurableTrainable (pid=4664) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4664) from ray.tune.trainable import Trainable, TrainableUtil (pid=4664) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4664) from ray.tune.logger import UnifiedLogger (pid=4664) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4664) from ray.tune.syncer import get_node_syncer (pid=4664) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4664) @dataclass (pid=4664) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4664) return wrap(_cls) (pid=4664) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4664) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4664) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4664) for name, type in cls_annotations.items()] (pid=4664) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4664) for name, type in cls_annotations.items()] (pid=4664) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4664) if (_is_classvar(a_type, typing) (pid=4664) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4664) return type(a_type) is typing._ClassVar (pid=4664) AttributeError: module 'typing' has no attribute '_ClassVar' 2020-11-04 18:40:57,565 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID c662ca5a01000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4667) 2020-11-04 18:40:57,536 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4667) Traceback (most recent call last): (pid=4667) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4667) actor_class = pickle.loads(pickled_class) (pid=4667) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4667) from ray.tune.tune import run_experiments, run (pid=4667) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4667) from ray.tune.analysis import ExperimentAnalysis (pid=4667) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4667) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4667) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4667) from ray.tune.trial import Trial (pid=4667) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4667) from ray.tune.durable_trainable import DurableTrainable (pid=4667) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4667) from ray.tune.trainable import Trainable, TrainableUtil (pid=4667) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4667) from ray.tune.logger import UnifiedLogger (pid=4667) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4667) from ray.tune.syncer import get_node_syncer (pid=4667) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4667) @dataclass (pid=4667) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4667) return wrap(_cls) (pid=4667) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4667) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4667) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4667) for name, type in cls_annotations.items()] (pid=4667) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4667) for name, type in cls_annotations.items()] (pid=4667) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4667) if (_is_classvar(a_type, typing) (pid=4667) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4667) return type(a_type) is typing._ClassVar (pid=4667) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 11-04 18:40:57] ax.service.ax_client: Generated new trial 22 with parameters {'lr': 0.0, 'momentum': 0.38}. [INFO 11-04 18:40:57] ax.service.ax_client: Generated new trial 23 with parameters {'lr': 0.0, 'momentum': 0.17}. 2020-11-04 18:40:58,098 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID a5450a3401000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4675) 2020-11-04 18:40:58,090 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4675) Traceback (most recent call last): (pid=4675) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4675) actor_class = pickle.loads(pickled_class) (pid=4675) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4675) from ray.tune.tune import run_experiments, run (pid=4675) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4675) from ray.tune.analysis import ExperimentAnalysis (pid=4675) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4675) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4675) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4675) from ray.tune.trial import Trial (pid=4675) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4675) from ray.tune.durable_trainable import DurableTrainable (pid=4675) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4675) from ray.tune.trainable import Trainable, TrainableUtil (pid=4675) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4675) from ray.tune.logger import UnifiedLogger (pid=4675) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4675) from ray.tune.syncer import get_node_syncer (pid=4675) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4675) @dataclass (pid=4675) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4675) return wrap(_cls) (pid=4675) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4675) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4675) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4675) for name, type in cls_annotations.items()] (pid=4675) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4675) for name, type in cls_annotations.items()] (pid=4675) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4675) if (_is_classvar(a_type, typing) (pid=4675) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4675) return type(a_type) is typing._ClassVar (pid=4675) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 11-04 18:40:58] ax.service.ax_client: Generated new trial 24 with parameters {'lr': 0.0, 'momentum': 0.42}. 2020-11-04 18:40:59,102 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID b299cffa01000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4719) 2020-11-04 18:40:59,085 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4719) Traceback (most recent call last): (pid=4719) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4719) actor_class = pickle.loads(pickled_class) (pid=4719) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4719) from ray.tune.tune import run_experiments, run (pid=4719) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4719) from ray.tune.analysis import ExperimentAnalysis (pid=4719) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4719) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4719) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4719) from ray.tune.trial import Trial (pid=4719) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4719) from ray.tune.durable_trainable import DurableTrainable (pid=4719) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4719) from ray.tune.trainable import Trainable, TrainableUtil (pid=4719) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4719) from ray.tune.logger import UnifiedLogger (pid=4719) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4719) from ray.tune.syncer import get_node_syncer (pid=4719) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4719) @dataclass (pid=4719) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4719) return wrap(_cls) (pid=4719) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4719) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4719) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4719) for name, type in cls_annotations.items()] (pid=4719) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4719) for name, type in cls_annotations.items()] (pid=4719) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4719) if (_is_classvar(a_type, typing) (pid=4719) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4719) return type(a_type) is typing._ClassVar (pid=4719) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 11-04 18:40:59] ax.service.ax_client: Generated new trial 25 with parameters {'lr': 0.0, 'momentum': 0.97}. 2020-11-04 18:40:59,496 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 65710efb01000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4722) 2020-11-04 18:40:59,484 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4722) Traceback (most recent call last): (pid=4722) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4722) actor_class = pickle.loads(pickled_class) (pid=4722) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4722) from ray.tune.tune import run_experiments, run (pid=4722) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4722) from ray.tune.analysis import ExperimentAnalysis (pid=4722) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4722) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4722) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4722) from ray.tune.trial import Trial (pid=4722) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4722) from ray.tune.durable_trainable import DurableTrainable (pid=4722) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4722) from ray.tune.trainable import Trainable, TrainableUtil (pid=4722) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4722) from ray.tune.logger import UnifiedLogger (pid=4722) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4722) from ray.tune.syncer import get_node_syncer (pid=4722) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4722) @dataclass (pid=4722) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4722) return wrap(_cls) (pid=4722) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4722) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4722) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4722) for name, type in cls_annotations.items()] (pid=4722) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4722) for name, type in cls_annotations.items()] (pid=4722) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4722) if (_is_classvar(a_type, typing) (pid=4722) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4722) return type(a_type) is typing._ClassVar (pid=4722) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 11-04 18:40:59] ax.service.ax_client: Generated new trial 26 with parameters {'lr': 0.0, 'momentum': 0.43}. 2020-11-04 18:40:59,819 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 1211da1301000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4731) 2020-11-04 18:40:59,815 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4731) Traceback (most recent call last): (pid=4731) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4731) actor_class = pickle.loads(pickled_class) (pid=4731) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4731) from ray.tune.tune import run_experiments, run (pid=4731) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4731) from ray.tune.analysis import ExperimentAnalysis (pid=4731) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4731) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4731) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4731) from ray.tune.trial import Trial (pid=4731) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4731) from ray.tune.durable_trainable import DurableTrainable (pid=4731) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4731) from ray.tune.trainable import Trainable, TrainableUtil (pid=4731) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4731) from ray.tune.logger import UnifiedLogger (pid=4731) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4731) from ray.tune.syncer import get_node_syncer (pid=4731) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4731) @dataclass (pid=4731) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4731) return wrap(_cls) (pid=4731) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4731) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4731) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4731) for name, type in cls_annotations.items()] (pid=4731) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4731) for name, type in cls_annotations.items()] (pid=4731) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4731) if (_is_classvar(a_type, typing) (pid=4731) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4731) return type(a_type) is typing._ClassVar (pid=4731) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 11-04 18:41:00] ax.service.ax_client: Generated new trial 27 with parameters {'lr': 0.0, 'momentum': 0.34}. 2020-11-04 18:41:00,892 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 03d9348501000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4773) 2020-11-04 18:41:00,887 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4773) Traceback (most recent call last): (pid=4773) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4773) actor_class = pickle.loads(pickled_class) (pid=4773) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4773) from ray.tune.tune import run_experiments, run (pid=4773) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4773) from ray.tune.analysis import ExperimentAnalysis (pid=4773) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4773) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4773) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4773) from ray.tune.trial import Trial (pid=4773) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4773) from ray.tune.durable_trainable import DurableTrainable (pid=4773) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4773) from ray.tune.trainable import Trainable, TrainableUtil (pid=4773) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4773) from ray.tune.logger import UnifiedLogger (pid=4773) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4773) from ray.tune.syncer import get_node_syncer (pid=4773) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4773) @dataclass (pid=4773) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4773) return wrap(_cls) (pid=4773) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4773) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4773) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4773) for name, type in cls_annotations.items()] (pid=4773) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4773) for name, type in cls_annotations.items()] (pid=4773) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4773) if (_is_classvar(a_type, typing) (pid=4773) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4773) return type(a_type) is typing._ClassVar (pid=4773) AttributeError: module 'typing' has no attribute '_ClassVar' 2020-11-04 18:41:01,119 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 814c5aa201000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4774) 2020-11-04 18:41:01,112 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4774) Traceback (most recent call last): (pid=4774) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4774) actor_class = pickle.loads(pickled_class) (pid=4774) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4774) from ray.tune.tune import run_experiments, run (pid=4774) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4774) from ray.tune.analysis import ExperimentAnalysis (pid=4774) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4774) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4774) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4774) from ray.tune.trial import Trial (pid=4774) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4774) from ray.tune.durable_trainable import DurableTrainable (pid=4774) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4774) from ray.tune.trainable import Trainable, TrainableUtil (pid=4774) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4774) from ray.tune.logger import UnifiedLogger (pid=4774) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4774) from ray.tune.syncer import get_node_syncer (pid=4774) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4774) @dataclass (pid=4774) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4774) return wrap(_cls) (pid=4774) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4774) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4774) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4774) for name, type in cls_annotations.items()] (pid=4774) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4774) for name, type in cls_annotations.items()] (pid=4774) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4774) if (_is_classvar(a_type, typing) (pid=4774) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4774) return type(a_type) is typing._ClassVar (pid=4774) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 11-04 18:41:01] ax.service.ax_client: Generated new trial 28 with parameters {'lr': 0.03, 'momentum': 0.72}. [INFO 11-04 18:41:01] ax.service.ax_client: Generated new trial 29 with parameters {'lr': 0.0, 'momentum': 0.17}. 2020-11-04 18:41:01,592 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 90e2cc4001000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4784) 2020-11-04 18:41:01,585 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4784) Traceback (most recent call last): (pid=4784) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4784) actor_class = pickle.loads(pickled_class) (pid=4784) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4784) from ray.tune.tune import run_experiments, run (pid=4784) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4784) from ray.tune.analysis import ExperimentAnalysis (pid=4784) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4784) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4784) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4784) from ray.tune.trial import Trial (pid=4784) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4784) from ray.tune.durable_trainable import DurableTrainable (pid=4784) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4784) from ray.tune.trainable import Trainable, TrainableUtil (pid=4784) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4784) from ray.tune.logger import UnifiedLogger (pid=4784) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4784) from ray.tune.syncer import get_node_syncer (pid=4784) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4784) @dataclass (pid=4784) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4784) return wrap(_cls) (pid=4784) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4784) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4784) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4784) for name, type in cls_annotations.items()] (pid=4784) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4784) for name, type in cls_annotations.items()] (pid=4784) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4784) if (_is_classvar(a_type, typing) (pid=4784) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4784) return type(a_type) is typing._ClassVar (pid=4784) AttributeError: module 'typing' has no attribute '_ClassVar' 2020-11-04 18:41:02,591 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 296a970401000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4827) 2020-11-04 18:41:02,581 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4827) Traceback (most recent call last): (pid=4827) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4827) actor_class = pickle.loads(pickled_class) (pid=4827) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4827) from ray.tune.tune import run_experiments, run (pid=4827) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4827) from ray.tune.analysis import ExperimentAnalysis (pid=4827) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4827) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4827) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4827) from ray.tune.trial import Trial (pid=4827) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4827) from ray.tune.durable_trainable import DurableTrainable (pid=4827) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4827) from ray.tune.trainable import Trainable, TrainableUtil (pid=4827) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4827) from ray.tune.logger import UnifiedLogger (pid=4827) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4827) from ray.tune.syncer import get_node_syncer (pid=4827) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4827) @dataclass (pid=4827) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4827) return wrap(_cls) (pid=4827) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4827) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4827) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4827) for name, type in cls_annotations.items()] (pid=4827) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4827) for name, type in cls_annotations.items()] (pid=4827) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4827) if (_is_classvar(a_type, typing) (pid=4827) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4827) return type(a_type) is typing._ClassVar (pid=4827) AttributeError: module 'typing' has no attribute '_ClassVar' 2020-11-04 18:41:02,646 WARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID f3b32ee701000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4828) 2020-11-04 18:41:02,637 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4828) Traceback (most recent call last): (pid=4828) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4828) actor_class = pickle.loads(pickled_class) (pid=4828) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4828) from ray.tune.tune import run_experiments, run (pid=4828) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 5, in <module> (pid=4828) from ray.tune.analysis import ExperimentAnalysis (pid=4828) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4828) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4828) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4828) from ray.tune.trial import Trial (pid=4828) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 16, in <module> (pid=4828) from ray.tune.durable_trainable import DurableTrainable (pid=4828) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4828) from ray.tune.trainable import Trainable, TrainableUtil (pid=4828) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4828) from ray.tune.logger import UnifiedLogger (pid=4828) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4828) from ray.tune.syncer import get_node_syncer (pid=4828) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4828) @dataclass (pid=4828) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4828) return wrap(_cls) (pid=4828) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4828) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4828) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4828) for name, type in cls_annotations.items()] (pid=4828) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4828) for name, type in cls_annotations.items()] (pid=4828) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4828) if (_is_classvar(a_type, typing) (pid=4828) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4828) return type(a_type) is typing._ClassVar (pid=4828) AttributeError: module 'typing' has no attribute '_ClassVar'
--------------------------------------------------------------------------- TuneError Traceback (most recent call last) <ipython-input-6-9007b87d2ec7> in <module> 3 num_samples=30, 4 search_alg=AxSearch(ax_client=ax, max_concurrent=3), ----> 5 verbose=0, # Set this level to 1 to see status updates and to 2 to also see trial results. 6 # To use GPU, specify: resources_per_trial={"gpu": 1}. 7 ) ~/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py in run(run_or_experiment, name, metric, mode, stop, time_budget_s, config, resources_per_trial, num_samples, local_dir, search_alg, scheduler, keep_checkpoints_num, checkpoint_score_attr, checkpoint_freq, checkpoint_at_end, verbose, progress_reporter, loggers, log_to_file, trial_name_creator, trial_dirname_creator, sync_config, export_formats, max_failures, fail_fast, restore, server_port, resume, queue_trials, reuse_actors, trial_executor, raise_on_failed_trial, ray_auto_init, run_errored_only, global_checkpoint_period, with_server, upload_dir, sync_to_cloud, sync_to_driver, sync_on_checkpoint) 425 if incomplete_trials: 426 if raise_on_failed_trial: --> 427 raise TuneError("Trials did not complete", incomplete_trials) 428 else: 429 logger.error("Trials did not complete: %s", incomplete_trials) TuneError: ('Trials did not complete', [train_evaluate_40de99b4, train_evaluate_40de99b5, train_evaluate_40de99b6, train_evaluate_40de99ba, train_evaluate_40de99bd, train_evaluate_40de99c0, train_evaluate_40de99c3, train_evaluate_423b1224, train_evaluate_423b1227, train_evaluate_423b122a, train_evaluate_423b122d, train_evaluate_4343b998, train_evaluate_4343b99b, train_evaluate_4343b99e, train_evaluate_4343b9a1, train_evaluate_446e43f5, train_evaluate_446e43f8, train_evaluate_446e43fb, train_evaluate_446e43fe, train_evaluate_45a19480, train_evaluate_45a19483, train_evaluate_45a19486, train_evaluate_4725842c, train_evaluate_4725842f, train_evaluate_47258432, train_evaluate_48110b2c, train_evaluate_48110b2f, train_evaluate_48110b32, train_evaluate_48110b35, train_evaluate_495895e2])
best_parameters, values = ax.get_best_parameters()
best_parameters
{'lr': 0.001200477766990381, 'momentum': 0.6754525434808601}
means, covariances = values
means
{'mean_accuracy': 0.9713334434324482}
render(
plot_contour(
model=ax.generation_strategy.model,
param_x="lr",
param_y="momentum",
metric_name="mean_accuracy",
)
)
/Users/deriksson/opt/anaconda3/lib/python3.8/site-packages/ax/modelbridge/torch.py:311: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
# `plot_single_method` expects a 2-d array of means, because it expects to average means from multiple
# optimization runs, so we wrap out best objectives array in another array.
best_objectives = np.array(
[[trial.objective_mean * 100 for trial in ax.experiment.trials.values()]]
)
best_objective_plot = optimization_trace_single_method(
y=np.maximum.accumulate(best_objectives, axis=1),
title="Model performance vs. # of iterations",
ylabel="Accuracy",
)
render(best_objective_plot)