Ax integrates easily with different scheduling frameworks and distributed training frameworks. In this example, Ax-driven optimization is executed in a distributed fashion using RayTune.
RayTune is a scalable framework for hyperparameter tuning that provides many state-of-the-art hyperparameter tuning algorithms and seamlessly scales from laptop to distributed cluster with fault tolerance. RayTune leverages Ray's Actor API to provide asynchronous parallel and distributed execution.
Ray 'Actors' are a simple and clean abstraction for replicating your Python classes across multiple workers and nodes. Each hyperparameter evaluation is asynchronously executed on a separate Ray actor and reports intermediate training progress back to RayTune. Upon reporting, RayTune then uses this information to performs actions such as early termination, re-prioritization, or checkpointing.
import logging
from ray import tune
from ray.tune import track
from ray.tune.suggest.ax import AxSearch
logger = logging.getLogger(tune.__name__)
logger.setLevel(
level=logging.CRITICAL
) # Reduce the number of Ray warnings that are not relevant here.
import numpy as np
import torch
from ax.plot.contour import plot_contour
from ax.plot.trace import optimization_trace_single_method
from ax.service.ax_client import AxClient
from ax.utils.notebook.plotting import init_notebook_plotting, render
from ax.utils.tutorials.cnn_utils import CNN, evaluate, load_mnist, train
init_notebook_plotting()
[INFO 12-08 18:53:21] ax.utils.notebook.plotting: Injecting Plotly library into cell. Do not overwrite or delete cell.
We specify enforce_sequential_optimization as False, because Ray runs many trials in parallel. With the sequential optimization enforcement, AxClient would expect the first few trials to be completed with data before generating more trials.
When high parallelism is not required, it is best to enforce sequential optimization, as it allows for achieving optimal results in fewer (but sequential) trials. In cases where parallelism is important, such as with distributed training using Ray, we choose to forego minimizing resource utilization and run more trials in parallel.
ax = AxClient(enforce_sequential_optimization=False)
[INFO 12-08 18:53:21] ax.service.ax_client: Starting optimization with verbose logging. To disable logging, set the `verbose_logging` argument to `False`. Note that float values in the logs are rounded to 2 decimal points.
Here we set up the search space and specify the objective; refer to the Ax API tutorials for more detail.
ax.create_experiment(
name="mnist_experiment",
parameters=[
{"name": "lr", "type": "range", "bounds": [1e-6, 0.4], "log_scale": True},
{"name": "momentum", "type": "range", "bounds": [0.0, 1.0]},
],
objective_name="mean_accuracy",
)
[INFO 12-08 18:53:21] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter lr. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 12-08 18:53:21] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter momentum. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 12-08 18:53:21] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+GPEI', steps=[Sobol for 5 trials, GPEI for subsequent trials]). Iterations after 5 will take longer to generate due to model-fitting.
Since we use the Ax Service API here, we evaluate the parameterizations that Ax suggests, using RayTune. The evaluation function follows its usual pattern, taking in a parameterization and outputting an objective value. For detail on evaluation functions, see Trial Evaluation.
def train_evaluate(parameterization):
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
train_loader, valid_loader, test_loader = load_mnist(data_path="~/.data")
net = train(
net=CNN(),
train_loader=train_loader,
parameters=parameterization,
dtype=torch.float,
device=device,
)
track.log(
mean_accuracy=evaluate(
net=net,
data_loader=valid_loader,
dtype=torch.float,
device=device,
)
)
Execute the Ax optimization and trial evaluation in RayTune using AxSearch algorithm:
tune.run(
train_evaluate,
num_samples=30,
search_alg=AxSearch(ax_client=ax, max_concurrent=3),
verbose=0, # Set this level to 1 to see status updates and to 2 to also see trial results.
# To use GPU, specify: resources_per_trial={"gpu": 1}.
)
2020-12-08 18:53:22,401 INFO services.py:1092 -- View the Ray dashboard at http://127.0.0.1:8265 [INFO 12-08 18:53:24] ax.service.ax_client: Generated new trial 0 with parameters {'lr': 0.0, 'momentum': 0.21}. [INFO 12-08 18:53:24] ax.service.ax_client: Generated new trial 1 with parameters {'lr': 0.0, 'momentum': 0.93}. [INFO 12-08 18:53:24] ax.service.ax_client: Generated new trial 2 with parameters {'lr': 0.01, 'momentum': 0.75}. 2020-12-08 18:53:24,594 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 7bbd902801000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4340) 2020-12-08 18:53:24,588 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4340) Traceback (most recent call last): (pid=4340) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4340) actor_class = pickle.loads(pickled_class) (pid=4340) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4340) from ray.tune.tune import run_experiments, run (pid=4340) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4340) from ray.tune.analysis import ExperimentAnalysis (pid=4340) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4340) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4340) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4340) from ray.tune.trial import Trial (pid=4340) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4340) from ray.tune.durable_trainable import DurableTrainable (pid=4340) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4340) from ray.tune.trainable import Trainable, TrainableUtil (pid=4340) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4340) from ray.tune.logger import UnifiedLogger (pid=4340) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4340) from ray.tune.syncer import get_node_syncer (pid=4340) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4340) @dataclass (pid=4340) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4340) return wrap(_cls) (pid=4340) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4340) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4340) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4340) for name, type in cls_annotations.items()] (pid=4340) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4340) for name, type in cls_annotations.items()] (pid=4340) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4340) if (_is_classvar(a_type, typing) (pid=4340) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4340) return type(a_type) is typing._ClassVar (pid=4340) AttributeError: module 'typing' has no attribute '_ClassVar' 2020-12-08 18:53:24,645 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID df5a1a8201000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4339) 2020-12-08 18:53:24,629 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4339) Traceback (most recent call last): (pid=4339) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4339) actor_class = pickle.loads(pickled_class) (pid=4339) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4339) from ray.tune.tune import run_experiments, run (pid=4339) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4339) from ray.tune.analysis import ExperimentAnalysis (pid=4339) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4339) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4339) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4339) from ray.tune.trial import Trial (pid=4339) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4339) from ray.tune.durable_trainable import DurableTrainable (pid=4339) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4339) from ray.tune.trainable import Trainable, TrainableUtil (pid=4339) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4339) from ray.tune.logger import UnifiedLogger (pid=4339) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4339) from ray.tune.syncer import get_node_syncer (pid=4339) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4339) @dataclass (pid=4339) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4339) return wrap(_cls) (pid=4339) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4339) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4339) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4339) for name, type in cls_annotations.items()] (pid=4339) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4339) for name, type in cls_annotations.items()] (pid=4339) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4339) if (_is_classvar(a_type, typing) (pid=4339) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4339) return type(a_type) is typing._ClassVar (pid=4339) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 12-08 18:53:24] ax.service.ax_client: Generated new trial 3 with parameters {'lr': 0.06, 'momentum': 0.34}. [INFO 12-08 18:53:24] ax.service.ax_client: Generated new trial 4 with parameters {'lr': 0.0, 'momentum': 0.86}. 2020-12-08 18:53:25,542 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 3bf0c85601000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' 2020-12-08 18:53:25,601 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID ae935fc001000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4423) 2020-12-08 18:53:25,537 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4423) Traceback (most recent call last): (pid=4423) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4423) actor_class = pickle.loads(pickled_class) (pid=4423) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4423) from ray.tune.tune import run_experiments, run (pid=4423) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4423) from ray.tune.analysis import ExperimentAnalysis (pid=4423) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4423) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4423) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4423) from ray.tune.trial import Trial (pid=4423) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4423) from ray.tune.durable_trainable import DurableTrainable (pid=4423) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4423) from ray.tune.trainable import Trainable, TrainableUtil (pid=4423) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4423) from ray.tune.logger import UnifiedLogger (pid=4423) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4423) from ray.tune.syncer import get_node_syncer (pid=4423) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4423) @dataclass (pid=4423) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4423) return wrap(_cls) (pid=4423) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4423) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4423) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4423) for name, type in cls_annotations.items()] (pid=4423) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4423) for name, type in cls_annotations.items()] (pid=4423) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4423) if (_is_classvar(a_type, typing) (pid=4423) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4423) return type(a_type) is typing._ClassVar (pid=4423) AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4422) 2020-12-08 18:53:25,591 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4422) Traceback (most recent call last): (pid=4422) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4422) actor_class = pickle.loads(pickled_class) (pid=4422) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4422) from ray.tune.tune import run_experiments, run (pid=4422) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4422) from ray.tune.analysis import ExperimentAnalysis (pid=4422) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4422) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4422) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4422) from ray.tune.trial import Trial (pid=4422) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4422) from ray.tune.durable_trainable import DurableTrainable (pid=4422) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4422) from ray.tune.trainable import Trainable, TrainableUtil (pid=4422) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4422) from ray.tune.logger import UnifiedLogger (pid=4422) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4422) from ray.tune.syncer import get_node_syncer (pid=4422) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4422) @dataclass (pid=4422) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4422) return wrap(_cls) (pid=4422) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4422) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4422) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4422) for name, type in cls_annotations.items()] (pid=4422) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4422) for name, type in cls_annotations.items()] (pid=4422) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4422) if (_is_classvar(a_type, typing) (pid=4422) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4422) return type(a_type) is typing._ClassVar (pid=4422) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 12-08 18:53:25] ax.service.ax_client: Generated new trial 5 with parameters {'lr': 0.0, 'momentum': 0.07}. [INFO 12-08 18:53:25] ax.service.ax_client: Generated new trial 6 with parameters {'lr': 0.0, 'momentum': 0.16}. (pid=4446) 2020-12-08 18:53:25,988 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4446) Traceback (most recent call last): (pid=4446) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4446) actor_class = pickle.loads(pickled_class) (pid=4446) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4446) from ray.tune.tune import run_experiments, run (pid=4446) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4446) from ray.tune.analysis import ExperimentAnalysis (pid=4446) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4446) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4446) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4446) from ray.tune.trial import Trial (pid=4446) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4446) from ray.tune.durable_trainable import DurableTrainable (pid=4446) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4446) from ray.tune.trainable import Trainable, TrainableUtil (pid=4446) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4446) from ray.tune.logger import UnifiedLogger (pid=4446) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4446) from ray.tune.syncer import get_node_syncer (pid=4446) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4446) @dataclass (pid=4446) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4446) return wrap(_cls) (pid=4446) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4446) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4446) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4446) for name, type in cls_annotations.items()] (pid=4446) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4446) for name, type in cls_annotations.items()] (pid=4446) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4446) if (_is_classvar(a_type, typing) (pid=4446) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4446) return type(a_type) is typing._ClassVar (pid=4446) AttributeError: module 'typing' has no attribute '_ClassVar' 2020-12-08 18:53:26,002 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID b7603b6c01000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 12-08 18:53:26] ax.service.ax_client: Generated new trial 7 with parameters {'lr': 0.12, 'momentum': 0.97}. 2020-12-08 18:53:26,687 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID be3cb80901000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4476) 2020-12-08 18:53:26,678 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4476) Traceback (most recent call last): (pid=4476) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4476) actor_class = pickle.loads(pickled_class) (pid=4476) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4476) from ray.tune.tune import run_experiments, run (pid=4476) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4476) from ray.tune.analysis import ExperimentAnalysis (pid=4476) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4476) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4476) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4476) from ray.tune.trial import Trial (pid=4476) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4476) from ray.tune.durable_trainable import DurableTrainable (pid=4476) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4476) from ray.tune.trainable import Trainable, TrainableUtil (pid=4476) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4476) from ray.tune.logger import UnifiedLogger (pid=4476) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4476) from ray.tune.syncer import get_node_syncer (pid=4476) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4476) @dataclass (pid=4476) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4476) return wrap(_cls) (pid=4476) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4476) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4476) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4476) for name, type in cls_annotations.items()] (pid=4476) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4476) for name, type in cls_annotations.items()] (pid=4476) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4476) if (_is_classvar(a_type, typing) (pid=4476) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4476) return type(a_type) is typing._ClassVar (pid=4476) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 12-08 18:53:26] ax.service.ax_client: Generated new trial 8 with parameters {'lr': 0.0, 'momentum': 0.38}. 2020-12-08 18:53:26,964 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID af23404c01000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4492) 2020-12-08 18:53:26,959 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4492) Traceback (most recent call last): (pid=4492) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4492) actor_class = pickle.loads(pickled_class) (pid=4492) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4492) from ray.tune.tune import run_experiments, run (pid=4492) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4492) from ray.tune.analysis import ExperimentAnalysis (pid=4492) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4492) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4492) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4492) from ray.tune.trial import Trial (pid=4492) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4492) from ray.tune.durable_trainable import DurableTrainable (pid=4492) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4492) from ray.tune.trainable import Trainable, TrainableUtil (pid=4492) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4492) from ray.tune.logger import UnifiedLogger (pid=4492) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4492) from ray.tune.syncer import get_node_syncer (pid=4492) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4492) @dataclass (pid=4492) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4492) return wrap(_cls) (pid=4492) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4492) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4492) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4492) for name, type in cls_annotations.items()] (pid=4492) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4492) for name, type in cls_annotations.items()] (pid=4492) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4492) if (_is_classvar(a_type, typing) (pid=4492) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4492) return type(a_type) is typing._ClassVar (pid=4492) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 12-08 18:53:27] ax.service.ax_client: Generated new trial 9 with parameters {'lr': 0.0, 'momentum': 0.61}. 2020-12-08 18:53:27,242 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 7d26912601000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4479) 2020-12-08 18:53:27,232 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4479) Traceback (most recent call last): (pid=4479) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4479) actor_class = pickle.loads(pickled_class) (pid=4479) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4479) from ray.tune.tune import run_experiments, run (pid=4479) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4479) from ray.tune.analysis import ExperimentAnalysis (pid=4479) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4479) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4479) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4479) from ray.tune.trial import Trial (pid=4479) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4479) from ray.tune.durable_trainable import DurableTrainable (pid=4479) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4479) from ray.tune.trainable import Trainable, TrainableUtil (pid=4479) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4479) from ray.tune.logger import UnifiedLogger (pid=4479) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4479) from ray.tune.syncer import get_node_syncer (pid=4479) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4479) @dataclass (pid=4479) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4479) return wrap(_cls) (pid=4479) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4479) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4479) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4479) for name, type in cls_annotations.items()] (pid=4479) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4479) for name, type in cls_annotations.items()] (pid=4479) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4479) if (_is_classvar(a_type, typing) (pid=4479) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4479) return type(a_type) is typing._ClassVar (pid=4479) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 12-08 18:53:27] ax.service.ax_client: Generated new trial 10 with parameters {'lr': 0.0, 'momentum': 0.7}. 2020-12-08 18:53:28,071 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 335d6f0801000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4531) 2020-12-08 18:53:28,067 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4531) Traceback (most recent call last): (pid=4531) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4531) actor_class = pickle.loads(pickled_class) (pid=4531) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4531) from ray.tune.tune import run_experiments, run (pid=4531) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4531) from ray.tune.analysis import ExperimentAnalysis (pid=4531) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4531) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4531) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4531) from ray.tune.trial import Trial (pid=4531) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4531) from ray.tune.durable_trainable import DurableTrainable (pid=4531) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4531) from ray.tune.trainable import Trainable, TrainableUtil (pid=4531) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4531) from ray.tune.logger import UnifiedLogger (pid=4531) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4531) from ray.tune.syncer import get_node_syncer (pid=4531) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4531) @dataclass (pid=4531) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4531) return wrap(_cls) (pid=4531) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4531) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4531) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4531) for name, type in cls_annotations.items()] (pid=4531) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4531) for name, type in cls_annotations.items()] (pid=4531) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4531) if (_is_classvar(a_type, typing) (pid=4531) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4531) return type(a_type) is typing._ClassVar (pid=4531) AttributeError: module 'typing' has no attribute '_ClassVar' 2020-12-08 18:53:28,242 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID ad5fa5cb01000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4534) 2020-12-08 18:53:28,231 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4534) Traceback (most recent call last): (pid=4534) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4534) actor_class = pickle.loads(pickled_class) (pid=4534) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4534) from ray.tune.tune import run_experiments, run (pid=4534) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4534) from ray.tune.analysis import ExperimentAnalysis (pid=4534) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4534) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4534) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4534) from ray.tune.trial import Trial (pid=4534) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4534) from ray.tune.durable_trainable import DurableTrainable (pid=4534) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4534) from ray.tune.trainable import Trainable, TrainableUtil (pid=4534) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4534) from ray.tune.logger import UnifiedLogger (pid=4534) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4534) from ray.tune.syncer import get_node_syncer (pid=4534) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4534) @dataclass (pid=4534) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4534) return wrap(_cls) (pid=4534) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4534) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4534) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4534) for name, type in cls_annotations.items()] (pid=4534) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4534) for name, type in cls_annotations.items()] (pid=4534) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4534) if (_is_classvar(a_type, typing) (pid=4534) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4534) return type(a_type) is typing._ClassVar (pid=4534) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 12-08 18:53:28] ax.service.ax_client: Generated new trial 11 with parameters {'lr': 0.0, 'momentum': 0.29}. 2020-12-08 18:53:28,570 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 35d7a3b901000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4549) 2020-12-08 18:53:28,565 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4549) Traceback (most recent call last): (pid=4549) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4549) actor_class = pickle.loads(pickled_class) (pid=4549) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4549) from ray.tune.tune import run_experiments, run (pid=4549) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4549) from ray.tune.analysis import ExperimentAnalysis (pid=4549) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4549) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4549) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4549) from ray.tune.trial import Trial (pid=4549) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4549) from ray.tune.durable_trainable import DurableTrainable (pid=4549) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4549) from ray.tune.trainable import Trainable, TrainableUtil (pid=4549) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4549) from ray.tune.logger import UnifiedLogger (pid=4549) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4549) from ray.tune.syncer import get_node_syncer (pid=4549) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4549) @dataclass (pid=4549) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4549) return wrap(_cls) (pid=4549) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4549) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4549) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4549) for name, type in cls_annotations.items()] (pid=4549) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4549) for name, type in cls_annotations.items()] (pid=4549) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4549) if (_is_classvar(a_type, typing) (pid=4549) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4549) return type(a_type) is typing._ClassVar (pid=4549) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 12-08 18:53:28] ax.service.ax_client: Generated new trial 12 with parameters {'lr': 0.0, 'momentum': 0.93}. [INFO 12-08 18:53:28] ax.service.ax_client: Generated new trial 13 with parameters {'lr': 0.01, 'momentum': 0.86}. 2020-12-08 18:53:29,466 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID b9c4a7c601000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' 2020-12-08 18:53:29,488 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 91fdfca801000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4585) 2020-12-08 18:53:29,478 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4585) Traceback (most recent call last): (pid=4585) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4585) actor_class = pickle.loads(pickled_class) (pid=4585) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4585) from ray.tune.tune import run_experiments, run (pid=4585) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4585) from ray.tune.analysis import ExperimentAnalysis (pid=4585) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4585) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4585) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4585) from ray.tune.trial import Trial (pid=4585) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4585) from ray.tune.durable_trainable import DurableTrainable (pid=4585) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4585) from ray.tune.trainable import Trainable, TrainableUtil (pid=4585) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4585) from ray.tune.logger import UnifiedLogger (pid=4585) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4585) from ray.tune.syncer import get_node_syncer (pid=4585) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4585) @dataclass (pid=4585) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4585) return wrap(_cls) (pid=4585) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4585) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4585) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4585) for name, type in cls_annotations.items()] (pid=4585) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4585) for name, type in cls_annotations.items()] (pid=4585) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4585) if (_is_classvar(a_type, typing) (pid=4585) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4585) return type(a_type) is typing._ClassVar (pid=4585) AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4588) 2020-12-08 18:53:29,453 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4588) Traceback (most recent call last): (pid=4588) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4588) actor_class = pickle.loads(pickled_class) (pid=4588) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4588) from ray.tune.tune import run_experiments, run (pid=4588) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4588) from ray.tune.analysis import ExperimentAnalysis (pid=4588) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4588) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4588) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4588) from ray.tune.trial import Trial (pid=4588) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4588) from ray.tune.durable_trainable import DurableTrainable (pid=4588) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4588) from ray.tune.trainable import Trainable, TrainableUtil (pid=4588) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4588) from ray.tune.logger import UnifiedLogger (pid=4588) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4588) from ray.tune.syncer import get_node_syncer (pid=4588) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4588) @dataclass (pid=4588) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4588) return wrap(_cls) (pid=4588) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4588) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4588) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4588) for name, type in cls_annotations.items()] (pid=4588) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4588) for name, type in cls_annotations.items()] (pid=4588) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4588) if (_is_classvar(a_type, typing) (pid=4588) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4588) return type(a_type) is typing._ClassVar (pid=4588) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 12-08 18:53:29] ax.service.ax_client: Generated new trial 14 with parameters {'lr': 0.0, 'momentum': 0.72}. 2020-12-08 18:53:29,831 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID c65d7ae001000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4602) 2020-12-08 18:53:29,822 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4602) Traceback (most recent call last): (pid=4602) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4602) actor_class = pickle.loads(pickled_class) (pid=4602) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4602) from ray.tune.tune import run_experiments, run (pid=4602) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4602) from ray.tune.analysis import ExperimentAnalysis (pid=4602) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4602) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4602) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4602) from ray.tune.trial import Trial (pid=4602) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4602) from ray.tune.durable_trainable import DurableTrainable (pid=4602) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4602) from ray.tune.trainable import Trainable, TrainableUtil (pid=4602) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4602) from ray.tune.logger import UnifiedLogger (pid=4602) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4602) from ray.tune.syncer import get_node_syncer (pid=4602) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4602) @dataclass (pid=4602) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4602) return wrap(_cls) (pid=4602) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4602) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4602) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4602) for name, type in cls_annotations.items()] (pid=4602) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4602) for name, type in cls_annotations.items()] (pid=4602) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4602) if (_is_classvar(a_type, typing) (pid=4602) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4602) return type(a_type) is typing._ClassVar (pid=4602) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 12-08 18:53:30] ax.service.ax_client: Generated new trial 15 with parameters {'lr': 0.0, 'momentum': 0.63}. [INFO 12-08 18:53:30] ax.service.ax_client: Generated new trial 16 with parameters {'lr': 0.0, 'momentum': 0.38}. 2020-12-08 18:53:30,765 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 090f0d8b01000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4639) 2020-12-08 18:53:30,757 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4639) Traceback (most recent call last): (pid=4639) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4639) actor_class = pickle.loads(pickled_class) (pid=4639) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4639) from ray.tune.tune import run_experiments, run (pid=4639) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4639) from ray.tune.analysis import ExperimentAnalysis (pid=4639) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4639) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4639) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4639) from ray.tune.trial import Trial (pid=4639) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4639) from ray.tune.durable_trainable import DurableTrainable (pid=4639) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4639) from ray.tune.trainable import Trainable, TrainableUtil (pid=4639) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4639) from ray.tune.logger import UnifiedLogger (pid=4639) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4639) from ray.tune.syncer import get_node_syncer (pid=4639) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4639) @dataclass (pid=4639) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4639) return wrap(_cls) (pid=4639) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4639) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4639) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4639) for name, type in cls_annotations.items()] (pid=4639) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4639) for name, type in cls_annotations.items()] (pid=4639) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4639) if (_is_classvar(a_type, typing) (pid=4639) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4639) return type(a_type) is typing._ClassVar (pid=4639) AttributeError: module 'typing' has no attribute '_ClassVar' 2020-12-08 18:53:31,028 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 3d80c2fe01000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4640) 2020-12-08 18:53:31,013 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4640) Traceback (most recent call last): (pid=4640) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4640) actor_class = pickle.loads(pickled_class) (pid=4640) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4640) from ray.tune.tune import run_experiments, run (pid=4640) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4640) from ray.tune.analysis import ExperimentAnalysis (pid=4640) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4640) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4640) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4640) from ray.tune.trial import Trial (pid=4640) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4640) from ray.tune.durable_trainable import DurableTrainable (pid=4640) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4640) from ray.tune.trainable import Trainable, TrainableUtil (pid=4640) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4640) from ray.tune.logger import UnifiedLogger (pid=4640) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4640) from ray.tune.syncer import get_node_syncer (pid=4640) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4640) @dataclass (pid=4640) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4640) return wrap(_cls) (pid=4640) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4640) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4640) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4640) for name, type in cls_annotations.items()] (pid=4640) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4640) for name, type in cls_annotations.items()] (pid=4640) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4640) if (_is_classvar(a_type, typing) (pid=4640) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4640) return type(a_type) is typing._ClassVar (pid=4640) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 12-08 18:53:31] ax.service.ax_client: Generated new trial 17 with parameters {'lr': 0.05, 'momentum': 0.34}. [INFO 12-08 18:53:31] ax.service.ax_client: Generated new trial 18 with parameters {'lr': 0.02, 'momentum': 0.08}. 2020-12-08 18:53:31,671 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 4305340a01000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4657) 2020-12-08 18:53:31,665 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4657) Traceback (most recent call last): (pid=4657) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4657) actor_class = pickle.loads(pickled_class) (pid=4657) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4657) from ray.tune.tune import run_experiments, run (pid=4657) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4657) from ray.tune.analysis import ExperimentAnalysis (pid=4657) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4657) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4657) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4657) from ray.tune.trial import Trial (pid=4657) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4657) from ray.tune.durable_trainable import DurableTrainable (pid=4657) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4657) from ray.tune.trainable import Trainable, TrainableUtil (pid=4657) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4657) from ray.tune.logger import UnifiedLogger (pid=4657) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4657) from ray.tune.syncer import get_node_syncer (pid=4657) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4657) @dataclass (pid=4657) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4657) return wrap(_cls) (pid=4657) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4657) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4657) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4657) for name, type in cls_annotations.items()] (pid=4657) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4657) for name, type in cls_annotations.items()] (pid=4657) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4657) if (_is_classvar(a_type, typing) (pid=4657) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4657) return type(a_type) is typing._ClassVar (pid=4657) AttributeError: module 'typing' has no attribute '_ClassVar' 2020-12-08 18:53:32,064 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID a7357af301000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4678) 2020-12-08 18:53:32,049 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4678) Traceback (most recent call last): (pid=4678) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4678) actor_class = pickle.loads(pickled_class) (pid=4678) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4678) from ray.tune.tune import run_experiments, run (pid=4678) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4678) from ray.tune.analysis import ExperimentAnalysis (pid=4678) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4678) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4678) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4678) from ray.tune.trial import Trial (pid=4678) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4678) from ray.tune.durable_trainable import DurableTrainable (pid=4678) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4678) from ray.tune.trainable import Trainable, TrainableUtil (pid=4678) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4678) from ray.tune.logger import UnifiedLogger (pid=4678) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4678) from ray.tune.syncer import get_node_syncer (pid=4678) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4678) @dataclass (pid=4678) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4678) return wrap(_cls) (pid=4678) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4678) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4678) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4678) for name, type in cls_annotations.items()] (pid=4678) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4678) for name, type in cls_annotations.items()] (pid=4678) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4678) if (_is_classvar(a_type, typing) (pid=4678) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4678) return type(a_type) is typing._ClassVar (pid=4678) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 12-08 18:53:32] ax.service.ax_client: Generated new trial 19 with parameters {'lr': 0.0, 'momentum': 0.28}. 2020-12-08 18:53:32,575 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 31965ca801000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4696) 2020-12-08 18:53:32,551 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4696) Traceback (most recent call last): (pid=4696) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4696) actor_class = pickle.loads(pickled_class) (pid=4696) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4696) from ray.tune.tune import run_experiments, run (pid=4696) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4696) from ray.tune.analysis import ExperimentAnalysis (pid=4696) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4696) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4696) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4696) from ray.tune.trial import Trial (pid=4696) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4696) from ray.tune.durable_trainable import DurableTrainable (pid=4696) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4696) from ray.tune.trainable import Trainable, TrainableUtil (pid=4696) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4696) from ray.tune.logger import UnifiedLogger (pid=4696) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4696) from ray.tune.syncer import get_node_syncer (pid=4696) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4696) @dataclass (pid=4696) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4696) return wrap(_cls) (pid=4696) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4696) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4696) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4696) for name, type in cls_annotations.items()] (pid=4696) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4696) for name, type in cls_annotations.items()] (pid=4696) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4696) if (_is_classvar(a_type, typing) (pid=4696) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4696) return type(a_type) is typing._ClassVar (pid=4696) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 12-08 18:53:32] ax.service.ax_client: Generated new trial 20 with parameters {'lr': 0.0, 'momentum': 0.0}. [INFO 12-08 18:53:33] ax.service.ax_client: Generated new trial 21 with parameters {'lr': 0.0, 'momentum': 0.11}. 2020-12-08 18:53:33,136 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 8fcb3d8501000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4714) 2020-12-08 18:53:33,126 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4714) Traceback (most recent call last): (pid=4714) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4714) actor_class = pickle.loads(pickled_class) (pid=4714) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4714) from ray.tune.tune import run_experiments, run (pid=4714) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4714) from ray.tune.analysis import ExperimentAnalysis (pid=4714) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4714) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4714) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4714) from ray.tune.trial import Trial (pid=4714) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4714) from ray.tune.durable_trainable import DurableTrainable (pid=4714) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4714) from ray.tune.trainable import Trainable, TrainableUtil (pid=4714) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4714) from ray.tune.logger import UnifiedLogger (pid=4714) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4714) from ray.tune.syncer import get_node_syncer (pid=4714) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4714) @dataclass (pid=4714) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4714) return wrap(_cls) (pid=4714) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4714) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4714) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4714) for name, type in cls_annotations.items()] (pid=4714) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4714) for name, type in cls_annotations.items()] (pid=4714) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4714) if (_is_classvar(a_type, typing) (pid=4714) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4714) return type(a_type) is typing._ClassVar (pid=4714) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 12-08 18:53:33] ax.service.ax_client: Generated new trial 22 with parameters {'lr': 0.0, 'momentum': 0.78}. 2020-12-08 18:53:33,657 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID c662ca5a01000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4732) 2020-12-08 18:53:33,646 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4732) Traceback (most recent call last): (pid=4732) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4732) actor_class = pickle.loads(pickled_class) (pid=4732) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4732) from ray.tune.tune import run_experiments, run (pid=4732) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4732) from ray.tune.analysis import ExperimentAnalysis (pid=4732) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4732) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4732) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4732) from ray.tune.trial import Trial (pid=4732) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4732) from ray.tune.durable_trainable import DurableTrainable (pid=4732) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4732) from ray.tune.trainable import Trainable, TrainableUtil (pid=4732) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4732) from ray.tune.logger import UnifiedLogger (pid=4732) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4732) from ray.tune.syncer import get_node_syncer (pid=4732) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4732) @dataclass (pid=4732) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4732) return wrap(_cls) (pid=4732) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4732) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4732) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4732) for name, type in cls_annotations.items()] (pid=4732) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4732) for name, type in cls_annotations.items()] (pid=4732) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4732) if (_is_classvar(a_type, typing) (pid=4732) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4732) return type(a_type) is typing._ClassVar (pid=4732) AttributeError: module 'typing' has no attribute '_ClassVar' 2020-12-08 18:53:34,139 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID a5450a3401000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4753) 2020-12-08 18:53:34,115 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4753) Traceback (most recent call last): (pid=4753) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4753) actor_class = pickle.loads(pickled_class) (pid=4753) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4753) from ray.tune.tune import run_experiments, run (pid=4753) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4753) from ray.tune.analysis import ExperimentAnalysis (pid=4753) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4753) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4753) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4753) from ray.tune.trial import Trial (pid=4753) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4753) from ray.tune.durable_trainable import DurableTrainable (pid=4753) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4753) from ray.tune.trainable import Trainable, TrainableUtil (pid=4753) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4753) from ray.tune.logger import UnifiedLogger (pid=4753) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4753) from ray.tune.syncer import get_node_syncer (pid=4753) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4753) @dataclass (pid=4753) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4753) return wrap(_cls) (pid=4753) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4753) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4753) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4753) for name, type in cls_annotations.items()] (pid=4753) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4753) for name, type in cls_annotations.items()] (pid=4753) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4753) if (_is_classvar(a_type, typing) (pid=4753) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4753) return type(a_type) is typing._ClassVar (pid=4753) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 12-08 18:53:34] ax.service.ax_client: Generated new trial 23 with parameters {'lr': 0.0, 'momentum': 0.64}. 2020-12-08 18:53:34,366 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID b299cffa01000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4771) 2020-12-08 18:53:34,350 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4771) Traceback (most recent call last): (pid=4771) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4771) actor_class = pickle.loads(pickled_class) (pid=4771) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4771) from ray.tune.tune import run_experiments, run (pid=4771) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4771) from ray.tune.analysis import ExperimentAnalysis (pid=4771) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4771) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4771) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4771) from ray.tune.trial import Trial (pid=4771) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4771) from ray.tune.durable_trainable import DurableTrainable (pid=4771) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4771) from ray.tune.trainable import Trainable, TrainableUtil (pid=4771) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4771) from ray.tune.logger import UnifiedLogger (pid=4771) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4771) from ray.tune.syncer import get_node_syncer (pid=4771) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4771) @dataclass (pid=4771) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4771) return wrap(_cls) (pid=4771) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4771) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4771) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4771) for name, type in cls_annotations.items()] (pid=4771) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4771) for name, type in cls_annotations.items()] (pid=4771) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4771) if (_is_classvar(a_type, typing) (pid=4771) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4771) return type(a_type) is typing._ClassVar (pid=4771) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 12-08 18:53:34] ax.service.ax_client: Generated new trial 24 with parameters {'lr': 0.0, 'momentum': 0.62}. 2020-12-08 18:53:35,066 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 65710efb01000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4789) 2020-12-08 18:53:35,060 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4789) Traceback (most recent call last): (pid=4789) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4789) actor_class = pickle.loads(pickled_class) (pid=4789) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4789) from ray.tune.tune import run_experiments, run (pid=4789) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4789) from ray.tune.analysis import ExperimentAnalysis (pid=4789) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4789) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4789) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4789) from ray.tune.trial import Trial (pid=4789) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4789) from ray.tune.durable_trainable import DurableTrainable (pid=4789) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4789) from ray.tune.trainable import Trainable, TrainableUtil (pid=4789) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4789) from ray.tune.logger import UnifiedLogger (pid=4789) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4789) from ray.tune.syncer import get_node_syncer (pid=4789) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4789) @dataclass (pid=4789) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4789) return wrap(_cls) (pid=4789) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4789) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4789) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4789) for name, type in cls_annotations.items()] (pid=4789) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4789) for name, type in cls_annotations.items()] (pid=4789) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4789) if (_is_classvar(a_type, typing) (pid=4789) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4789) return type(a_type) is typing._ClassVar (pid=4789) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 12-08 18:53:35] ax.service.ax_client: Generated new trial 25 with parameters {'lr': 0.0, 'momentum': 0.09}. [INFO 12-08 18:53:35] ax.service.ax_client: Generated new trial 26 with parameters {'lr': 0.0, 'momentum': 0.95}. (pid=4808) 2020-12-08 18:53:35,764 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4808) Traceback (most recent call last): (pid=4808) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4808) actor_class = pickle.loads(pickled_class) (pid=4808) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module>2020-12-08 18:53:35,779 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 1211da1301000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4808) from ray.tune.tune import run_experiments, run (pid=4808) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4808) from ray.tune.analysis import ExperimentAnalysis (pid=4808) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4808) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4808) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4808) from ray.tune.trial import Trial (pid=4808) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4808) from ray.tune.durable_trainable import DurableTrainable (pid=4808) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4808) from ray.tune.trainable import Trainable, TrainableUtil (pid=4808) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4808) from ray.tune.logger import UnifiedLogger (pid=4808) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4808) from ray.tune.syncer import get_node_syncer (pid=4808) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4808) @dataclass (pid=4808) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4808) return wrap(_cls) (pid=4808) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4808) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4808) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4808) for name, type in cls_annotations.items()] (pid=4808) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4808) for name, type in cls_annotations.items()] (pid=4808) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4808) if (_is_classvar(a_type, typing) (pid=4808) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4808) return type(a_type) is typing._ClassVar (pid=4808) AttributeError: module 'typing' has no attribute '_ClassVar' 2020-12-08 18:53:36,152 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 03d9348501000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4826) 2020-12-08 18:53:36,132 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4826) Traceback (most recent call last): (pid=4826) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4826) actor_class = pickle.loads(pickled_class) (pid=4826) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4826) from ray.tune.tune import run_experiments, run (pid=4826) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4826) from ray.tune.analysis import ExperimentAnalysis (pid=4826) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4826) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4826) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4826) from ray.tune.trial import Trial (pid=4826) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4826) from ray.tune.durable_trainable import DurableTrainable (pid=4826) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4826) from ray.tune.trainable import Trainable, TrainableUtil (pid=4826) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4826) from ray.tune.logger import UnifiedLogger (pid=4826) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4826) from ray.tune.syncer import get_node_syncer (pid=4826) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4826) @dataclass (pid=4826) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4826) return wrap(_cls) (pid=4826) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4826) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4826) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4826) for name, type in cls_annotations.items()] (pid=4826) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4826) for name, type in cls_annotations.items()] (pid=4826) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4826) if (_is_classvar(a_type, typing) (pid=4826) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4826) return type(a_type) is typing._ClassVar (pid=4826) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 12-08 18:53:36] ax.service.ax_client: Generated new trial 27 with parameters {'lr': 0.1, 'momentum': 0.04}. 2020-12-08 18:53:36,689 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 814c5aa201000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4844) 2020-12-08 18:53:36,670 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4844) Traceback (most recent call last): (pid=4844) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4844) actor_class = pickle.loads(pickled_class) (pid=4844) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4844) from ray.tune.tune import run_experiments, run (pid=4844) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4844) from ray.tune.analysis import ExperimentAnalysis (pid=4844) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4844) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4844) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4844) from ray.tune.trial import Trial (pid=4844) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4844) from ray.tune.durable_trainable import DurableTrainable (pid=4844) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4844) from ray.tune.trainable import Trainable, TrainableUtil (pid=4844) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4844) from ray.tune.logger import UnifiedLogger (pid=4844) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4844) from ray.tune.syncer import get_node_syncer (pid=4844) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4844) @dataclass (pid=4844) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4844) return wrap(_cls) (pid=4844) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4844) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4844) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4844) for name, type in cls_annotations.items()] (pid=4844) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4844) for name, type in cls_annotations.items()] (pid=4844) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4844) if (_is_classvar(a_type, typing) (pid=4844) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4844) return type(a_type) is typing._ClassVar (pid=4844) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 12-08 18:53:36] ax.service.ax_client: Generated new trial 28 with parameters {'lr': 0.01, 'momentum': 0.48}. 2020-12-08 18:53:37,005 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 90e2cc4001000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4862) 2020-12-08 18:53:37,000 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4862) Traceback (most recent call last): (pid=4862) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4862) actor_class = pickle.loads(pickled_class) (pid=4862) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4862) from ray.tune.tune import run_experiments, run (pid=4862) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4862) from ray.tune.analysis import ExperimentAnalysis (pid=4862) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4862) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4862) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4862) from ray.tune.trial import Trial (pid=4862) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4862) from ray.tune.durable_trainable import DurableTrainable (pid=4862) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4862) from ray.tune.trainable import Trainable, TrainableUtil (pid=4862) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4862) from ray.tune.logger import UnifiedLogger (pid=4862) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4862) from ray.tune.syncer import get_node_syncer (pid=4862) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4862) @dataclass (pid=4862) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4862) return wrap(_cls) (pid=4862) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4862) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4862) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4862) for name, type in cls_annotations.items()] (pid=4862) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4862) for name, type in cls_annotations.items()] (pid=4862) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4862) if (_is_classvar(a_type, typing) (pid=4862) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4862) return type(a_type) is typing._ClassVar (pid=4862) AttributeError: module 'typing' has no attribute '_ClassVar' [INFO 12-08 18:53:37] ax.service.ax_client: Generated new trial 29 with parameters {'lr': 0.06, 'momentum': 0.89}. 2020-12-08 18:53:37,608 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 296a970401000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4880) 2020-12-08 18:53:37,601 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4880) Traceback (most recent call last): (pid=4880) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4880) actor_class = pickle.loads(pickled_class) (pid=4880) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4880) from ray.tune.tune import run_experiments, run (pid=4880) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4880) from ray.tune.analysis import ExperimentAnalysis (pid=4880) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4880) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4880) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4880) from ray.tune.trial import Trial (pid=4880) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4880) from ray.tune.durable_trainable import DurableTrainable (pid=4880) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4880) from ray.tune.trainable import Trainable, TrainableUtil (pid=4880) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4880) from ray.tune.logger import UnifiedLogger (pid=4880) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4880) from ray.tune.syncer import get_node_syncer (pid=4880) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4880) @dataclass (pid=4880) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4880) return wrap(_cls) (pid=4880) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4880) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4880) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4880) for name, type in cls_annotations.items()] (pid=4880) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4880) for name, type in cls_annotations.items()] (pid=4880) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4880) if (_is_classvar(a_type, typing) (pid=4880) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4880) return type(a_type) is typing._ClassVar (pid=4880) AttributeError: module 'typing' has no attribute '_ClassVar' 2020-12-08 18:53:37,901 WARNING worker.py:1091 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID f3b32ee701000000. Traceback: Traceback (most recent call last): File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> from ray.tune.tune import run_experiments, run File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> from ray.tune.analysis import ExperimentAnalysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> from ray.tune.trial import Trial File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> from ray.tune.durable_trainable import DurableTrainable File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> from ray.tune.trainable import Trainable, TrainableUtil File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> from ray.tune.logger import UnifiedLogger File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> from ray.tune.syncer import get_node_syncer File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> @dataclass File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass return wrap(_cls) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> for name, type in cls_annotations.items()] File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field if (_is_classvar(a_type, typing) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar return type(a_type) is typing._ClassVar AttributeError: module 'typing' has no attribute '_ClassVar' (pid=4898) 2020-12-08 18:53:37,894 ERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc. (pid=4898) Traceback (most recent call last): (pid=4898) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py", line 493, in _load_actor_class_from_gcs (pid=4898) actor_class = pickle.loads(pickled_class) (pid=4898) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py", line 2, in <module> (pid=4898) from ray.tune.tune import run_experiments, run (pid=4898) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py", line 7, in <module> (pid=4898) from ray.tune.analysis import ExperimentAnalysis (pid=4898) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 1, in <module> (pid=4898) from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis (pid=4898) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 19, in <module> (pid=4898) from ray.tune.trial import Trial (pid=4898) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py", line 15, in <module> (pid=4898) from ray.tune.durable_trainable import DurableTrainable (pid=4898) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py", line 4, in <module> (pid=4898) from ray.tune.trainable import Trainable, TrainableUtil (pid=4898) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py", line 23, in <module> (pid=4898) from ray.tune.logger import UnifiedLogger (pid=4898) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py", line 15, in <module> (pid=4898) from ray.tune.syncer import get_node_syncer (pid=4898) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py", line 80, in <module> (pid=4898) @dataclass (pid=4898) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 958, in dataclass (pid=4898) return wrap(_cls) (pid=4898) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 950, in wrap (pid=4898) return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) (pid=4898) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in _process_class (pid=4898) for name, type in cls_annotations.items()] (pid=4898) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 801, in <listcomp> (pid=4898) for name, type in cls_annotations.items()] (pid=4898) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 659, in _get_field (pid=4898) if (_is_classvar(a_type, typing) (pid=4898) File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py", line 550, in _is_classvar (pid=4898) return type(a_type) is typing._ClassVar (pid=4898) AttributeError: module 'typing' has no attribute '_ClassVar'
--------------------------------------------------------------------------- TuneError Traceback (most recent call last) <ipython-input-6-9007b87d2ec7> in <module> 3 num_samples=30, 4 search_alg=AxSearch(ax_client=ax, max_concurrent=3), ----> 5 verbose=0, # Set this level to 1 to see status updates and to 2 to also see trial results. 6 # To use GPU, specify: resources_per_trial={"gpu": 1}. 7 ) ~/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py in run(run_or_experiment, name, metric, mode, stop, time_budget_s, config, resources_per_trial, num_samples, local_dir, search_alg, scheduler, keep_checkpoints_num, checkpoint_score_attr, checkpoint_freq, checkpoint_at_end, verbose, progress_reporter, loggers, log_to_file, trial_name_creator, trial_dirname_creator, sync_config, export_formats, max_failures, fail_fast, restore, server_port, resume, queue_trials, reuse_actors, trial_executor, raise_on_failed_trial, callbacks, ray_auto_init, run_errored_only, global_checkpoint_period, with_server, upload_dir, sync_to_cloud, sync_to_driver, sync_on_checkpoint) 432 if incomplete_trials: 433 if raise_on_failed_trial: --> 434 raise TuneError("Trials did not complete", incomplete_trials) 435 else: 436 logger.error("Trials did not complete: %s", incomplete_trials) TuneError: ('Trials did not complete', [train_evaluate_a62ae542, train_evaluate_a62ae543, train_evaluate_a62ae544, train_evaluate_a62ae545, train_evaluate_a62ae546, train_evaluate_a62ae547, train_evaluate_a62ae548, train_evaluate_a7501014, train_evaluate_a7501015, train_evaluate_a7501016, train_evaluate_a7501017, train_evaluate_a88ec902, train_evaluate_a88ec903, train_evaluate_a88ec904, train_evaluate_a88ec905, train_evaluate_a88ec906, train_evaluate_a9c379ee, train_evaluate_a9c379ef, train_evaluate_a9c379f0, train_evaluate_a9c379f1, train_evaluate_aaf1e22e, train_evaluate_aaf1e22f, train_evaluate_aaf1e230, train_evaluate_aaf1e231, train_evaluate_ac475294, train_evaluate_ac475295, train_evaluate_ac475296, train_evaluate_ac475297, train_evaluate_ad761cc2, train_evaluate_ad761cc3])
best_parameters, values = ax.get_best_parameters()
best_parameters
{'lr': 0.001200477766990381, 'momentum': 0.6754525434808601}
means, covariances = values
means
{'mean_accuracy': 0.9713334434324482}
render(
plot_contour(
model=ax.generation_strategy.model,
param_x="lr",
param_y="momentum",
metric_name="mean_accuracy",
)
)
/Users/deriksson/opt/anaconda3/lib/python3.8/site-packages/ax/modelbridge/torch.py:311: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
# `plot_single_method` expects a 2-d array of means, because it expects to average means from multiple
# optimization runs, so we wrap out best objectives array in another array.
best_objectives = np.array(
[[trial.objective_mean * 100 for trial in ax.experiment.trials.values()]]
)
best_objective_plot = optimization_trace_single_method(
y=np.maximum.accumulate(best_objectives, axis=1),
title="Model performance vs. # of iterations",
ylabel="Accuracy",
)
render(best_objective_plot)