{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Ax Service API with RayTune on PyTorch CNN\n",
"\n",
"Ax integrates easily with different scheduling frameworks and distributed training frameworks. In this example, Ax-driven optimization is executed in a distributed fashion using [RayTune](https://ray.readthedocs.io/en/latest/tune.html). \n",
"\n",
"RayTune is a scalable framework for hyperparameter tuning that provides many state-of-the-art hyperparameter tuning algorithms and seamlessly scales from laptop to distributed cluster with fault tolerance. RayTune leverages [Ray](https://ray.readthedocs.io/)'s Actor API to provide asynchronous parallel and distributed execution.\n",
"\n",
"Ray 'Actors' are a simple and clean abstraction for replicating your Python classes across multiple workers and nodes. Each hyperparameter evaluation is asynchronously executed on a separate Ray actor and reports intermediate training progress back to RayTune. Upon reporting, RayTune then uses this information to performs actions such as early termination, re-prioritization, or checkpointing."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import logging\n",
"\n",
"from ray import tune\n",
"from ray.tune import track\n",
"from ray.tune.suggest.ax import AxSearch\n",
"\n",
"logger = logging.getLogger(tune.__name__)\n",
"logger.setLevel(\n",
" level=logging.CRITICAL\n",
") # Reduce the number of Ray warnings that are not relevant here."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 11-04 18:40:44] ax.utils.notebook.plotting: Injecting Plotly library into cell. Do not overwrite or delete cell.\n"
]
},
{
"data": {
"text/html": [
" \n",
" "
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import numpy as np\n",
"import torch\n",
"from ax.plot.contour import plot_contour\n",
"from ax.plot.trace import optimization_trace_single_method\n",
"from ax.service.ax_client import AxClient\n",
"from ax.utils.notebook.plotting import init_notebook_plotting, render\n",
"from ax.utils.tutorials.cnn_utils import CNN, evaluate, load_mnist, train\n",
"\n",
"init_notebook_plotting()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Initialize client\n",
"We specify `enforce_sequential_optimization` as False, because Ray runs many trials in parallel. With the sequential optimization enforcement, `AxClient` would expect the first few trials to be completed with data before generating more trials.\n",
"\n",
"When high parallelism is not required, it is best to enforce sequential optimization, as it allows for achieving optimal results in fewer (but sequential) trials. In cases where parallelism is important, such as with distributed training using Ray, we choose to forego minimizing resource utilization and run more trials in parallel."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 11-04 18:40:44] ax.service.ax_client: Starting optimization with verbose logging. To disable logging, set the `verbose_logging` argument to `False`. Note that float values in the logs are rounded to 2 decimal points.\n"
]
}
],
"source": [
"ax = AxClient(enforce_sequential_optimization=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Set up experiment\n",
"Here we set up the search space and specify the objective; refer to the Ax API tutorials for more detail."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 11-04 18:40:44] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter lr. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 11-04 18:40:44] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter momentum. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 11-04 18:40:44] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+GPEI', steps=[Sobol for 5 trials, GPEI for subsequent trials]). Iterations after 5 will take longer to generate due to model-fitting.\n"
]
}
],
"source": [
"ax.create_experiment(\n",
" name=\"mnist_experiment\",\n",
" parameters=[\n",
" {\"name\": \"lr\", \"type\": \"range\", \"bounds\": [1e-6, 0.4], \"log_scale\": True},\n",
" {\"name\": \"momentum\", \"type\": \"range\", \"bounds\": [0.0, 1.0]},\n",
" ],\n",
" objective_name=\"mean_accuracy\",\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Define how to evaluate trials\n",
"Since we use the Ax Service API here, we evaluate the parameterizations that Ax suggests, using RayTune. The evaluation function follows its usual pattern, taking in a parameterization and outputting an objective value. For detail on evaluation functions, see [Trial Evaluation](https://ax.dev/docs/runner.html). "
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"def train_evaluate(parameterization):\n",
" device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
" train_loader, valid_loader, test_loader = load_mnist(data_path=\"~/.data\")\n",
" net = train(\n",
" net=CNN(),\n",
" train_loader=train_loader,\n",
" parameters=parameterization,\n",
" dtype=torch.float,\n",
" device=device,\n",
" )\n",
" track.log(\n",
" mean_accuracy=evaluate(\n",
" net=net,\n",
" data_loader=valid_loader,\n",
" dtype=torch.float,\n",
" device=device,\n",
" )\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Run optimization\n",
"Execute the Ax optimization and trial evaluation in RayTune using [AxSearch algorithm](https://ray.readthedocs.io/en/latest/tune-searchalg.html#ax-search):"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2020-11-04 18:40:44,887\tINFO services.py:1166 -- View the Ray dashboard at \u001b[1m\u001b[32mhttp://127.0.0.1:8265\u001b[39m\u001b[22m\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 11-04 18:40:47] ax.service.ax_client: Generated new trial 0 with parameters {'lr': 0.0, 'momentum': 0.86}.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 11-04 18:40:47] ax.service.ax_client: Generated new trial 1 with parameters {'lr': 0.0, 'momentum': 0.25}.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 11-04 18:40:47] ax.service.ax_client: Generated new trial 2 with parameters {'lr': 0.0, 'momentum': 0.91}.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2020-11-04 18:40:47,431\tWARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID df5a1a8201000000. Traceback:\n",
"Traceback (most recent call last):\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py\", line 493, in _load_actor_class_from_gcs\n",
" actor_class = pickle.loads(pickled_class)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py\", line 2, in \n",
" from ray.tune.tune import run_experiments, run\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py\", line 5, in \n",
" from ray.tune.analysis import ExperimentAnalysis\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py\", line 1, in \n",
" from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py\", line 19, in \n",
" from ray.tune.trial import Trial\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py\", line 16, in \n",
" from ray.tune.durable_trainable import DurableTrainable\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py\", line 4, in \n",
" from ray.tune.trainable import Trainable, TrainableUtil\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py\", line 23, in \n",
" from ray.tune.logger import UnifiedLogger\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py\", line 15, in \n",
" from ray.tune.syncer import get_node_syncer\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py\", line 80, in \n",
" @dataclass\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 958, in dataclass\n",
" return wrap(_cls)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 950, in wrap\n",
" return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in _process_class\n",
" for name, type in cls_annotations.items()]\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in \n",
" for name, type in cls_annotations.items()]\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 659, in _get_field\n",
" if (_is_classvar(a_type, typing)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 550, in _is_classvar\n",
" return type(a_type) is typing._ClassVar\n",
"AttributeError: module 'typing' has no attribute '_ClassVar'\n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2020-11-04 18:40:47,477\tWARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 7bbd902801000000. Traceback:\n",
"Traceback (most recent call last):\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py\", line 493, in _load_actor_class_from_gcs\n",
" actor_class = pickle.loads(pickled_class)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py\", line 2, in \n",
" from ray.tune.tune import run_experiments, run\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py\", line 5, in \n",
" from ray.tune.analysis import ExperimentAnalysis\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py\", line 1, in \n",
" from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py\", line 19, in \n",
" from ray.tune.trial import Trial\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py\", line 16, in \n",
" from ray.tune.durable_trainable import DurableTrainable\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py\", line 4, in \n",
" from ray.tune.trainable import Trainable, TrainableUtil\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py\", line 23, in \n",
" from ray.tune.logger import UnifiedLogger\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py\", line 15, in \n",
" from ray.tune.syncer import get_node_syncer\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py\", line 80, in \n",
" @dataclass\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 958, in dataclass\n",
" return wrap(_cls)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 950, in wrap\n",
" return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in _process_class\n",
" for name, type in cls_annotations.items()]\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in \n",
" for name, type in cls_annotations.items()]\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 659, in _get_field\n",
" if (_is_classvar(a_type, typing)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 550, in _is_classvar\n",
" return type(a_type) is typing._ClassVar\n",
"AttributeError: module 'typing' has no attribute '_ClassVar'\n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m 2020-11-04 18:40:47,450\tERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc.\n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m Traceback (most recent call last):\n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py\", line 493, in _load_actor_class_from_gcs\n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m actor_class = pickle.loads(pickled_class)\n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py\", line 2, in \n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m from ray.tune.tune import run_experiments, run\n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py\", line 5, in \n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m from ray.tune.analysis import ExperimentAnalysis\n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py\", line 1, in \n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis\n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py\", line 19, in \n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m from ray.tune.trial import Trial\n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py\", line 16, in \n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m from ray.tune.durable_trainable import DurableTrainable\n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py\", line 4, in \n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m from ray.tune.trainable import Trainable, TrainableUtil\n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py\", line 23, in \n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m from ray.tune.logger import UnifiedLogger\n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py\", line 15, in \n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m from ray.tune.syncer import get_node_syncer\n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py\", line 80, in \n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m @dataclass\n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 958, in dataclass\n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m return wrap(_cls)\n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 950, in wrap\n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)\n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in _process_class\n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m for name, type in cls_annotations.items()]\n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in \n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m for name, type in cls_annotations.items()]\n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 659, in _get_field\n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m if (_is_classvar(a_type, typing)\n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 550, in _is_classvar\n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m return type(a_type) is typing._ClassVar\n",
"\u001b[2m\u001b[36m(pid=4270)\u001b[0m AttributeError: module 'typing' has no attribute '_ClassVar'\n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m 2020-11-04 18:40:47,425\tERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc.\n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m Traceback (most recent call last):\n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py\", line 493, in _load_actor_class_from_gcs\n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m actor_class = pickle.loads(pickled_class)\n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py\", line 2, in \n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m from ray.tune.tune import run_experiments, run\n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py\", line 5, in \n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m from ray.tune.analysis import ExperimentAnalysis\n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py\", line 1, in \n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis\n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py\", line 19, in \n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m from ray.tune.trial import Trial\n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py\", line 16, in \n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m from ray.tune.durable_trainable import DurableTrainable\n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py\", line 4, in \n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m from ray.tune.trainable import Trainable, TrainableUtil\n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py\", line 23, in \n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m from ray.tune.logger import UnifiedLogger\n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py\", line 15, in \n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m from ray.tune.syncer import get_node_syncer\n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py\", line 80, in \n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m @dataclass\n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 958, in dataclass\n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m return wrap(_cls)\n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 950, in wrap\n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)\n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in _process_class\n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m for name, type in cls_annotations.items()]\n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in \n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m for name, type in cls_annotations.items()]\n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 659, in _get_field\n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m if (_is_classvar(a_type, typing)\n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 550, in _is_classvar\n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m return type(a_type) is typing._ClassVar\n",
"\u001b[2m\u001b[36m(pid=4269)\u001b[0m AttributeError: module 'typing' has no attribute '_ClassVar'\n",
"[INFO 11-04 18:40:47] ax.service.ax_client: Generated new trial 3 with parameters {'lr': 0.0, 'momentum': 0.24}.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 11-04 18:40:47] ax.service.ax_client: Generated new trial 4 with parameters {'lr': 0.0, 'momentum': 0.99}.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2020-11-04 18:40:48,572\tWARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 3bf0c85601000000. Traceback:\n",
"Traceback (most recent call last):\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py\", line 493, in _load_actor_class_from_gcs\n",
" actor_class = pickle.loads(pickled_class)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py\", line 2, in \n",
" from ray.tune.tune import run_experiments, run\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py\", line 5, in \n",
" from ray.tune.analysis import ExperimentAnalysis\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py\", line 1, in \n",
" from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py\", line 19, in \n",
" from ray.tune.trial import Trial\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py\", line 16, in \n",
" from ray.tune.durable_trainable import DurableTrainable\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py\", line 4, in \n",
" from ray.tune.trainable import Trainable, TrainableUtil\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py\", line 23, in \n",
" from ray.tune.logger import UnifiedLogger\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py\", line 15, in \n",
" from ray.tune.syncer import get_node_syncer\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py\", line 80, in \n",
" @dataclass\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 958, in dataclass\n",
" return wrap(_cls)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 950, in wrap\n",
" return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in _process_class\n",
" for name, type in cls_annotations.items()]\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in \n",
" for name, type in cls_annotations.items()]\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 659, in _get_field\n",
" if (_is_classvar(a_type, typing)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 550, in _is_classvar\n",
" return type(a_type) is typing._ClassVar\n",
"AttributeError: module 'typing' has no attribute '_ClassVar'\n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m 2020-11-04 18:40:48,551\tERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc.\n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m Traceback (most recent call last):\n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py\", line 493, in _load_actor_class_from_gcs\n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m actor_class = pickle.loads(pickled_class)\n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py\", line 2, in \n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m from ray.tune.tune import run_experiments, run\n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py\", line 5, in \n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m from ray.tune.analysis import ExperimentAnalysis\n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py\", line 1, in \n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis\n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py\", line 19, in \n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m from ray.tune.trial import Trial\n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py\", line 16, in \n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m from ray.tune.durable_trainable import DurableTrainable\n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py\", line 4, in \n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m from ray.tune.trainable import Trainable, TrainableUtil\n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py\", line 23, in \n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m from ray.tune.logger import UnifiedLogger\n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py\", line 15, in \n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m from ray.tune.syncer import get_node_syncer\n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py\", line 80, in \n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m @dataclass\n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 958, in dataclass\n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m return wrap(_cls)\n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 950, in wrap\n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)\n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in _process_class\n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m for name, type in cls_annotations.items()]\n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in \n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m for name, type in cls_annotations.items()]\n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 659, in _get_field\n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m if (_is_classvar(a_type, typing)\n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 550, in _is_classvar\n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m return type(a_type) is typing._ClassVar\n",
"\u001b[2m\u001b[36m(pid=4355)\u001b[0m AttributeError: module 'typing' has no attribute '_ClassVar'\n",
"[INFO 11-04 18:40:48] ax.service.ax_client: Generated new trial 5 with parameters {'lr': 0.26, 'momentum': 0.17}.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2020-11-04 18:40:48,735\tWARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID ae935fc001000000. Traceback:\n",
"Traceback (most recent call last):\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py\", line 493, in _load_actor_class_from_gcs\n",
" actor_class = pickle.loads(pickled_class)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py\", line 2, in \n",
" from ray.tune.tune import run_experiments, run\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py\", line 5, in \n",
" from ray.tune.analysis import ExperimentAnalysis\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py\", line 1, in \n",
" from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py\", line 19, in \n",
" from ray.tune.trial import Trial\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py\", line 16, in \n",
" from ray.tune.durable_trainable import DurableTrainable\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py\", line 4, in \n",
" from ray.tune.trainable import Trainable, TrainableUtil\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py\", line 23, in \n",
" from ray.tune.logger import UnifiedLogger\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py\", line 15, in \n",
" from ray.tune.syncer import get_node_syncer\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py\", line 80, in \n",
" @dataclass\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 958, in dataclass\n",
" return wrap(_cls)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 950, in wrap\n",
" return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in _process_class\n",
" for name, type in cls_annotations.items()]\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in \n",
" for name, type in cls_annotations.items()]\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 659, in _get_field\n",
" if (_is_classvar(a_type, typing)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 550, in _is_classvar\n",
" return type(a_type) is typing._ClassVar\n",
"AttributeError: module 'typing' has no attribute '_ClassVar'\n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m 2020-11-04 18:40:48,721\tERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc.\n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m Traceback (most recent call last):\n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py\", line 493, in _load_actor_class_from_gcs\n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m actor_class = pickle.loads(pickled_class)\n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py\", line 2, in \n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m from ray.tune.tune import run_experiments, run\n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py\", line 5, in \n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m from ray.tune.analysis import ExperimentAnalysis\n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py\", line 1, in \n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis\n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py\", line 19, in \n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m from ray.tune.trial import Trial\n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py\", line 16, in \n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m from ray.tune.durable_trainable import DurableTrainable\n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py\", line 4, in \n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m from ray.tune.trainable import Trainable, TrainableUtil\n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py\", line 23, in \n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m from ray.tune.logger import UnifiedLogger\n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py\", line 15, in \n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m from ray.tune.syncer import get_node_syncer\n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py\", line 80, in \n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m @dataclass\n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 958, in dataclass\n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m return wrap(_cls)\n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 950, in wrap\n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)\n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in _process_class\n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m for name, type in cls_annotations.items()]\n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in \n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m for name, type in cls_annotations.items()]\n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 659, in _get_field\n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m if (_is_classvar(a_type, typing)\n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 550, in _is_classvar\n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m return type(a_type) is typing._ClassVar\n",
"\u001b[2m\u001b[36m(pid=4354)\u001b[0m AttributeError: module 'typing' has no attribute '_ClassVar'\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 11-04 18:40:48] ax.service.ax_client: Generated new trial 6 with parameters {'lr': 0.0, 'momentum': 0.15}.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m 2020-11-04 18:40:49,302\tERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc.\n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m Traceback (most recent call last):\n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py\", line 493, in _load_actor_class_from_gcs\n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m actor_class = pickle.loads(pickled_class)\n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py\", line 2, in \n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m from ray.tune.tune import run_experiments, run\n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py\", line 5, in \n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m from ray.tune.analysis import ExperimentAnalysis\n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py\", line 1, in \n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis\n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py\", line 19, in \n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m from ray.tune.trial import Trial\n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py\", line 16, in \n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m from ray.tune.durable_trainable import DurableTrainable\n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py\", line 4, in \n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m from ray.tune.trainable import Trainable, TrainableUtil\n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py\", line 23, in \n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m from ray.tune.logger import UnifiedLogger\n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py\", line 15, in \n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m from ray.tune.syncer import get_node_syncer\n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py\", line 80, in \n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m @dataclass\n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 958, in dataclass\n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m return wrap(_cls)\n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 950, in wrap\n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)\n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in _process_class\n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m for name, type in cls_annotations.items()]\n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in \n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m for name, type in cls_annotations.items()]\n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 659, in _get_field\n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m if (_is_classvar(a_type, typing)\n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 550, in _is_classvar\n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m return type(a_type) is typing._ClassVar\n",
"\u001b[2m\u001b[36m(pid=4365)\u001b[0m AttributeError: module 'typing' has no attribute '_ClassVar'\n",
"2020-11-04 18:40:49,313\tWARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID b7603b6c01000000. Traceback:\n",
"Traceback (most recent call last):\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py\", line 493, in _load_actor_class_from_gcs\n",
" actor_class = pickle.loads(pickled_class)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py\", line 2, in \n",
" from ray.tune.tune import run_experiments, run\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py\", line 5, in \n",
" from ray.tune.analysis import ExperimentAnalysis\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py\", line 1, in \n",
" from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py\", line 19, in \n",
" from ray.tune.trial import Trial\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py\", line 16, in \n",
" from ray.tune.durable_trainable import DurableTrainable\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py\", line 4, in \n",
" from ray.tune.trainable import Trainable, TrainableUtil\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py\", line 23, in \n",
" from ray.tune.logger import UnifiedLogger\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py\", line 15, in \n",
" from ray.tune.syncer import get_node_syncer\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py\", line 80, in \n",
" @dataclass\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 958, in dataclass\n",
" return wrap(_cls)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 950, in wrap\n",
" return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in _process_class\n",
" for name, type in cls_annotations.items()]\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in \n",
" for name, type in cls_annotations.items()]\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 659, in _get_field\n",
" if (_is_classvar(a_type, typing)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 550, in _is_classvar\n",
" return type(a_type) is typing._ClassVar\n",
"AttributeError: module 'typing' has no attribute '_ClassVar'\n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 11-04 18:40:49] ax.service.ax_client: Generated new trial 7 with parameters {'lr': 0.16, 'momentum': 0.7}.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2020-11-04 18:40:49,916\tWARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID be3cb80901000000. Traceback:\n",
"Traceback (most recent call last):\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py\", line 493, in _load_actor_class_from_gcs\n",
" actor_class = pickle.loads(pickled_class)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py\", line 2, in \n",
" from ray.tune.tune import run_experiments, run\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py\", line 5, in \n",
" from ray.tune.analysis import ExperimentAnalysis\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py\", line 1, in \n",
" from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py\", line 19, in \n",
" from ray.tune.trial import Trial\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py\", line 16, in \n",
" from ray.tune.durable_trainable import DurableTrainable\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py\", line 4, in \n",
" from ray.tune.trainable import Trainable, TrainableUtil\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py\", line 23, in \n",
" from ray.tune.logger import UnifiedLogger\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py\", line 15, in \n",
" from ray.tune.syncer import get_node_syncer\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py\", line 80, in \n",
" @dataclass\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 958, in dataclass\n",
" return wrap(_cls)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 950, in wrap\n",
" return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in _process_class\n",
" for name, type in cls_annotations.items()]\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in \n",
" for name, type in cls_annotations.items()]\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 659, in _get_field\n",
" if (_is_classvar(a_type, typing)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 550, in _is_classvar\n",
" return type(a_type) is typing._ClassVar\n",
"AttributeError: module 'typing' has no attribute '_ClassVar'\n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m 2020-11-04 18:40:49,911\tERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc.\n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m Traceback (most recent call last):\n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py\", line 493, in _load_actor_class_from_gcs\n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m actor_class = pickle.loads(pickled_class)\n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py\", line 2, in \n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m from ray.tune.tune import run_experiments, run\n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py\", line 5, in \n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m from ray.tune.analysis import ExperimentAnalysis\n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py\", line 1, in \n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis\n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py\", line 19, in \n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m from ray.tune.trial import Trial\n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py\", line 16, in \n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m from ray.tune.durable_trainable import DurableTrainable\n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py\", line 4, in \n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m from ray.tune.trainable import Trainable, TrainableUtil\n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py\", line 23, in \n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m from ray.tune.logger import UnifiedLogger\n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py\", line 15, in \n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m from ray.tune.syncer import get_node_syncer\n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py\", line 80, in \n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m @dataclass\n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 958, in dataclass\n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m return wrap(_cls)\n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 950, in wrap\n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)\n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in _process_class\n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m for name, type in cls_annotations.items()]\n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in \n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m for name, type in cls_annotations.items()]\n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 659, in _get_field\n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m if (_is_classvar(a_type, typing)\n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 550, in _is_classvar\n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m return type(a_type) is typing._ClassVar\n",
"\u001b[2m\u001b[36m(pid=4393)\u001b[0m AttributeError: module 'typing' has no attribute '_ClassVar'\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 11-04 18:40:50] ax.service.ax_client: Generated new trial 8 with parameters {'lr': 0.26, 'momentum': 0.67}.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2020-11-04 18:40:50,178\tWARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID af23404c01000000. Traceback:\n",
"Traceback (most recent call last):\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py\", line 493, in _load_actor_class_from_gcs\n",
" actor_class = pickle.loads(pickled_class)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py\", line 2, in \n",
" from ray.tune.tune import run_experiments, run\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py\", line 5, in \n",
" from ray.tune.analysis import ExperimentAnalysis\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py\", line 1, in \n",
" from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py\", line 19, in \n",
" from ray.tune.trial import Trial\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py\", line 16, in \n",
" from ray.tune.durable_trainable import DurableTrainable\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py\", line 4, in \n",
" from ray.tune.trainable import Trainable, TrainableUtil\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py\", line 23, in \n",
" from ray.tune.logger import UnifiedLogger\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py\", line 15, in \n",
" from ray.tune.syncer import get_node_syncer\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py\", line 80, in \n",
" @dataclass\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 958, in dataclass\n",
" return wrap(_cls)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 950, in wrap\n",
" return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in _process_class\n",
" for name, type in cls_annotations.items()]\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in \n",
" for name, type in cls_annotations.items()]\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 659, in _get_field\n",
" if (_is_classvar(a_type, typing)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 550, in _is_classvar\n",
" return type(a_type) is typing._ClassVar\n",
"AttributeError: module 'typing' has no attribute '_ClassVar'\n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m 2020-11-04 18:40:50,165\tERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc.\n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m Traceback (most recent call last):\n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py\", line 493, in _load_actor_class_from_gcs\n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m actor_class = pickle.loads(pickled_class)\n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py\", line 2, in \n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m from ray.tune.tune import run_experiments, run\n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py\", line 5, in \n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m from ray.tune.analysis import ExperimentAnalysis\n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py\", line 1, in \n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis\n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py\", line 19, in \n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m from ray.tune.trial import Trial\n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py\", line 16, in \n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m from ray.tune.durable_trainable import DurableTrainable\n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py\", line 4, in \n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m from ray.tune.trainable import Trainable, TrainableUtil\n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py\", line 23, in \n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m from ray.tune.logger import UnifiedLogger\n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py\", line 15, in \n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m from ray.tune.syncer import get_node_syncer\n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py\", line 80, in \n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m @dataclass\n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 958, in dataclass\n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m return wrap(_cls)\n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 950, in wrap\n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)\n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in _process_class\n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m for name, type in cls_annotations.items()]\n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in \n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m for name, type in cls_annotations.items()]\n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 659, in _get_field\n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m if (_is_classvar(a_type, typing)\n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 550, in _is_classvar\n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m return type(a_type) is typing._ClassVar\n",
"\u001b[2m\u001b[36m(pid=4408)\u001b[0m AttributeError: module 'typing' has no attribute '_ClassVar'\n",
"[INFO 11-04 18:40:50] ax.service.ax_client: Generated new trial 9 with parameters {'lr': 0.0, 'momentum': 0.66}.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2020-11-04 18:40:50,820\tWARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 7d26912601000000. Traceback:\n",
"Traceback (most recent call last):\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py\", line 493, in _load_actor_class_from_gcs\n",
" actor_class = pickle.loads(pickled_class)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py\", line 2, in \n",
" from ray.tune.tune import run_experiments, run\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py\", line 5, in \n",
" from ray.tune.analysis import ExperimentAnalysis\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py\", line 1, in \n",
" from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py\", line 19, in \n",
" from ray.tune.trial import Trial\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py\", line 16, in \n",
" from ray.tune.durable_trainable import DurableTrainable\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py\", line 4, in \n",
" from ray.tune.trainable import Trainable, TrainableUtil\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py\", line 23, in \n",
" from ray.tune.logger import UnifiedLogger\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py\", line 15, in \n",
" from ray.tune.syncer import get_node_syncer\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py\", line 80, in \n",
" @dataclass\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 958, in dataclass\n",
" return wrap(_cls)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 950, in wrap\n",
" return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in _process_class\n",
" for name, type in cls_annotations.items()]\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in \n",
" for name, type in cls_annotations.items()]\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 659, in _get_field\n",
" if (_is_classvar(a_type, typing)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 550, in _is_classvar\n",
" return type(a_type) is typing._ClassVar\n",
"AttributeError: module 'typing' has no attribute '_ClassVar'\n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m 2020-11-04 18:40:50,808\tERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc.\n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m Traceback (most recent call last):\n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py\", line 493, in _load_actor_class_from_gcs\n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m actor_class = pickle.loads(pickled_class)\n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py\", line 2, in \n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m from ray.tune.tune import run_experiments, run\n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py\", line 5, in \n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m from ray.tune.analysis import ExperimentAnalysis\n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py\", line 1, in \n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis\n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py\", line 19, in \n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m from ray.tune.trial import Trial\n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py\", line 16, in \n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m from ray.tune.durable_trainable import DurableTrainable\n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py\", line 4, in \n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m from ray.tune.trainable import Trainable, TrainableUtil\n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py\", line 23, in \n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m from ray.tune.logger import UnifiedLogger\n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py\", line 15, in \n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m from ray.tune.syncer import get_node_syncer\n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py\", line 80, in \n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m @dataclass\n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 958, in dataclass\n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m return wrap(_cls)\n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 950, in wrap\n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)\n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in _process_class\n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m for name, type in cls_annotations.items()]\n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in \n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m for name, type in cls_annotations.items()]\n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 659, in _get_field\n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m if (_is_classvar(a_type, typing)\n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 550, in _is_classvar\n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m return type(a_type) is typing._ClassVar\n",
"\u001b[2m\u001b[36m(pid=4427)\u001b[0m AttributeError: module 'typing' has no attribute '_ClassVar'\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 11-04 18:40:51] ax.service.ax_client: Generated new trial 10 with parameters {'lr': 0.0, 'momentum': 0.28}.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2020-11-04 18:40:51,327\tWARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 335d6f0801000000. Traceback:\n",
"Traceback (most recent call last):\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py\", line 493, in _load_actor_class_from_gcs\n",
" actor_class = pickle.loads(pickled_class)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py\", line 2, in \n",
" from ray.tune.tune import run_experiments, run\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py\", line 5, in \n",
" from ray.tune.analysis import ExperimentAnalysis\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py\", line 1, in \n",
" from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py\", line 19, in \n",
" from ray.tune.trial import Trial\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py\", line 16, in \n",
" from ray.tune.durable_trainable import DurableTrainable\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py\", line 4, in \n",
" from ray.tune.trainable import Trainable, TrainableUtil\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py\", line 23, in \n",
" from ray.tune.logger import UnifiedLogger\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py\", line 15, in \n",
" from ray.tune.syncer import get_node_syncer\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py\", line 80, in \n",
" @dataclass\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 958, in dataclass\n",
" return wrap(_cls)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 950, in wrap\n",
" return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in _process_class\n",
" for name, type in cls_annotations.items()]\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in \n",
" for name, type in cls_annotations.items()]\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 659, in _get_field\n",
" if (_is_classvar(a_type, typing)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 550, in _is_classvar\n",
" return type(a_type) is typing._ClassVar\n",
"AttributeError: module 'typing' has no attribute '_ClassVar'\n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m 2020-11-04 18:40:51,315\tERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc.\n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m Traceback (most recent call last):\n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py\", line 493, in _load_actor_class_from_gcs\n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m actor_class = pickle.loads(pickled_class)\n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py\", line 2, in \n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m from ray.tune.tune import run_experiments, run\n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py\", line 5, in \n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m from ray.tune.analysis import ExperimentAnalysis\n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py\", line 1, in \n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis\n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py\", line 19, in \n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m from ray.tune.trial import Trial\n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py\", line 16, in \n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m from ray.tune.durable_trainable import DurableTrainable\n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py\", line 4, in \n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m from ray.tune.trainable import Trainable, TrainableUtil\n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py\", line 23, in \n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m from ray.tune.logger import UnifiedLogger\n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py\", line 15, in \n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m from ray.tune.syncer import get_node_syncer\n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py\", line 80, in \n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m @dataclass\n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 958, in dataclass\n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m return wrap(_cls)\n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 950, in wrap\n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)\n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in _process_class\n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m for name, type in cls_annotations.items()]\n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in \n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m for name, type in cls_annotations.items()]\n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 659, in _get_field\n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m if (_is_classvar(a_type, typing)\n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 550, in _is_classvar\n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m return type(a_type) is typing._ClassVar\n",
"\u001b[2m\u001b[36m(pid=4448)\u001b[0m AttributeError: module 'typing' has no attribute '_ClassVar'\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 11-04 18:40:51] ax.service.ax_client: Generated new trial 11 with parameters {'lr': 0.0, 'momentum': 0.83}.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2020-11-04 18:40:51,701\tWARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID ad5fa5cb01000000. Traceback:\n",
"Traceback (most recent call last):\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py\", line 493, in _load_actor_class_from_gcs\n",
" actor_class = pickle.loads(pickled_class)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py\", line 2, in \n",
" from ray.tune.tune import run_experiments, run\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py\", line 5, in \n",
" from ray.tune.analysis import ExperimentAnalysis\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py\", line 1, in \n",
" from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py\", line 19, in \n",
" from ray.tune.trial import Trial\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py\", line 16, in \n",
" from ray.tune.durable_trainable import DurableTrainable\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py\", line 4, in \n",
" from ray.tune.trainable import Trainable, TrainableUtil\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py\", line 23, in \n",
" from ray.tune.logger import UnifiedLogger\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py\", line 15, in \n",
" from ray.tune.syncer import get_node_syncer\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py\", line 80, in \n",
" @dataclass\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 958, in dataclass\n",
" return wrap(_cls)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 950, in wrap\n",
" return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in _process_class\n",
" for name, type in cls_annotations.items()]\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in \n",
" for name, type in cls_annotations.items()]\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 659, in _get_field\n",
" if (_is_classvar(a_type, typing)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 550, in _is_classvar\n",
" return type(a_type) is typing._ClassVar\n",
"AttributeError: module 'typing' has no attribute '_ClassVar'\n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m 2020-11-04 18:40:51,695\tERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc.\n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m Traceback (most recent call last):\n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py\", line 493, in _load_actor_class_from_gcs\n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m actor_class = pickle.loads(pickled_class)\n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py\", line 2, in \n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m from ray.tune.tune import run_experiments, run\n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py\", line 5, in \n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m from ray.tune.analysis import ExperimentAnalysis\n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py\", line 1, in \n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis\n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py\", line 19, in \n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m from ray.tune.trial import Trial\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py\", line 16, in \n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m from ray.tune.durable_trainable import DurableTrainable\n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py\", line 4, in \n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m from ray.tune.trainable import Trainable, TrainableUtil\n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py\", line 23, in \n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m from ray.tune.logger import UnifiedLogger\n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py\", line 15, in \n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m from ray.tune.syncer import get_node_syncer\n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py\", line 80, in \n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m @dataclass\n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 958, in dataclass\n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m return wrap(_cls)\n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 950, in wrap\n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)\n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in _process_class\n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m for name, type in cls_annotations.items()]\n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in \n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m for name, type in cls_annotations.items()]\n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 659, in _get_field\n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m if (_is_classvar(a_type, typing)\n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 550, in _is_classvar\n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m return type(a_type) is typing._ClassVar\n",
"\u001b[2m\u001b[36m(pid=4465)\u001b[0m AttributeError: module 'typing' has no attribute '_ClassVar'\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 11-04 18:40:51] ax.service.ax_client: Generated new trial 12 with parameters {'lr': 0.0, 'momentum': 0.93}.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2020-11-04 18:40:52,452\tWARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 35d7a3b901000000. Traceback:\n",
"Traceback (most recent call last):\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py\", line 493, in _load_actor_class_from_gcs\n",
" actor_class = pickle.loads(pickled_class)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py\", line 2, in \n",
" from ray.tune.tune import run_experiments, run\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py\", line 5, in \n",
" from ray.tune.analysis import ExperimentAnalysis\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py\", line 1, in \n",
" from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py\", line 19, in \n",
" from ray.tune.trial import Trial\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py\", line 16, in \n",
" from ray.tune.durable_trainable import DurableTrainable\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py\", line 4, in \n",
" from ray.tune.trainable import Trainable, TrainableUtil\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py\", line 23, in \n",
" from ray.tune.logger import UnifiedLogger\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py\", line 15, in \n",
" from ray.tune.syncer import get_node_syncer\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py\", line 80, in \n",
" @dataclass\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 958, in dataclass\n",
" return wrap(_cls)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 950, in wrap\n",
" return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in _process_class\n",
" for name, type in cls_annotations.items()]\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in \n",
" for name, type in cls_annotations.items()]\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 659, in _get_field\n",
" if (_is_classvar(a_type, typing)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 550, in _is_classvar\n",
" return type(a_type) is typing._ClassVar\n",
"AttributeError: module 'typing' has no attribute '_ClassVar'\n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m 2020-11-04 18:40:52,436\tERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc.\n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m Traceback (most recent call last):\n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py\", line 493, in _load_actor_class_from_gcs\n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m actor_class = pickle.loads(pickled_class)\n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py\", line 2, in \n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m from ray.tune.tune import run_experiments, run\n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py\", line 5, in \n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m from ray.tune.analysis import ExperimentAnalysis\n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py\", line 1, in \n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis\n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py\", line 19, in \n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m from ray.tune.trial import Trial\n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py\", line 16, in \n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m from ray.tune.durable_trainable import DurableTrainable\n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py\", line 4, in \n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m from ray.tune.trainable import Trainable, TrainableUtil\n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py\", line 23, in \n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m from ray.tune.logger import UnifiedLogger\n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py\", line 15, in \n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m from ray.tune.syncer import get_node_syncer\n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py\", line 80, in \n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m @dataclass\n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 958, in dataclass\n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m return wrap(_cls)\n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 950, in wrap\n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)\n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in _process_class\n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m for name, type in cls_annotations.items()]\n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in \n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m for name, type in cls_annotations.items()]\n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 659, in _get_field\n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m if (_is_classvar(a_type, typing)\n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 550, in _is_classvar\n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m return type(a_type) is typing._ClassVar\n",
"\u001b[2m\u001b[36m(pid=4484)\u001b[0m AttributeError: module 'typing' has no attribute '_ClassVar'\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 11-04 18:40:52] ax.service.ax_client: Generated new trial 13 with parameters {'lr': 0.04, 'momentum': 0.26}.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2020-11-04 18:40:52,830\tWARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 91fdfca801000000. Traceback:\n",
"Traceback (most recent call last):\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py\", line 493, in _load_actor_class_from_gcs\n",
" actor_class = pickle.loads(pickled_class)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py\", line 2, in \n",
" from ray.tune.tune import run_experiments, run\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py\", line 5, in \n",
" from ray.tune.analysis import ExperimentAnalysis\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py\", line 1, in \n",
" from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py\", line 19, in \n",
" from ray.tune.trial import Trial\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py\", line 16, in \n",
" from ray.tune.durable_trainable import DurableTrainable\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py\", line 4, in \n",
" from ray.tune.trainable import Trainable, TrainableUtil\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py\", line 23, in \n",
" from ray.tune.logger import UnifiedLogger\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py\", line 15, in \n",
" from ray.tune.syncer import get_node_syncer\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py\", line 80, in \n",
" @dataclass\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 958, in dataclass\n",
" return wrap(_cls)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 950, in wrap\n",
" return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in _process_class\n",
" for name, type in cls_annotations.items()]\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in \n",
" for name, type in cls_annotations.items()]\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 659, in _get_field\n",
" if (_is_classvar(a_type, typing)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 550, in _is_classvar\n",
" return type(a_type) is typing._ClassVar\n",
"AttributeError: module 'typing' has no attribute '_ClassVar'\n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m 2020-11-04 18:40:52,822\tERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc.\n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m Traceback (most recent call last):\n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py\", line 493, in _load_actor_class_from_gcs\n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m actor_class = pickle.loads(pickled_class)\n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py\", line 2, in \n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m from ray.tune.tune import run_experiments, run\n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py\", line 5, in \n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m from ray.tune.analysis import ExperimentAnalysis\n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py\", line 1, in \n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis\n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py\", line 19, in \n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m from ray.tune.trial import Trial\n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py\", line 16, in \n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m from ray.tune.durable_trainable import DurableTrainable\n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py\", line 4, in \n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m from ray.tune.trainable import Trainable, TrainableUtil\n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py\", line 23, in \n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m from ray.tune.logger import UnifiedLogger\n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py\", line 15, in \n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m from ray.tune.syncer import get_node_syncer\n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py\", line 80, in \n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m @dataclass\n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 958, in dataclass\n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m return wrap(_cls)\n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 950, in wrap\n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)\n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in _process_class\n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m for name, type in cls_annotations.items()]\n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in \n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m for name, type in cls_annotations.items()]\n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 659, in _get_field\n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m if (_is_classvar(a_type, typing)\n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 550, in _is_classvar\n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m return type(a_type) is typing._ClassVar\n",
"\u001b[2m\u001b[36m(pid=4502)\u001b[0m AttributeError: module 'typing' has no attribute '_ClassVar'\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 11-04 18:40:53] ax.service.ax_client: Generated new trial 14 with parameters {'lr': 0.01, 'momentum': 0.8}.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2020-11-04 18:40:53,078\tWARNING worker.py:1072 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID b9c4a7c601000000. Traceback:\n",
"Traceback (most recent call last):\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py\", line 493, in _load_actor_class_from_gcs\n",
" actor_class = pickle.loads(pickled_class)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py\", line 2, in \n",
" from ray.tune.tune import run_experiments, run\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py\", line 5, in \n",
" from ray.tune.analysis import ExperimentAnalysis\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py\", line 1, in \n",
" from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py\", line 19, in \n",
" from ray.tune.trial import Trial\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py\", line 16, in \n",
" from ray.tune.durable_trainable import DurableTrainable\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/durable_trainable.py\", line 4, in \n",
" from ray.tune.trainable import Trainable, TrainableUtil\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trainable.py\", line 23, in \n",
" from ray.tune.logger import UnifiedLogger\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/logger.py\", line 15, in \n",
" from ray.tune.syncer import get_node_syncer\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/syncer.py\", line 80, in \n",
" @dataclass\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 958, in dataclass\n",
" return wrap(_cls)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 950, in wrap\n",
" return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in _process_class\n",
" for name, type in cls_annotations.items()]\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 801, in \n",
" for name, type in cls_annotations.items()]\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 659, in _get_field\n",
" if (_is_classvar(a_type, typing)\n",
" File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/dataclasses.py\", line 550, in _is_classvar\n",
" return type(a_type) is typing._ClassVar\n",
"AttributeError: module 'typing' has no attribute '_ClassVar'\n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[2m\u001b[36m(pid=4519)\u001b[0m 2020-11-04 18:40:53,067\tERROR function_manager.py:495 -- Failed to load actor class ImplicitFunc.\n",
"\u001b[2m\u001b[36m(pid=4519)\u001b[0m Traceback (most recent call last):\n",
"\u001b[2m\u001b[36m(pid=4519)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/function_manager.py\", line 493, in _load_actor_class_from_gcs\n",
"\u001b[2m\u001b[36m(pid=4519)\u001b[0m actor_class = pickle.loads(pickled_class)\n",
"\u001b[2m\u001b[36m(pid=4519)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/__init__.py\", line 2, in \n",
"\u001b[2m\u001b[36m(pid=4519)\u001b[0m from ray.tune.tune import run_experiments, run\n",
"\u001b[2m\u001b[36m(pid=4519)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/tune.py\", line 5, in \n",
"\u001b[2m\u001b[36m(pid=4519)\u001b[0m from ray.tune.analysis import ExperimentAnalysis\n",
"\u001b[2m\u001b[36m(pid=4519)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/__init__.py\", line 1, in \n",
"\u001b[2m\u001b[36m(pid=4519)\u001b[0m from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis\n",
"\u001b[2m\u001b[36m(pid=4519)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py\", line 19, in \n",
"\u001b[2m\u001b[36m(pid=4519)\u001b[0m from ray.tune.trial import Trial\n",
"\u001b[2m\u001b[36m(pid=4519)\u001b[0m File \"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ray/tune/trial.py\", line 16, in