Skip to main content
Version: Next
Open in GitHubRun in Google Colab

Using external methods for candidate generation in Ax

Out of the box, Ax offers many options for candidate generation, most of which utilize Bayesian optimization algorithms built using BoTorch. Users can allow Ax to select which methods from BoTorch will be used for optimization automatically (optionally guiding the choice by calling configure_generation_strategy) and can also direct Ax to use specific BoTorch components directly via the Modular Botorch framework. Users that want to leverage Ax for experiment state management and orchestration via the Client and other features (e.g., early stopping, analyses, storage), while relying on other methods not implemented in BoTorch for candidate generation can implement their own ExternalGenerationNode.

A GenerationNode is a building block of a GenerationStrategy. They can be combined together utilize different methods for generating candidates at different stages of an experiment. ExternalGenerationNode exposes a lightweight interface to allow the users to easily integrate their methods into Ax, and use them as standalone or with other GenerationNodes in a GenerationStrategy.

In this tutorial, we will implement a simple generation node using RandomForestRegressor from sklearn, and combine it with Sobol (for initialization) to optimize the Hartmann6 problem.

NOTE: This generation node using RandomForestRegressor is a naive implementation and for demonstration purposes only; it is not representative of how more sophisticated random forest Bayesian optimization algorithms perform. We do not recommend using this strategy as it typically does not perform well compared to Ax's default algorithms due to its overly greedy behavior.

import time
from typing import Any

import numpy as np

from ax.adapter.registry import Generators
from ax.api.client import Client
from ax.api.configs import RangeParameterConfig
from ax.analysis.plotly.arm_effects import ArmEffectsPlot
from ax.analysis.summary import Summary
from ax.core.base_trial import TrialStatus
from ax.core.data import Data
from ax.core.experiment import Experiment
from ax.core.parameter import RangeParameter
from ax.core.types import TParameterization
from ax.generation_strategy.external_generation_node import ExternalGenerationNode
from ax.generation_strategy.generation_node import GenerationNode
from ax.generation_strategy.generation_strategy import GenerationStrategy
from ax.generation_strategy.generator_spec import GeneratorSpec
from ax.generation_strategy.transition_criterion import MaxTrials

from sklearn.ensemble import RandomForestRegressor
class RandomForestGenerationNode(ExternalGenerationNode):
    """A generation node that uses the RandomForestRegressor
    from sklearn to predict candidate performance and picks the
    next point as the random sample that has the best prediction.

    To leverage external methods for candidate generation, the user must
    create a subclass that implements ``update_generator_state`` and
    ``get_next_candidate`` methods. This can then be provided
    as a node into a ``GenerationStrategy``, either as standalone or as
    part of a larger generation strategy with other generation nodes,
    e.g., with a Sobol node for initialization.
    """

    def __init__(self, num_samples: int, regressor_options: dict[str, Any]) -> None:
        """Initialize the generation node.

        Args:
            regressor_options: Options to pass to the random forest regressor.
            num_samples: Number of random samples from the search space
                used during candidate generation. The sample with the best
                prediction is recommended as the next candidate.
        """
        t_init_start = time.monotonic()
        super().__init__(node_name="RandomForest")
        self.num_samples: int = num_samples
        self.regressor: RandomForestRegressor = RandomForestRegressor(
            **regressor_options
        )
        # We will set these later when updating the state.
        # Alternatively, we could have required experiment as an input
        # and extracted them here.
        self.parameters: list[RangeParameter] | None = None
        self.minimize: bool | None = None
        # Recording time spent in initializing the generator. This is
        # used to compute the time spent in candidate generation.
        self.fit_time_since_gen: float = time.monotonic() - t_init_start

    def update_generator_state(self, experiment: Experiment, data: Data) -> None:
        """A method used to update the state of the generator. This includes any
        models, predictors or any other custom state used by the generation node.
        This method will be called with the up-to-date experiment and data before
        ``get_next_candidate`` is called to generate the next trial(s). Note
        that ``get_next_candidate`` may be called multiple times (to generate
        multiple candidates) after a call to  ``update_generator_state``.

        For this example, we will train the regressor using the latest data from
        the experiment.

        Args:
            experiment: The ``Experiment`` object representing the current state of the
                experiment. The key properties includes ``trials``, ``search_space``,
                and ``optimization_config``. The data is provided as a separate arg.
            data: The data / metrics collected on the experiment so far.
        """
        search_space = experiment.search_space
        parameter_names = list(search_space.parameters.keys())
        metric_names = list(experiment.optimization_config.metrics.keys())
        if any(
            not isinstance(p, RangeParameter) for p in search_space.parameters.values()
        ):
            raise NotImplementedError(
                "This example only supports RangeParameters in the search space."
            )
        if search_space.parameter_constraints:
            raise NotImplementedError(
                "This example does not support parameter constraints."
            )
        if len(metric_names) != 1:
            raise NotImplementedError(
                "This example only supports single-objective optimization."
            )
        # Get the data for the completed trials.
        num_completed_trials = len(experiment.trials_by_status[TrialStatus.COMPLETED])
        x = np.zeros([num_completed_trials, len(parameter_names)])
        y = np.zeros([num_completed_trials, 1])
        for t_idx, trial in experiment.trials.items():
            if trial.status == "COMPLETED":
                trial_parameters = trial.arm.parameters
                x[t_idx, :] = np.array([trial_parameters[p] for p in parameter_names])
                trial_df = data.df[data.df["trial_index"] == t_idx]
                y[t_idx, 0] = trial_df[trial_df["metric_name"] == metric_names[0]][
                    "mean"
                ].item()

        # Train the regressor.
        self.regressor.fit(x, y)
        # Update the attributes not set in __init__.
        self.parameters = search_space.parameters
        self.minimize = experiment.optimization_config.objective.minimize

    def get_next_candidate(
        self, pending_parameters: list[TParameterization]
    ) -> TParameterization:
        """Get the parameters for the next candidate configuration to evaluate.

        We will draw ``self.num_samples`` random samples from the search space
        and predict the objective value for each sample. We will then return
        the sample with the best predicted value.

        Args:
            pending_parameters: A list of parameters of the candidates pending
                evaluation. This is often used to avoid generating duplicate candidates.
                We ignore this here for simplicity.

        Returns:
            A dictionary mapping parameter names to parameter values for the next
            candidate suggested by the method.
        """
        bounds = np.array([[p.lower, p.upper] for p in self.parameters.values()])
        unit_samples = np.random.random_sample([self.num_samples, len(bounds)])
        samples = bounds[:, 0] + (bounds[:, 1] - bounds[:, 0]) * unit_samples
        # Predict the objective value for each sample.
        y_pred = self.regressor.predict(samples)
        # Find the best sample.
        best_idx = np.argmin(y_pred) if self.minimize else np.argmax(y_pred)
        best_sample = samples[best_idx, :]
        # Convert the sample to a parameterization.
        candidate = {
            p_name: best_sample[i].item()
            for i, p_name in enumerate(self.parameters.keys())
        }
        return candidate

Construct the GenerationStrategy​

We will use Sobol for the first 25 trials and defer to random forest for the rest.

generation_strategy = GenerationStrategy(
    name="Sobol+RandomForest",
    nodes=[
        GenerationNode(
            node_name="Sobol",
            generator_specs=[GeneratorSpec(Generators.SOBOL)],
            transition_criteria=[
                MaxTrials(
                    # This specifies the maximum number of trials to generate from this node,
                    # and the next node in the strategy.
                    threshold=25,
                    block_transition_if_unmet=True,
                    transition_to="RandomForest",
                )
            ],
        ),
        RandomForestGenerationNode(num_samples=128, regressor_options={}),
    ],
)

Run a simple experiment using the Client​

More details on how to use the Client can be found in the following tutorial.

# Define the Hartmann6 function which we will minimize.
def hartmann6(x1, x2, x3, x4, x5, x6):
    alpha = np.array([1.0, 1.2, 3.0, 3.2])
    A = np.array(
        [
            [10, 3, 17, 3.5, 1.7, 8],
            [0.05, 10, 17, 0.1, 8, 14],
            [3, 3.5, 1.7, 10, 17, 8],
            [17, 8, 0.05, 10, 0.1, 14],
        ]
    )
    P = 10**-4 * np.array(
        [
            [1312, 1696, 5569, 124, 8283, 5886],
            [2329, 4135, 8307, 3736, 1004, 9991],
            [2348, 1451, 3522, 2883, 3047, 6650],
            [4047, 8828, 8732, 5743, 1091, 381],
        ]
    )

    outer = 0.0
    for i in range(4):
        inner = 0.0
        for j, x in enumerate([x1, x2, x3, x4, x5, x6]):
            inner += A[i, j] * (x - P[i, j]) ** 2
        outer += alpha[i] * np.exp(-inner)
    return -outer

Configure the experiment and set is GenerationStrategy to be our Sobol+RandomForest strategy defined previously.

client = Client()

client.configure_experiment(
    parameters=[
        RangeParameterConfig(name=f"x{i}", parameter_type="float", bounds=(0, 1))
        for i in range(1, 7)
    ]
)
client.configure_optimization(objective="-1 * hartmann6")

client.set_generation_strategy(generation_strategy=generation_strategy)

Run the optimization loop as you would conduct any other Ax experiment. Under the hood Ax will dispatch to your custom ExternalGenerationNode when generating new candidate trials.

for _ in range(50):  # Run 50 rounds of trials (25 Sobol points, 25 with the RandomForest node)
    trials = client.get_next_trials(max_trials=1)

    for trial_index, parameters in trials.items():
        x1 = parameters["x1"]
        x2 = parameters["x2"]
        x3 = parameters["x3"]
        x4 = parameters["x4"]
        x5 = parameters["x5"]
        x6 = parameters["x6"]

        # Set raw_data as a dictionary with metric names as keys and results as values
        raw_data = {"hartmann6": hartmann6(x1, x2, x3, x4, x5, x6)}

        # Complete the trial with the result
        client.complete_trial(trial_index=trial_index, raw_data=raw_data)
Output:
[INFO 09-21 05:09:31] ax.api.client: Generated new trial 0 with parameters {'x1': 0.331417, 'x2': 0.440283, 'x3': 0.726876, 'x4': 0.133118, 'x5': 0.686948, 'x6': 0.357847} using GenerationNode Sobol.
[INFO 09-21 05:09:31] ax.api.client: Trial 0 marked COMPLETED.
[INFO 09-21 05:09:31] ax.api.client: Generated new trial 1 with parameters {'x1': 0.884509, 'x2': 0.714793, 'x3': 0.310586, 'x4': 0.909164, 'x5': 0.226006, 'x6': 0.971387} using GenerationNode Sobol.
[INFO 09-21 05:09:31] ax.api.client: Trial 1 marked COMPLETED.
[INFO 09-21 05:09:31] ax.api.client: Generated new trial 2 with parameters {'x1': 0.643623, 'x2': 0.011294, 'x3': 0.781757, 'x4': 0.262352, 'x5': 0.355122, 'x6': 0.577043} using GenerationNode Sobol.
[INFO 09-21 05:09:31] ax.api.client: Trial 2 marked COMPLETED.
[INFO 09-21 05:09:31] ax.api.client: Generated new trial 3 with parameters {'x1': 0.072219, 'x2': 0.768293, 'x3': 0.241077, 'x4': 0.537163, 'x5': 0.800431, 'x6': 0.189669} using GenerationNode Sobol.
[INFO 09-21 05:09:31] ax.api.client: Trial 3 marked COMPLETED.
[INFO 09-21 05:09:31] ax.api.client: Generated new trial 4 with parameters {'x1': 0.179923, 'x2': 0.134754, 'x3': 0.453114, 'x4': 0.655867, 'x5': 0.969291, 'x6': 0.441183} using GenerationNode Sobol.
[INFO 09-21 05:09:31] ax.api.client: Trial 4 marked COMPLETED.
[INFO 09-21 05:09:31] ax.api.client: Generated new trial 5 with parameters {'x1': 0.596556, 'x2': 0.891446, 'x3': 0.525044, 'x4': 0.426942, 'x5': 0.430225, 'x6': 0.827246} using GenerationNode Sobol.
[INFO 09-21 05:09:31] ax.api.client: Trial 5 marked COMPLETED.
[INFO 09-21 05:09:31] ax.api.client: Generated new trial 6 with parameters {'x1': 0.867624, 'x2': 0.313191, 'x3': 0.007002, 'x4': 0.776788, 'x5': 0.051056, 'x6': 0.72198} using GenerationNode Sobol.
[INFO 09-21 05:09:31] ax.api.client: Trial 6 marked COMPLETED.
[INFO 09-21 05:09:31] ax.api.client: Generated new trial 7 with parameters {'x1': 0.408755, 'x2': 0.587521, 'x3': 0.954542, 'x4': 0.048605, 'x5': 0.605739, 'x6': 0.108956} using GenerationNode Sobol.
[INFO 09-21 05:09:31] ax.api.client: Trial 7 marked COMPLETED.
[INFO 09-21 05:09:31] ax.api.client: Generated new trial 8 with parameters {'x1': 0.45363, 'x2': 0.07053, 'x3': 0.097225, 'x4': 0.49651, 'x5': 0.153843, 'x6': 0.171054} using GenerationNode Sobol.
[INFO 09-21 05:09:31] ax.api.client: Trial 8 marked COMPLETED.
[INFO 09-21 05:09:31] ax.api.client: Generated new trial 9 with parameters {'x1': 0.760156, 'x2': 0.829174, 'x3': 0.927578, 'x4': 0.709743, 'x5': 0.692909, 'x6': 0.535111} using GenerationNode Sobol.
[INFO 09-21 05:09:31] ax.api.client: Trial 9 marked COMPLETED.
[INFO 09-21 05:09:31] ax.api.client: Generated new trial 10 with parameters {'x1': 0.516401, 'x2': 0.376407, 'x3': 0.417329, 'x4': 0.11777, 'x5': 0.821544, 'x6': 0.890271} using GenerationNode Sobol.
[INFO 09-21 05:09:31] ax.api.client: Trial 10 marked COMPLETED.
[INFO 09-21 05:09:31] ax.api.client: Generated new trial 11 with parameters {'x1': 0.197424, 'x2': 0.648785, 'x3': 0.622072, 'x4': 0.830273, 'x5': 0.266861, 'x6': 0.253418} using GenerationNode Sobol.
[INFO 09-21 05:09:31] ax.api.client: Trial 11 marked COMPLETED.
[INFO 09-21 05:09:31] ax.api.client: Generated new trial 12 with parameters {'x1': 0.058499, 'x2': 0.253039, 'x3': 0.816534, 'x4': 0.980128, 'x5': 0.439858, 'x6': 0.00245} using GenerationNode Sobol.
[INFO 09-21 05:09:32] ax.api.client: Trial 12 marked COMPLETED.
[INFO 09-21 05:09:32] ax.api.client: Generated new trial 13 with parameters {'x1': 0.719746, 'x2': 0.525601, 'x3': 0.143042, 'x4': 0.188528, 'x5': 0.900799, 'x6': 0.639034} using GenerationNode Sobol.
[INFO 09-21 05:09:32] ax.api.client: Trial 13 marked COMPLETED.
[INFO 09-21 05:09:32] ax.api.client: Generated new trial 14 with parameters {'x1': 0.995758, 'x2': 0.197591, 'x3': 0.637662, 'x4': 0.608717, 'x5': 0.522095, 'x6': 0.783241} using GenerationNode Sobol.
[INFO 09-21 05:09:32] ax.api.client: Trial 14 marked COMPLETED.
[INFO 09-21 05:09:32] ax.api.client: Generated new trial 15 with parameters {'x1': 0.28251, 'x2': 0.956537, 'x3': 0.338559, 'x4': 0.318331, 'x5': 0.076787, 'x6': 0.420735} using GenerationNode Sobol.
[INFO 09-21 05:09:32] ax.api.client: Trial 15 marked COMPLETED.
[INFO 09-21 05:09:32] ax.api.client: Generated new trial 16 with parameters {'x1': 0.278533, 'x2': 0.223593, 'x3': 0.893497, 'x4': 0.800621, 'x5': 0.772771, 'x6': 0.676555} using GenerationNode Sobol.
[INFO 09-21 05:09:32] ax.api.client: Trial 16 marked COMPLETED.
[INFO 09-21 05:09:32] ax.api.client: Generated new trial 17 with parameters {'x1': 0.945151, 'x2': 0.997919, 'x3': 0.06705, 'x4': 0.024895, 'x5': 0.327454, 'x6': 0.054772} using GenerationNode Sobol.
[INFO 09-21 05:09:32] ax.api.client: Trial 17 marked COMPLETED.
[INFO 09-21 05:09:32] ax.api.client: Generated new trial 18 with parameters {'x1': 0.71628, 'x2': 0.294664, 'x3': 0.588036, 'x4': 0.678834, 'x5': 0.191022, 'x6': 0.394779} using GenerationNode Sobol.
[INFO 09-21 05:09:32] ax.api.client: Trial 18 marked COMPLETED.
[INFO 09-21 05:09:32] ax.api.client: Generated new trial 19 with parameters {'x1': 0.007426, 'x2': 0.551359, 'x3': 0.387199, 'x4': 0.403854, 'x5': 0.651956, 'x6': 0.77404} using GenerationNode Sobol.
[INFO 09-21 05:09:32] ax.api.client: Trial 19 marked COMPLETED.
[INFO 09-21 05:09:32] ax.api.client: Generated new trial 20 with parameters {'x1': 0.240558, 'x2': 0.421692, 'x3': 0.177123, 'x4': 0.285272, 'x5': 0.570999, 'x6': 0.524781} using GenerationNode Sobol.
[INFO 09-21 05:09:32] ax.api.client: Trial 20 marked COMPLETED.
[INFO 09-21 05:09:32] ax.api.client: Generated new trial 21 with parameters {'x1': 0.543666, 'x2': 0.678695, 'x3': 0.84671, 'x4': 0.514365, 'x5': 0.016308, 'x6': 0.146228} using GenerationNode Sobol.
[INFO 09-21 05:09:32] ax.api.client: Trial 21 marked COMPLETED.
[INFO 09-21 05:09:32] ax.api.client: Generated new trial 22 with parameters {'x1': 0.802841, 'x2': 0.100196, 'x3': 0.372594, 'x4': 0.157149, 'x5': 0.402809, 'x6': 0.306564} using GenerationNode Sobol.
[INFO 09-21 05:09:32] ax.api.client: Trial 22 marked COMPLETED.
[INFO 09-21 05:09:32] ax.api.client: Generated new trial 23 with parameters {'x1': 0.481422, 'x2': 0.874703, 'x3': 0.667791, 'x4': 0.885011, 'x5': 0.941868, 'x6': 0.926968} using GenerationNode Sobol.
[INFO 09-21 05:09:32] ax.api.client: Trial 23 marked COMPLETED.
[INFO 09-21 05:09:32] ax.api.client: Generated new trial 24 with parameters {'x1': 0.405202, 'x2': 0.357506, 'x3': 0.274582, 'x4': 0.570224, 'x5': 0.302456, 'x6': 0.864798} using GenerationNode Sobol.
[INFO 09-21 05:09:32] ax.api.client: Trial 24 marked COMPLETED.
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/sklearn/base.py:1365: DataConversionWarning:
A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
[INFO 09-21 05:09:32] ax.api.client: Generated new trial 25 with parameters {'x1': 0.242848, 'x2': 0.766958, 'x3': 0.144753, 'x4': 0.339223, 'x5': 0.553891, 'x6': 0.036107} using GenerationNode RandomForest.
[INFO 09-21 05:09:32] ax.api.client: Trial 25 marked COMPLETED.
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/sklearn/base.py:1365: DataConversionWarning:
A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
[INFO 09-21 05:09:32] ax.api.client: Generated new trial 26 with parameters {'x1': 0.483604, 'x2': 0.506459, 'x3': 0.47215, 'x4': 0.584948, 'x5': 0.259142, 'x6': 0.961842} using GenerationNode RandomForest.
[INFO 09-21 05:09:32] ax.api.client: Trial 26 marked COMPLETED.
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/sklearn/base.py:1365: DataConversionWarning:
A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
[INFO 09-21 05:09:32] ax.api.client: Generated new trial 27 with parameters {'x1': 0.638819, 'x2': 0.635674, 'x3': 0.084228, 'x4': 0.144131, 'x5': 0.680864, 'x6': 0.017266} using GenerationNode RandomForest.
[INFO 09-21 05:09:32] ax.api.client: Trial 27 marked COMPLETED.
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/sklearn/base.py:1365: DataConversionWarning:
A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
[INFO 09-21 05:09:32] ax.api.client: Generated new trial 28 with parameters {'x1': 0.260773, 'x2': 0.896815, 'x3': 0.559878, 'x4': 0.686772, 'x5': 0.15346, 'x6': 0.609451} using GenerationNode RandomForest.
[INFO 09-21 05:09:32] ax.api.client: Trial 28 marked COMPLETED.
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/sklearn/base.py:1365: DataConversionWarning:
A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
[INFO 09-21 05:09:32] ax.api.client: Generated new trial 29 with parameters {'x1': 0.125019, 'x2': 0.278793, 'x3': 0.332179, 'x4': 0.771631, 'x5': 0.54654, 'x6': 0.061237} using GenerationNode RandomForest.
[INFO 09-21 05:09:32] ax.api.client: Trial 29 marked COMPLETED.
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/sklearn/base.py:1365: DataConversionWarning:
A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
[INFO 09-21 05:09:32] ax.api.client: Generated new trial 30 with parameters {'x1': 0.884537, 'x2': 0.487939, 'x3': 0.122992, 'x4': 0.768143, 'x5': 0.562872, 'x6': 0.26195} using GenerationNode RandomForest.
[INFO 09-21 05:09:32] ax.api.client: Trial 30 marked COMPLETED.
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/sklearn/base.py:1365: DataConversionWarning:
A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
[INFO 09-21 05:09:32] ax.api.client: Generated new trial 31 with parameters {'x1': 0.271196, 'x2': 0.566097, 'x3': 0.8093, 'x4': 0.50328, 'x5': 0.117585, 'x6': 0.884512} using GenerationNode RandomForest.
[INFO 09-21 05:09:32] ax.api.client: Trial 31 marked COMPLETED.
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/sklearn/base.py:1365: DataConversionWarning:
A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
[INFO 09-21 05:09:33] ax.api.client: Generated new trial 32 with parameters {'x1': 0.320206, 'x2': 0.722646, 'x3': 0.663946, 'x4': 0.466369, 'x5': 0.826566, 'x6': 0.922746} using GenerationNode RandomForest.
[INFO 09-21 05:09:33] ax.api.client: Trial 32 marked COMPLETED.
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/sklearn/base.py:1365: DataConversionWarning:
A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
[INFO 09-21 05:09:33] ax.api.client: Generated new trial 33 with parameters {'x1': 0.122749, 'x2': 0.299301, 'x3': 0.773283, 'x4': 0.242459, 'x5': 0.686683, 'x6': 0.744457} using GenerationNode RandomForest.
[INFO 09-21 05:09:33] ax.api.client: Trial 33 marked COMPLETED.
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/sklearn/base.py:1365: DataConversionWarning:
A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
[INFO 09-21 05:09:33] ax.api.client: Generated new trial 34 with parameters {'x1': 0.886963, 'x2': 0.084726, 'x3': 0.896356, 'x4': 0.713198, 'x5': 0.576136, 'x6': 0.117818} using GenerationNode RandomForest.
[INFO 09-21 05:09:33] ax.api.client: Trial 34 marked COMPLETED.
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/sklearn/base.py:1365: DataConversionWarning:
A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
[INFO 09-21 05:09:33] ax.api.client: Generated new trial 35 with parameters {'x1': 0.738438, 'x2': 0.992811, 'x3': 0.165132, 'x4': 0.933638, 'x5': 0.414243, 'x6': 0.037996} using GenerationNode RandomForest.
[INFO 09-21 05:09:33] ax.api.client: Trial 35 marked COMPLETED.
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/sklearn/base.py:1365: DataConversionWarning:
A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
[INFO 09-21 05:09:33] ax.api.client: Generated new trial 36 with parameters {'x1': 0.532045, 'x2': 0.252672, 'x3': 0.848211, 'x4': 0.479989, 'x5': 0.913371, 'x6': 0.950651} using GenerationNode RandomForest.
[INFO 09-21 05:09:33] ax.api.client: Trial 36 marked COMPLETED.
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/sklearn/base.py:1365: DataConversionWarning:
A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
[INFO 09-21 05:09:33] ax.api.client: Generated new trial 37 with parameters {'x1': 0.5701, 'x2': 0.646536, 'x3': 0.653522, 'x4': 0.510946, 'x5': 0.03968, 'x6': 0.898508} using GenerationNode RandomForest.
[INFO 09-21 05:09:33] ax.api.client: Trial 37 marked COMPLETED.
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/sklearn/base.py:1365: DataConversionWarning:
A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
[INFO 09-21 05:09:33] ax.api.client: Generated new trial 38 with parameters {'x1': 0.633296, 'x2': 0.508261, 'x3': 0.120722, 'x4': 0.199435, 'x5': 0.597074, 'x6': 0.258951} using GenerationNode RandomForest.
[INFO 09-21 05:09:33] ax.api.client: Trial 38 marked COMPLETED.
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/sklearn/base.py:1365: DataConversionWarning:
A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
[INFO 09-21 05:09:33] ax.api.client: Generated new trial 39 with parameters {'x1': 0.729472, 'x2': 0.471035, 'x3': 0.45584, 'x4': 0.884497, 'x5': 0.33883, 'x6': 0.577395} using GenerationNode RandomForest.
[INFO 09-21 05:09:33] ax.api.client: Trial 39 marked COMPLETED.
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/sklearn/base.py:1365: DataConversionWarning:
A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
[INFO 09-21 05:09:33] ax.api.client: Generated new trial 40 with parameters {'x1': 0.696179, 'x2': 0.219478, 'x3': 0.336892, 'x4': 0.329622, 'x5': 0.33116, 'x6': 0.385098} using GenerationNode RandomForest.
[INFO 09-21 05:09:33] ax.api.client: Trial 40 marked COMPLETED.
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/sklearn/base.py:1365: DataConversionWarning:
A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
[INFO 09-21 05:09:33] ax.api.client: Generated new trial 41 with parameters {'x1': 0.605244, 'x2': 0.910719, 'x3': 0.94642, 'x4': 0.425336, 'x5': 0.698403, 'x6': 0.471982} using GenerationNode RandomForest.
[INFO 09-21 05:09:33] ax.api.client: Trial 41 marked COMPLETED.
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/sklearn/base.py:1365: DataConversionWarning:
A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
[INFO 09-21 05:09:33] ax.api.client: Generated new trial 42 with parameters {'x1': 0.977949, 'x2': 0.904655, 'x3': 0.055582, 'x4': 0.749028, 'x5': 0.814394, 'x6': 0.330281} using GenerationNode RandomForest.
[INFO 09-21 05:09:33] ax.api.client: Trial 42 marked COMPLETED.
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/sklearn/base.py:1365: DataConversionWarning:
A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
[INFO 09-21 05:09:34] ax.api.client: Generated new trial 43 with parameters {'x1': 0.852869, 'x2': 0.15347, 'x3': 0.742763, 'x4': 0.972014, 'x5': 0.896605, 'x6': 0.554705} using GenerationNode RandomForest.
[INFO 09-21 05:09:34] ax.api.client: Trial 43 marked COMPLETED.
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/sklearn/base.py:1365: DataConversionWarning:
A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
[INFO 09-21 05:09:34] ax.api.client: Generated new trial 44 with parameters {'x1': 0.588709, 'x2': 0.527721, 'x3': 0.167654, 'x4': 0.066855, 'x5': 0.503505, 'x6': 0.225451} using GenerationNode RandomForest.
[INFO 09-21 05:09:34] ax.api.client: Trial 44 marked COMPLETED.
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/sklearn/base.py:1365: DataConversionWarning:
A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
[INFO 09-21 05:09:34] ax.api.client: Generated new trial 45 with parameters {'x1': 0.986062, 'x2': 0.691918, 'x3': 0.597749, 'x4': 0.909848, 'x5': 0.217368, 'x6': 0.702613} using GenerationNode RandomForest.
[INFO 09-21 05:09:34] ax.api.client: Trial 45 marked COMPLETED.
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/sklearn/base.py:1365: DataConversionWarning:
A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
[INFO 09-21 05:09:34] ax.api.client: Generated new trial 46 with parameters {'x1': 0.030735, 'x2': 0.341686, 'x3': 0.529146, 'x4': 0.432684, 'x5': 0.114867, 'x6': 0.955257} using GenerationNode RandomForest.
[INFO 09-21 05:09:34] ax.api.client: Trial 46 marked COMPLETED.
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/sklearn/base.py:1365: DataConversionWarning:
A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
[INFO 09-21 05:09:34] ax.api.client: Generated new trial 47 with parameters {'x1': 0.676377, 'x2': 0.400062, 'x3': 0.656862, 'x4': 0.473236, 'x5': 0.883026, 'x6': 0.758763} using GenerationNode RandomForest.
[INFO 09-21 05:09:34] ax.api.client: Trial 47 marked COMPLETED.
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/sklearn/base.py:1365: DataConversionWarning:
A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
[INFO 09-21 05:09:34] ax.api.client: Generated new trial 48 with parameters {'x1': 0.826151, 'x2': 0.234512, 'x3': 0.835579, 'x4': 0.724106, 'x5': 0.058299, 'x6': 0.728682} using GenerationNode RandomForest.
[INFO 09-21 05:09:34] ax.api.client: Trial 48 marked COMPLETED.
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/sklearn/base.py:1365: DataConversionWarning:
A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
[INFO 09-21 05:09:34] ax.api.client: Generated new trial 49 with parameters {'x1': 0.131432, 'x2': 0.388241, 'x3': 0.488445, 'x4': 0.801837, 'x5': 0.57145, 'x6': 0.700262} using GenerationNode RandomForest.
[INFO 09-21 05:09:34] ax.api.client: Trial 49 marked COMPLETED.

View your results by computing Analyses. Not all Analyses implemented by Ax will be compatible with an ExternalGenerationNode (specifically those which rely on an Ax Adapter for their input data) though many which rely solely on raw observation will work out of the box.

client.compute_analyses(
    analyses=[ArmEffectsPlot(use_model_predictions=False), Summary()]
)

Observed Arm Effects on hartmann6

Observed effects on hartmann6. This plot visualizes the effects from previously-run arms on a specific metric, providing insights into their performance. This plot allows one to compare and contrast the effectiveness of different arms, highlighting which configurations have yielded the most favorable outcomes.

loading...

Summary for Experiment

High-level summary of the Trial-s in this Experiment

trial_indexarm_nametrial_statusgeneration_nodehartmann6x1x2x3x4x5x6
000_0COMPLETEDSobol-0.2698590.3314170.4402830.7268760.1331180.6869480.357847
111_0COMPLETEDSobol-0.0064820.8845090.7147930.3105860.9091640.2260060.971387
222_0COMPLETEDSobol-1.141910.6436230.0112940.7817570.2623520.3551220.577043
333_0COMPLETEDSobol-0.3013930.0722190.7682930.2410770.5371630.8004310.189669
444_0COMPLETEDSobol-0.1561820.1799230.1347540.4531140.6558670.9692910.441183
555_0COMPLETEDSobol-0.1535040.5965560.8914460.5250440.4269420.4302250.827246
666_0COMPLETEDSobol-0.0200480.8676240.3131910.0070020.7767880.0510560.72198
777_0COMPLETEDSobol-0.1015970.4087550.5875210.9545420.0486050.6057390.108956
888_0COMPLETEDSobol-0.1543120.453630.070530.0972250.496510.1538430.171054
999_0COMPLETEDSobol-0.0115120.7601560.8291740.9275780.7097430.6929090.535111
101010_0COMPLETEDSobol-0.077710.5164010.3764070.4173290.117770.8215440.890271
111111_0COMPLETEDSobol-0.293250.1974240.6487850.6220720.8302730.2668610.253418
121212_0COMPLETEDSobol-0.0041540.0584990.2530390.8165340.9801280.4398580.00245
131313_0COMPLETEDSobol-0.0030710.7197460.5256010.1430420.1885280.9007990.639034
141414_0COMPLETEDSobol-0.1138630.9957580.1975910.6376620.6087170.5220950.783241
151515_0COMPLETEDSobol-0.2444390.282510.9565370.3385590.3183310.0767870.420735
161616_0COMPLETEDSobol-0.020410.2785330.2235930.8934970.8006210.7727710.676555
171717_0COMPLETEDSobol-0.0020770.9451510.9979190.067050.0248950.3274540.054772
181818_0COMPLETEDSobol-0.1329310.716280.2946640.5880360.6788340.1910220.394779
191919_0COMPLETEDSobol-0.2919950.0074260.5513590.3871990.4038540.6519560.77404
202020_0COMPLETEDSobol-0.6051520.2405580.4216920.1771230.2852720.5709990.524781
212121_0COMPLETEDSobol-1.361210.5436660.6786950.846710.5143650.0163080.146228
222222_0COMPLETEDSobol-0.2915130.8028410.1001960.3725940.1571490.4028090.306564
232323_0COMPLETEDSobol-0.0017660.4814220.8747030.6677910.8850110.9418680.926968
242424_0COMPLETEDSobol-0.7786060.4052020.3575060.2745820.5702240.3024560.864798
252525_0COMPLETEDRandomForest-1.022760.2428480.7669580.1447530.3392230.5538910.036107
262626_0COMPLETEDRandomForest-0.4139670.4836040.5064590.472150.5849480.2591420.961842
272727_0COMPLETEDRandomForest-0.1151670.6388190.6356740.0842280.1441310.6808640.017266
282828_0COMPLETEDRandomForest-0.0924680.2607730.8968150.5598780.6867720.153460.609451
292929_0COMPLETEDRandomForest-0.0401160.1250190.2787930.3321790.7716310.546540.061237
303030_0COMPLETEDRandomForest-0.0104550.8845370.4879390.1229920.7681430.5628720.26195
313131_0COMPLETEDRandomForest-1.063610.2711960.5660970.80930.503280.1175850.884512
323232_0COMPLETEDRandomForest-0.0529330.3202060.7226460.6639460.4663690.8265660.922746
333333_0COMPLETEDRandomForest-0.4623110.1227490.2993010.7732830.2424590.6866830.744457
343434_0COMPLETEDRandomForest-0.0024320.8869630.0847260.8963560.7131980.5761360.117818
353535_0COMPLETEDRandomForest-0.1162220.7384380.9928110.1651320.9336380.4142430.037996
363636_0COMPLETEDRandomForest-0.0129230.5320450.2526720.8482110.4799890.9133710.950651
373737_0COMPLETEDRandomForest-0.4370270.57010.6465360.6535220.5109460.039680.898508
383838_0COMPLETEDRandomForest-0.1131020.6332960.5082610.1207220.1994350.5970740.258951
393939_0COMPLETEDRandomForest-0.0328030.7294720.4710350.455840.8844970.338830.577395
404040_0COMPLETEDRandomForest-0.8143560.6961790.2194780.3368920.3296220.331160.385098
414141_0COMPLETEDRandomForest-0.0960070.6052440.9107190.946420.4253360.6984030.471982
424242_0COMPLETEDRandomForest-0.0024910.9779490.9046550.0555820.7490280.8143940.330281
434343_0COMPLETEDRandomForest-0.0003420.8528690.153470.7427630.9720140.8966050.554705
444444_0COMPLETEDRandomForest-0.108870.5887090.5277210.1676540.0668550.5035050.225451
454545_0COMPLETEDRandomForest-0.0575370.9860620.6919180.5977490.9098480.2173680.702613
464646_0COMPLETEDRandomForest-0.7903540.0307350.3416860.5291460.4326840.1148670.955257
474747_0COMPLETEDRandomForest-0.0188130.6763770.4000620.6568620.4732360.8830260.758763
484848_0COMPLETEDRandomForest-0.3348810.8261510.2345120.8355790.7241060.0582990.728682
494949_0COMPLETEDRandomForest-0.1293690.1314320.3882410.4884450.8018370.571450.700262
Output:
[<ax.analysis.plotly.plotly_analysis.PlotlyAnalysisCard at 0x7efdc2707ce0>,
<ax.analysis.analysis_card.AnalysisCard at 0x7efdc1c8efc0>]