Version: Next

Ask-tell Optimization in a Human-in-the-loop Setting

Some optimization experiments, like the one described in this tutorial, can be conducted in a completely automated manner. Other experiments may require a human in the loop, for instance a scientist manually conducting and evaluating each trial in a lab. In this tutorial we demonstrate this ask-tell optimization in a human-in-the-loop setting by imagining the task of maximizing the strength of a 3D printed part using compression testing (i.e., crushing the part) where different print settings will have to be manually tried and evaluated.

Background

In 3D printing, several parameters can significantly affect the mechanical properties of the printed object:

Infill Density: The percentage of material used inside the object. Higher infill density generally increases strength but also weight and material usage.
Layer Height: The thickness of each layer of material. Smaller layer heights can improve surface finish and detail but increase print time.
Infill Type: The pattern used to fill the interior of the object. Different patterns (e.g., honeycomb, gyroid, lines, rectilinear) offer various balances of strength, speed, and material efficiency.
Strength Measurement: In this tutorial, we assume the strength of the 3D printed part is measured using compression testing, which evaluates how the object performs under compressive stress.

Learning Objectives

Understand black box optimization concepts
Define an optimization problem using Ax
Configure and run an experiment using Ax's Client
Analyze the results of the optimization

Prerequisites

Familiarity with Python and basic programming concepts
Understanding of adaptive experimentation and Bayesian optimization

Step 1: Import Necessary Modules

from ax.api.client import Client
from ax.api.configs import ExperimentConfig, RangeParameterConfig, ChoiceParameterConfig, ParameterType

Step 2: Initialize Client

Create an instance of the Client to manage the state of your experiment.

client = Client()

Step 3: Configure Experiment

Define the parameters for the 3D printing optimization problem. The infill density and layer height can take on any value within their respective bounds so we will configure both using RangeParameterConfigs. On the other hand, infill type be either have one of four distinct values: "honeycomb", "gyroid", "lines", or "rectilinear". We will use a ChoiceParameterConfig to represent it in the optimization.

infill_density = RangeParameterConfig(name="infill_density", parameter_type=ParameterType.FLOAT, bounds=(0, 100))
layer_height = RangeParameterConfig(name="layer_height", parameter_type=ParameterType.FLOAT, bounds=(0.1, 0.4))
infill_type = ChoiceParameterConfig(name="infill_type", parameter_type=ParameterType.STRING, values=["honeycomb", "gyroid", "lines", "rectilinear"])

experiment_config = ExperimentConfig(
    name="3d_print_strength_experiment",
    parameters=[infill_density, layer_height, infill_type],
    # The following arguments are optional
    description="Maximize strength of 3D printed parts",
    owner="developer",
)

client.configure_experiment(experiment_config=experiment_config)

Output:
/home/runner/work/Ax/Ax/ax/api/utils/instantiation/from_config.py:89: AxParameterWarning: is_ordered is not specified for ChoiceParameter "infill_type". Defaulting to False  since the parameter is a string with more than 2 choices.. To override this behavior (or avoid this warning), specify is_ordered during ChoiceParameter construction. Note that choice parameters with exactly 2 choices are always considered ordered and that the user-supplied is_ordered has no effect in this particular case.
return ChoiceParameter(
/home/runner/work/Ax/Ax/ax/api/utils/instantiation/from_config.py:89: AxParameterWarning: sort_values is not specified for ChoiceParameter "infill_type". Defaulting to False for parameters of ParameterType STRING. To override this behavior (or avoid this warning), specify sort_values during ChoiceParameter construction.
return ChoiceParameter(

Step 4: Configure Optimization

We want to maximize the compressive strength of our part, so we will set the objective to compressive_strength. However, we know that modifying the infill density, layer height, and infill type will affect the weight of the part as well. We'll include a requirement that the part must not weigh more than 10 grams by setting an outcome constraint when we call configure_experiment.

The following code will tell the Client that we intend to maximize compressive strength while keeping the weight less than 10 grams.

client.configure_optimization(objective="compressive_strength", outcome_constraints=["weight <= 10"])

Step 5: Run Trials

Now the Client has been configured we can begin conducting the experiment. Use attach_trial to attach any existing data, use get_next_trials to generate parameter suggestions, and use complete_trial to report manually observed results.

Attach Preexisting Trials

Sometimes in our optimization experiments we may already have some previously collected data from manual "trials" conducted before the Ax experiment began. This can be incredibly useful! If we attach this data as custom trials, Ax will be able to use the data points in its optimization algorithm and improve performance.

# Pairs of previously evaluated parameterizations and associated metric readings
preexisting_trials = [
    (
        {"infill_density": 10.43, "layer_height": 0.3, "infill_type": "gyroid"},
        {"compressive_strength": 1.74, "weight": 0.52},
    ),
    (
        {"infill_density": 55.54, "layer_height": 0.12, "infill_type": "lines"},
        {"compressive_strength": 4.63, "weight": 2.31},
    ),
    (
        {"infill_density": 99.43, "layer_height": 0.35, "infill_type": "rectilinear"},
        {"compressive_strength": 5.68, "weight": 2.84},
    ),
    (
        {"infill_density": 41.44, "layer_height": 0.21, "infill_type": "rectilinear"},
        {"compressive_strength": 3.95, "weight": 1.97},
    ),
    (
        {"infill_density": 27.23, "layer_height": 0.37, "infill_type": "honeycomb"},
        {"compressive_strength": 7.36, "weight": 3.31},
    ),
    (
        {"infill_density": 33.57, "layer_height": 0.24, "infill_type": "honeycomb"},
        {"compressive_strength": 13.99, "weight": 6.29},
    ),
]

for parameters, data in preexisting_trials:
    # Attach the parameterization to the Client as a trial and immediately complete it with the preexisting data
    trial_index = client.attach_trial(parameters=parameters)
    client.complete_trial(trial_index=trial_index, raw_data=data)

Output:
[INFO 04-18 05:04:26] ax.core.experiment: Attached custom parameterizations [{'infill_density': 10.43, 'layer_height': 0.3, 'infill_type': 'gyroid'}] as trial 0.
[INFO 04-18 05:04:26] ax.core.experiment: Attached custom parameterizations [{'infill_density': 55.54, 'layer_height': 0.12, 'infill_type': 'lines'}] as trial 1.
[INFO 04-18 05:04:26] ax.core.experiment: Attached custom parameterizations [{'infill_density': 99.43, 'layer_height': 0.35, 'infill_type': 'rectilinear'}] as trial 2.
[INFO 04-18 05:04:26] ax.core.experiment: Attached custom parameterizations [{'infill_density': 41.44, 'layer_height': 0.21, 'infill_type': 'rectilinear'}] as trial 3.
[INFO 04-18 05:04:26] ax.core.experiment: Attached custom parameterizations [{'infill_density': 27.23, 'layer_height': 0.37, 'infill_type': 'honeycomb'}] as trial 4.
[INFO 04-18 05:04:26] ax.core.experiment: Attached custom parameterizations [{'infill_density': 33.57, 'layer_height': 0.24, 'infill_type': 'honeycomb'}] as trial 5.

Ask for trials

Now, let's have Ax suggest which trials to evaluate so that we can find the optimal configuration more efficiently. We'll do this by calling get_next_trials. We'll make use of Ax's support for parallelism, i.e. suggesting more than one trial at a time -- this can allow us to conduct our experiment much faster! If our lab had three identical 3D printers, we could ask Ax for a batch of three trials and evaluate three different infill density, layer height, and infill types at once.

Note that there will always be a tradeoff between "parallelism" and optimization performance since the quality of a suggested trial is often proportional to the amount of data Ax has access to, see this recipe for a more detailed explanation.

trials = client.get_next_trials(maximum_trials=3)
trials

Output:
{6: {'infill_density': 55.01488739012378,
'layer_height': 0.2038138965099706,
'infill_type': 'honeycomb'},
7: {'infill_density': 3.597331767568907,
'layer_height': 0.2237046722455596,
'infill_type': 'honeycomb'},
8: {'infill_density': 64.6364478179494,
'layer_height': 0.26050077475810346,
'infill_type': 'honeycomb'}}

Tell Ax the results

In a real-world scenerio we would print parts using the three suggested parameterizations and measure the compressive strength and weight manually, though in this tutorial we will simulate by calling a function. Once the data is collected we will tell Ax the result by calling complete_trial.

def evaluate(
    infill_density: float, layer_height: float, infill_type: str
) -> dict[str, float]:
    strength_map = {"lines": 1, "rectilinear": 2, "gyroid": 5, "honeycomb": 10}
    weight_map = {"lines": 1, "rectilinear": 2, "gyroid": 3, "honeycomb": 9}

    return {
        "compressive_strength": (
            infill_density / layer_height * strength_map[infill_type]
        )
        / 100,
        "weight": (infill_density / layer_height * weight_map[infill_type]) / 200,
    }


for trial_index, parameters in trials.items():
    client.complete_trial(trial_index=trial_index, raw_data=evaluate(**parameters))

We'll repeat this process a number of times. Typically experimentation will continue until a satisfactory combination has been found, experimentation resources (in this example our 3D printing filliment) have been exhausted, or we feel we have spent enough time on optimization.

# Ask Ax for the next trials
trials = client.get_next_trials(maximum_trials=3)
trials

Output:
{9: {'infill_density': 41.81389808717594,
'layer_height': 0.1373329300981579,
'infill_type': 'honeycomb'},
10: {'infill_density': 72.00513461281771,
'layer_height': 0.1,
'infill_type': 'honeycomb'},
11: {'infill_density': 77.75864682244575,
'layer_height': 0.1,
'infill_type': 'honeycomb'}}

# Tell Ax the result of those trials
for trial_index, parameters in trials.items():
    client.complete_trial(trial_index=trial_index, raw_data=evaluate(**parameters))

# Ask Ax for the next trials
trials = client.get_next_trials(maximum_trials=3)
trials

Output:
{12: {'infill_density': 51.92158995930188,
'layer_height': 0.39049720213769673,
'infill_type': 'lines'},
13: {'infill_density': 100.0,
'layer_height': 0.1,
'infill_type': 'rectilinear'},
14: {'infill_density': 37.22802518601801,
'layer_height': 0.1,
'infill_type': 'gyroid'}}

# Tell Ax the result of those trials
for trial_index, parameters in trials.items():
    client.complete_trial(trial_index=trial_index, raw_data=evaluate(**parameters))

Step 6: Analyze Results

At any time during the experiment you may analyze the results of the experiment. Most commonly this means extracting the parameterization from the best performing trial you conducted. The best trial will have the optimal objective value without violating any outcome constraints.

best_parameters, prediction, index, name = client.get_best_parameterization()
print("Best Parameters:", best_parameters)
print("Prediction (mean, variance):", prediction)

Output:
Best Parameters: {'infill_density': 33.57, 'layer_height': 0.24, 'infill_type': 'honeycomb'}
Prediction (mean, variance): {'weight': (np.float64(6.64625245749128), np.float64(0.38801156894432054)), 'compressive_strength': (np.float64(13.416617418935491), np.float64(2.9677876720318412))}

Step 7: Compute Analyses

Ax can also produce a number of analyses to help interpret the results of the experiment via client.compute_analyses. Users can manually select which analyses to run, or can allow Ax to select which would be most relevant. In this case Ax selects the following:

Parrellel Coordinates Plot shows which parameterizations were evaluated and what metric values were observed -- this is useful for getting a high level overview of how thoroughly the search space was explored and which regions tend to produce which outcomes
Scatter Plot shows both metric values for each trial -- this is useful for understanding the tradeoffs between competing metrics, like compressive strength and weight.
Interaction Analysis Plot shows which parameters have the largest affect on the function and plots the most important parameters as 1 or 2 dimensional surfaces
Summary lists all trials generated along with their parameterizations, observations, and miscellaneous metadata

# display=True instructs Ax to sort then render the resulting analyses
cards = client.compute_analyses(display=True)

Modeled compressive_strength vs. weight

This plot displays the effects of each arm on the two selected metrics. It is useful for understanding the trade-off between the two metrics and for visualizing the Pareto frontier.

Parallel Coordinates for compressive_strength

The parallel coordinates plot displays multi-dimensional data by representing each parameter as a parallel axis. This plot helps in assessing how thoroughly the search space has been explored and in identifying patterns or clusterings associated with high-performing (good) or low-performing (bad) arms. By tracing lines across the axes, one can observe correlations and interactions between parameters, gaining insights into the relationships that contribute to the success or failure of different configurations within the experiment.

Sensitivity Analysis for compressive_strength

Understand how each parameter affects compressive_strength according to a second-order sensitivity analysis.

infill_density vs. compressive_strength

The slice plot provides a one-dimensional view of predicted outcomes for compressive_strength as a function of a single parameter, while keeping all other parameters fixed at their status_quo value (or mean value if status_quo is unavailable). This visualization helps in understanding the sensitivity and impact of changes in the selected parameter on the predicted metric outcomes.

infill_density, layer_height vs. compressive_strength

The contour plot visualizes the predicted outcomes for compressive_strength across a two-dimensional parameter space, with other parameters held fixed at their status_quo value (or mean value if status_quo is unavailable). This plot helps in identifying regions of optimal performance and understanding how changes in the selected parameters influence the predicted outcomes. Contour lines represent levels of constant predicted values, providing insights into the gradient and potential optima within the parameter space.

layer_height vs. compressive_strength

Summary for 3d_print_strength_experiment

High-level summary of the Trial-s in this Experiment

	trial_index	arm_name	trial_status	generation_node	weight	compressive_strength	infill_density	layer_height	infill_type
0	0	0_0	COMPLETED	nan	0.52	1.74	10.43	0.3	gyroid
1	1	1_0	COMPLETED	nan	2.31	4.63	55.54	0.12	lines
2	2	2_0	COMPLETED	nan	2.84	5.68	99.43	0.35	rectilinear
3	3	3_0	COMPLETED	nan	1.97	3.95	41.44	0.21	rectilinear
4	4	4_0	COMPLETED	nan	3.31	7.36	27.23	0.37	honeycomb
5	5	5_0	COMPLETED	nan	6.29	13.99	33.57	0.24	honeycomb
6	6	6_0	COMPLETED	MBM	12.1467	26.9927	55.0149	0.203814	honeycomb
7	7	7_0	COMPLETED	MBM	0.723632	1.60807	3.59733	0.223705	honeycomb
8	8	8_0	COMPLETED	MBM	11.1656	24.8124	64.6364	0.260501	honeycomb
9	9	9_0	COMPLETED	MBM	13.7012	30.4471	41.8139	0.137333	honeycomb
10	10	10_0	COMPLETED	MBM	32.4023	72.0051	72.0051	0.1	honeycomb
11	11	11_0	COMPLETED	MBM	34.9914	77.7586	77.7586	0.1	honeycomb
12	12	12_0	COMPLETED	MBM	0.664814	1.32963	51.9216	0.390497	lines
13	13	13_0	COMPLETED	MBM	10	20	100	0.1	rectilinear
14	14	14_0	COMPLETED	MBM	5.5842	18.614	37.228	0.1	gyroid

Cross Validation for compressive_strength

The cross-validation plot displays the model fit for each metric in the experiment. It employs a leave-one-out approach, where the model is trained on all data except one sample, which is used for validation. The plot shows the predicted outcome for the validation set on the y-axis against its actual value on the x-axis. Points that align closely with the dotted diagonal line indicate a strong model fit, signifying accurate predictions. Additionally, the plot includes 95% confidence intervals that provide insight into the noise in observations and the uncertainty in model predictions. A horizontal, flat line of predictions indicates that the model has not picked up on sufficient signal in the data, and instead is just predicting the mean.

Cross Validation for weight

Conclusion

This tutorial demonstrates how to use Ax's Client for optimizing the strength of 3D printed parts in a human-in-the-loop setting. By iteratively collecting data and refining parameters, you can effectively apply black box optimization to real-world experiments.

Background​

Learning Objectives​

Prerequisites​

Step 1: Import Necessary Modules​

Step 2: Initialize Client​

Step 3: Configure Experiment​

Step 4: Configure Optimization​

Step 5: Run Trials​

Attach Preexisting Trials​

Ask for trials​

Tell Ax the results​

Step 6: Analyze Results​

Step 7: Compute Analyses​

Conclusion​