Open in GitHubRun in Google Colab

Ask-tell Optimization in a Human-in-the-loop Setting

Some optimization experiments, like the one described in this tutorial, can be conducted in a completely automated manner. Other experiments may require a human in the loop, for instance a scientist manually conducting and evaluating each trial in a lab. In this tutorial we demonstrate this ask-tell optimization in a human-in-the-loop setting by imagining the task of maximizing the strength of a 3D printed part using compression testing (i.e., crushing the part) where different print settings will have to be manually tried and evaluated.

Background

In 3D printing, several parameters can significantly affect the mechanical properties of the printed object:

• Infill Density: The percentage of material used inside the object. Higher infill density generally increases strength but also weight and material usage.

• Layer Height: The thickness of each layer of material. Smaller layer heights can improve surface finish and detail but increase print time.

• Infill Type: The pattern used to fill the interior of the object. Different patterns (e.g., honeycomb, gyroid, lines, rectilinear) offer various balances of strength, speed, and material efficiency.

• Strength Measurement: In this tutorial, we assume the strength of the 3D printed part is measured using compression testing, which evaluates how the object performs under compressive stress.

Learning Objectives

• Understand black box optimization concepts
• Define an optimization problem using Ax
• Configure and run an experiment using Ax's Client
• Analyze the results of the optimization

Prerequisites

• Familiarity with Python and basic programming concepts
• Understanding of adaptive experimentation and Bayesian optimization

Step 1: Import Necessary Modules

import pandas as pd

from ax.preview.api.client import Client
from ax.preview.api.configs import ExperimentConfig, RangeParameterConfig, ChoiceParameterConfig, ParameterType, GenerationStrategyConfig

Step 2: Initialize Client

Create an instance of the Client to manage the state of your experiment.

client = Client()

Step 3: Configure Experiment

Define the parameters for the 3D printing optimization problem. The infill density and layer height can take on any value within their respective bounds so we will configure both using RangeParameterConfigs. On the other hand, infill type be either have one of four distinct values: "honeycomb", "gyroid", "lines", or "rectilinear". We will use a ChoiceParameterConfig to represent it in the optimization.

infill_density = RangeParameterConfig(name="infill_density", parameter_type=ParameterType.FLOAT, bounds=(0, 100))
layer_height = RangeParameterConfig(name="layer_height", parameter_type=ParameterType.FLOAT, bounds=(0.1, 0.4))
infill_type = ChoiceParameterConfig(name="infill_type", parameter_type=ParameterType.STRING, values=["honeycomb", "gyroid", "lines", "rectilinear"])

experiment_config = ExperimentConfig(
    name="3d_print_strength_experiment",
    parameters=[infill_density, layer_height, infill_type],
    # The following arguments are optional
    description="Maximize strength of 3D printed parts",
    owner="developer",
)

client.configure_experiment(experiment_config=experiment_config)

Out:

/home/runner/work/Ax/Ax/ax/preview/api/utils/instantiation/from_config.py:89: AxParameterWarning: is_ordered is not specified for ChoiceParameter "infill_type". Defaulting to False  since the parameter is a string with more than 2 choices.. To override this behavior (or avoid this warning), specify is_ordered during ChoiceParameter construction. Note that choice parameters with exactly 2 choices are always considered ordered and that the user-supplied is_ordered has no effect in this particular case.
return ChoiceParameter(
/home/runner/work/Ax/Ax/ax/preview/api/utils/instantiation/from_config.py:89: AxParameterWarning: sort_values is not specified for ChoiceParameter "infill_type". Defaulting to False for parameters of ParameterType STRING. To override this behavior (or avoid this warning), specify sort_values during ChoiceParameter construction.
return ChoiceParameter(

Step 4: Configure Optimization

We want to maximize the compressive strength of our part, so we will set the objective to compressive_strength. However, we know that modifying the infill density, layer height, and infill type will affect the weight of the part as well. We'll include a requirement that the part must not weigh more than 10 grams by setting an outcome constraint when we call configure_experiment.

The following code will tell the Client that we intend to maximize compressive strength while keeping the weight less than 10 grams.

client.configure_optimization(objective="compressive_strength", outcome_constraints=["weight <= 10"])

Step 5: Run Trials

Now the Client has been configured we can begin conducting the experiment. Use attach_trial to attach any existing data, use get_next_trials to generate parameter suggestions, and use complete_trial to report manually observed results.

Attach Preexisting Trials

Sometimes in our optimization experiments we may already have some previously collected data from manual "trials" conducted before the Ax experiment began. This can be incredibly useful! If we attach this data as custom trials, Ax will be able to use the data points in its optimization algorithm and improve performance.

# Pairs of previously evaluated parameterizations and associated metric readings
preexisting_trials = [
    (
        {"infill_density": 10.43, "layer_height": 0.3, "infill_type": "gyroid"},
        {"compressive_strength": 1.74, "weight": 0.52},
    ),
    (
        {"infill_density": 55.54, "layer_height": 0.12, "infill_type": "lines"},
        {"compressive_strength": 4.63, "weight": 2.31},
    ),
    (
        {"infill_density": 99.43, "layer_height": 0.35, "infill_type": "rectilinear"},
        {"compressive_strength": 5.68, "weight": 2.84},
    ),
    (
        {"infill_density": 41.44, "layer_height": 0.21, "infill_type": "rectilinear"},
        {"compressive_strength": 3.95, "weight": 1.97},
    ),
    (
        {"infill_density": 27.23, "layer_height": 0.37, "infill_type": "honeycomb"},
        {"compressive_strength": 7.36, "weight": 3.31},
    ),
    (
        {"infill_density": 33.57, "layer_height": 0.24, "infill_type": "honeycomb"},
        {"compressive_strength": 13.99, "weight": 6.29},
    ),
]

for parameters, data in preexisting_trials:
    # Attach the parameterization to the Client as a trial and immediately complete it with the preexisting data
    trial_index = client.attach_trial(parameters=parameters)
    client.complete_trial(trial_index=trial_index, raw_data=data)

Out:

[INFO 03-14 05:05:04] ax.core.experiment: Attached custom parameterizations [{'infill_density': 10.43, 'layer_height': 0.3, 'infill_type': 'gyroid'}] as trial 0.

Out:

[INFO 03-14 05:05:04] ax.core.experiment: Attached custom parameterizations [{'infill_density': 55.54, 'layer_height': 0.12, 'infill_type': 'lines'}] as trial 1.

Out:

[INFO 03-14 05:05:04] ax.core.experiment: Attached custom parameterizations [{'infill_density': 99.43, 'layer_height': 0.35, 'infill_type': 'rectilinear'}] as trial 2.

Out:

[INFO 03-14 05:05:04] ax.core.experiment: Attached custom parameterizations [{'infill_density': 41.44, 'layer_height': 0.21, 'infill_type': 'rectilinear'}] as trial 3.

Out:

[INFO 03-14 05:05:04] ax.core.experiment: Attached custom parameterizations [{'infill_density': 27.23, 'layer_height': 0.37, 'infill_type': 'honeycomb'}] as trial 4.

Out:

[INFO 03-14 05:05:04] ax.core.experiment: Attached custom parameterizations [{'infill_density': 33.57, 'layer_height': 0.24, 'infill_type': 'honeycomb'}] as trial 5.

Ask for trials

Now, let's have Ax suggest which trials to evaluate so that we can find the optimal configuration more efficiently. We'll do this by calling get_next_trials. We'll make use of Ax's support for parallelism, i.e. suggesting more than one trial at a time -- this can allow us to conduct our experiment much faster! If our lab had three identical 3D printers, we could ask Ax for a batch of three trials and evaluate three different infill density, layer height, and infill types at once.

Note that there will always be a tradeoff between "parallelism" and optimization performance since the quality of a suggested trial is often proportional to the amount of data Ax has access to, see this recipe for a more detailed explanation.

trials = client.get_next_trials(maximum_trials=3)
trials

Out:

{6: {'infill_density': 55.02548380731564,
'layer_height': 0.20387010605198386,
'infill_type': 'honeycomb'},
7: {'infill_density': 3.2958416483009896,
'layer_height': 0.2248493262361199,
'infill_type': 'honeycomb'},
8: {'infill_density': 67.20869259940962,
'layer_height': 0.26006183313027703,
'infill_type': 'honeycomb'}}

Tell Ax the results

In a real-world scenerio we would print parts using the three suggested parameterizations and measure the compressive strength and weight manually, though in this tutorial we will simulate by calling a function. Once the data is collected we will tell Ax the result by calling complete_trial.

def evaluate(
    infill_density: float, layer_height: float, infill_type: str
) -> dict[str, float]:
    strength_map = {"lines": 1, "rectilinear": 2, "gyroid": 5, "honeycomb": 10}
    weight_map = {"lines": 1, "rectilinear": 2, "gyroid": 3, "honeycomb": 9}

    return {
        "compressive_strength": (
            infill_density / layer_height * strength_map[infill_type]
        )
        / 100,
        "weight": (infill_density / layer_height * weight_map[infill_type]) / 200,
    }


for trial_index, parameters in trials.items():
    client.complete_trial(trial_index=trial_index, raw_data=evaluate(**parameters))

We'll repeat this process a number of times. Typically experimentation will continue until a satisfactory combination has been found, experimentation resources (in this example our 3D printing filliment) have been exhausted, or we feel we have spent enough time on optimization.

# Ask Ax for the next trials
trials = client.get_next_trials(maximum_trials=3)
trials

Out:

{9: {'infill_density': 42.48823373473928,
'layer_height': 0.2082183760229763,
'infill_type': 'honeycomb'},
10: {'infill_density': 74.02766374130601,
'layer_height': 0.1,
'infill_type': 'honeycomb'},
11: {'infill_density': 60.614822544399445,
'layer_height': 0.3867126791970298,
'infill_type': 'honeycomb'}}

# Tell Ax the result of those trials
for trial_index, parameters in trials.items():
    client.complete_trial(trial_index=trial_index, raw_data=evaluate(**parameters))

# Ask Ax for the next trials
trials = client.get_next_trials(maximum_trials=3)
trials

Out:

{12: {'infill_density': 86.43057227718593,
'layer_height': 0.3049941512676289,
'infill_type': 'lines'},
13: {'infill_density': 9.008794381901243,
'layer_height': 0.11630844904541969,
'infill_type': 'gyroid'},
14: {'infill_density': 5.503263747296792,
'layer_height': 0.4,
'infill_type': 'lines'}}

# Tell Ax the result of those trials
for trial_index, parameters in trials.items():
    client.complete_trial(trial_index=trial_index, raw_data=evaluate(**parameters))

Step 6: Analyze Results

At any time during the experiment you may analyze the results of the experiment. Most commonly this means extracting the parameterization from the best performing trial you conducted. The best trial will have the optimal objective value without violating any outcome constraints.

best_parameters, prediction, index, name = client.get_best_parameterization()
print("Best Parameters:", best_parameters)
print("Prediction (mean, variance):", prediction)

Out:

[WARNING 03-14 05:06:45] ax.modelbridge.cross_validation: Metrics compressive_strength , weight were unable to be reliably fit.

Out:

[WARNING 03-14 05:06:45] ax.service.utils.best_point: Model fit is poor; falling back on raw data for best point.

Out:

[WARNING 03-14 05:06:45] ax.service.utils.best_point: Model fit is poor and data on objective metric compressive_strength is noisy; interpret best points results carefully.

Out:

Best Parameters: {'infill_density': 42.48823373473928, 'layer_height': 0.2082183760229763, 'infill_type': 'honeycomb'}
Prediction (mean, variance): {'compressive_strength': (20.405611909128915, nan), 'weight': (9.182525359108013, nan)}

Step 7: Compute Analyses

Ax can also produce a number of analyses to help interpret the results of the experiment via client.compute_analyses. Users can manually select which analyses to run, or can allow Ax to select which would be most relevant. In this case Ax selects the following:

• Parrellel Coordinates Plot shows which parameterizations were evaluated and what metric values were observed -- this is useful for getting a high level overview of how thoroughly the search space was explored and which regions tend to produce which outcomes
• Scatter Plot shows both metric values for each trial -- this is useful for understanding the tradeoffs between competing metrics, like compressive strength and weight.
• Interaction Analysis Plot shows which parameters have the largest affect on the function and plots the most important parameters as 1 or 2 dimensional surfaces
• Summary lists all trials generated along with their parameterizations, observations, and miscellaneous metadata

client.compute_analyses()

Out:

[ERROR 03-14 05:06:45] ax.analysis.analysis: Failed to compute ScatterPlot: 'compressive_strength_sem'

Out:

[ERROR 03-14 05:06:45] ax.analysis.analysis: Failed to compute InteractionPlot: Expected the input to have 3 dimensionality (based on the ard_num_dims argument). Got 6.

Parallel Coordinates for compressive_strength

View arm parameterizations with their respective metric values

loading...

Summary for 3d_print_strength_experiment

High-level summary of the Trial-s in this Experiment

	trial_index	arm_name	trial_status	generation_method	generation_node	weight	compressive_strength	infill_density	layer_height	infill_type
0	0	0_0	COMPLETED	nan	nan	0.52	1.74	10.43	0.3	gyroid
1	1	1_0	COMPLETED	nan	nan	2.31	4.63	55.54	0.12	lines
2	2	2_0	COMPLETED	nan	nan	2.84	5.68	99.43	0.35	rectilinear
3	3	3_0	COMPLETED	nan	nan	1.97	3.95	41.44	0.21	rectilinear
4	4	4_0	COMPLETED	nan	nan	3.31	7.36	27.23	0.37	honeycomb
5	5	5_0	COMPLETED	nan	nan	6.29	13.99	33.57	0.24	honeycomb
6	6	6_0	COMPLETED	BoTorch	MBM	12.1457	26.9905	55.0255	0.20387	honeycomb
7	7	7_0	COMPLETED	BoTorch	MBM	0.65961	1.4658	3.29584	0.224849	honeycomb
8	8	8_0	COMPLETED	BoTorch	MBM	11.6295	25.8434	67.2087	0.260062	honeycomb
9	9	9_0	COMPLETED	BoTorch	MBM	9.18253	20.4056	42.4882	0.208218	honeycomb
10	10	10_0	COMPLETED	BoTorch	MBM	33.3124	74.0277	74.0277	0.1	honeycomb
11	11	11_0	COMPLETED	BoTorch	MBM	7.05347	15.6744	60.6148	0.386713	honeycomb
12	12	12_0	COMPLETED	BoTorch	MBM	1.41692	2.83384	86.4306	0.304994	lines
13	13	13_0	COMPLETED	BoTorch	MBM	1.16184	3.8728	9.00879	0.116308	gyroid
14	14	14_0	COMPLETED	BoTorch	MBM	0.068791	0.137582	5.50326	0.4	lines

Cross Validation for compressive_strength

Out-of-sample predictions using leave-one-out CV

loading...

Cross Validation for weight

Out-of-sample predictions using leave-one-out CV

loading...

Out:

[<ax.analysis.plotly.plotly_analysis.PlotlyAnalysisCard at 0x7f5190fe1640>,
<ax.analysis.markdown.markdown_analysis.MarkdownAnalysisCard at 0x7f5190f00fb0>,
<ax.analysis.markdown.markdown_analysis.MarkdownAnalysisCard at 0x7f5190d6d4c0>,
<ax.analysis.plotly.plotly_analysis.PlotlyAnalysisCard at 0x7f5190f6acf0>,
<ax.analysis.plotly.plotly_analysis.PlotlyAnalysisCard at 0x7f5190d6d5e0>,
<ax.analysis.analysis.AnalysisCard at 0x7f5190f03fe0>]

Conclusion

This tutorial demonstrates how to use Ax's Client for optimizing the strength of 3D printed parts in a human-in-the-loop setting. By iteratively collecting data and refining parameters, you can effectively apply black box optimization to real-world experiments.