For instance, consider a banner with a title and an image. We are considering two different titles and three different images. A full-factorial experiment will compare all 2*3=6 possible combinations of title and image, to see which version of the banner performs the best.
In this example, we first run an exploratory batch to collect data on all possible combinations. Then we use empirical Bayes to model the data and shrink noisy estimates toward the mean. Next, we use Thompson Sampling to suggest a set of arms (combinations of factors and levels) on which to collect more data. We repeat the process until we have identified the best performing combination(s).
import numpy as np import pandas as pd import sklearn as skl from typing import Dict, Optional, Tuple, Union from ax import Arm, ChoiceParameter, Models, ParameterType, SearchSpace, Experiment, OptimizationConfig, Objective from ax.plot.scatter import plot_fitted from ax.utils.notebook.plotting import render, init_notebook_plotting from ax.utils.stats.statstools import agresti_coull_sem
[INFO 09-15 03:50:20] ax.utils.notebook.plotting: Injecting Plotly library into cell. Do not overwrite or delete cell.