Data

Fetching Data

Metrics provide an interface for fetching data for an experiment or trial. Experiment objectives and outcome constraints are special types of metrics, and you can also attach additional metrics for tracking purposes.

Each metric is responsible for fetching its own data. Thus, all metric classes must implement the method fetch_trial_data, which accepts a Trial and returns an instance of Data, a wrapper around a Pandas DataFrame.

To fetch data for an experiment or trial, use exp.fetch_data or trial.fetch_data. These methods fetch data for all metrics on the experiment and then combine the results into a new aggregate Data instance.

Each row of the final DataFrame represents the evaluation of an arm on a metric. As such, the required columns are: arm_name, metric_name, mean, and sem. Additional optional columns are also supported: trial_index, start_time, and end_time.

arm_name	metric_name	mean	sem
0_0	metric1	...	...
0_0	metric2	...	...
0_1	metric1	...	...
0_1	metric2	...	...

Adding Your Own Metric

Our base Metric class is meant to be subclassed. Subclasses must provide an implementation of fetch_trial_data.

An example of a custom metric:

import pandas as pd
from ax import Metric

class CustomMetric(Metric):

    def fetch_trial_data(self, trial, **kwargs):
        records = []
        for arm_name, arm in trial.arms_by_name.items():
            records.append({
                "arm_name": arm_name,
                "metric_name": self.name,
                "mean": 0.0,  # mean value of this metric when this arm is used
                "sem": 0.0,  # standard error of the above mean
                "trial_index": trial.index,
            })
        return Data(df=pd.DataFrame.from_records(records))

Advanced Data Fetching

If you need to fetch data for multiple metrics or trials simultaneously, your Metric can implement the methods fetch_experiment_data, fetch_trial_data_multi, and fetch_experiment_data_multi. The default implementations of these methods use fetch_trial_data internally, but can be overridden if bulk data fetching is more appropriate for the metric type.