ax.utils¶

Equality¶

ax.utils.common.equality.datetime_equals(dt1, dt2)[source]¶

Compare equality of two datetimes, ignoring microseconds.

Return type

bool

ax.utils.common.equality.equality_typechecker(eq_func)[source]¶

A decorator to wrap all __eq__ methods to ensure that the inputs are of the right type.

Return type

Callable

ax.utils.common.equality.same_elements(list1, list2)[source]¶

Compare equality of two lists of core Ax objects.

Assumptions:

– The contents of each list are types that implement __eq__ – The lists do not contain duplicates

Checking equality is then the same as checking that the lists are the same length, and that one is a subset of the other.

Return type

bool

Logger¶

ax.utils.common.logger.get_logger(name)[source]¶

Get an Axlogger.

Sets default level to INFO, instead of WARNING. Adds timestamps to logger messages.

Return type

Logger

Docutils¶

Support functions for sphinx et. al

ax.utils.common.docutils.copy_doc(src)[source]¶

A decorator that copies the docstring of another object

Since sphinx actually loads the python modules to grab the docstrings this works with both sphinx and the help function.

class Cat(Mamal):

  @property
  @copy_doc(Mamal.is_feline)
  def is_feline(self) -> true:
      ...

Return type

~_T

Typeutils¶

ax.utils.common.typeutils.checked_cast(typ, val)[source]¶

Cast a value to a type (with a runtime safety check).

Returns the value unchanged and checks its type at runtime. This signals to the typechecker that the value has the designated type.

Like typing.cast check_cast performs no runtime conversion on its argument, but, unlike typing.cast, checked_cast will throw an error if the value is not of the expected type. The type passed as an argument should be a python class.

Parameters

• typ (Type[~T]) – the type to cast to

• val (~V) – the value that we are casting

Return type

~T

Returns

the val argument, unchanged

ax.utils.common.typeutils.checked_cast_dict(key_typ, value_typ, d)[source]¶

Calls checked_cast on all keys and values in the dictionary.

Return type

Dict[~K, ~V]

ax.utils.common.typeutils.checked_cast_list(typ, l)[source]¶

Calls checked_cast on all items in a list.

Return type

List[~T]

ax.utils.common.typeutils.checked_cast_optional(typ, val)[source]¶

Calls checked_cast only if value is not None.

Return type

Optional[~T]

ax.utils.common.typeutils.checked_cast_to_tuple(typ, val)[source]¶

Cast a value to a union of multiple types (with a runtime safety check). This function is similar to checked_cast, but allows for the type to be defined as a tuple of types, in which case the value is cast as a union of the types in the tuple.

Parameters

• typ (Tuple[Type[~V], …]) – the tuple of types to cast to

• val (~V) – the value that we are casting

Return type

~T

Returns

the val argument, unchanged

ax.utils.common.typeutils.not_none(val)[source]¶

Unbox an optional type.

Parameters

val (Optional[~T]) – the value to cast to a non None type

Retruns:

V: val when val is not None

Throws:

ValueError if val is None

Return type

~T

ax.utils.common.typeutils.numpy_type_to_python_type(value)[source]¶

If value is a Numpy int or float, coerce to a Python int or float. This is necessary because some of our transforms return Numpy values.

Return type

Any

Plotting¶

ax.utils.notebook.plotting.init_notebook_plotting(offline=False)[source]¶

Initialize plotting in notebooks, either in online or offline mode.

ax.utils.notebook.plotting.render(plot_config, inject_helpers=False)[source]¶

Render plot config.

Return type

None

Render¶

ax.utils.report.render.h2_html(text)[source]¶

Embed text in subheading tag.

Return type

str

ax.utils.report.render.h3_html(text)[source]¶

Embed text in subsubheading tag.

Return type

str

ax.utils.report.render.link_html(text, href)[source]¶

Embed text and reference address into link tag.

Return type

str

ax.utils.report.render.list_item_html(text)[source]¶

Embed text in list element tag.

Return type

str

ax.utils.report.render.p_html(text)[source]¶

Embed text in paragraph tag.

Return type

str

ax.utils.report.render.render_report_elements(experiment_name, html_elements, header=True, offline=False)[source]¶

Generate Ax HTML report for a given experiment from HTML elements.

Uses Jinja2 for template. Injects Plotly JS for graph rendering.

Example:

html_elements = [
    h2_html("Subsection with plot"),
    p_html("This is an example paragraph."),
    plot_html(plot_fitted(gp_model, 'perf_metric')),
    h2_html("Subsection with table"),
    pandas_html(data.df),
]
html = render_report_elements('My experiment', html_elements)

Parameters

• experiment_name (str) – the name of the experiment to use for title.

• html_elements (List[str]) – list of HTML strings to render in report body.

• header (bool) – if True, render experiment title as a header. Meant to be used for standalone reports (e.g. via email), as opposed to served on the front-end.

• offline (bool) – if True, entire Plotly library is bundled with report.

Returns

HTML string.

Return type

str

ax.utils.report.render.table_cell_html(text, width=None)[source]¶

Embed text or an HTML element into table cell tag.

Return type

str

ax.utils.report.render.table_heading_cell_html(text)[source]¶

Embed text or an HTML element into table heading cell tag.

Return type

str

ax.utils.report.render.table_html(table_rows)[source]¶

Embed list of HTML elements into table tag.

Return type

str

ax.utils.report.render.table_row_html(table_cells)[source]¶

Embed list of HTML elements into table row tag.

Return type

str

ax.utils.report.render.unordered_list_html(list_items)[source]¶

Embed list of html elements into an unordered list tag.

Return type

str

Statstools¶

ax.utils.stats.statstools.agresti_coull_sem(n_numer, n_denom, prior_successes=2, prior_failures=2)[source]¶

Compute the Agresti-Coull style standard error for a binomial proportion.

Reference: Agresti, Alan, and Brent A. Coull. Approximate Is Better than ‘Exact’ for Interval Estimation of Binomial Proportions.” The American Statistician, vol. 52, no. 2, 1998, pp. 119-126. JSTOR, www.jstor.org/stable/2685469.

Return type

Union[ndarray, float]

ax.utils.stats.statstools.inverse_variance_weight(means, variances, conflicting_noiseless='warn')[source]¶

Perform inverse variance weighting.

Parameters

• means (ndarray) – The means of the observations.

• variances (ndarray) – The variances of the observations.

• conflicting_noiseless (str) – How to handle the case of multiple observations with zero variance but different means. Options are “warn” (default), “ignore” or “raise”.

Return type

Tuple[float, float]

ax.utils.stats.statstools.marginal_effects(df)[source]¶

This method calculates the relative (in %) change in the outcome achieved by using any individual factor level versus randomizing across all factor levels. It does this by estimating a baseline under the experiment by marginalizing over all factors/levels. For each factor level, then, it conditions on that level for the individual factor and then marginalizes over all levels for all other factors.

Parameters

df (DataFrame) – Dataframe containing columns named mean and sem. All other columns are assumed to be factors for which to calculate marginal effects.

Return type

DataFrame

Returns

A dataframe containing columns “Name”, “Level”, “Beta” and “SE”

corresponding to the factor, level, effect and standard error. Results are relativized as percentage changes.

ax.utils.stats.statstools.positive_part_james_stein(means, sems)[source]¶

Estimation method for Positive-part James-Stein estimator.

This method takes a vector of K means (y_i) and standard errors (sigma_i) and calculates the positive-part James Stein estimator.

Resulting estimates are the shrunk means and standard errors. The positive part James-Stein estimator shrinks each constituent average to the grand average:

y_i - phi_i * y_i + phi_i * ybar

The variable phi_i determines the amount of shrinkage. For phi_i = 1, mu_hat is equal to ybar (the mean of all y_i), while for phi_i = 0, mu_hat is equal to y_i. It can be shown that restricting phi_i <= 1 dominates the unrestricted estimator, so this method restricts phi_i in this manner. The amount of shrinkage, phi_i, is determined by:

(K - 3) * sigma2_i / s2

That is, less shrinkage is applied when individual means are estimated with greater precision, and more shrinkage is applied when individual means are very tightly clustered together. We also restrict phi_i to never be larger than 1.

The variance of the mean estimator is:

(1 - phi_i) * sigma2_i + phi * sigma2_i / K + 2 * phi_i ** 2 * (y_i - ybar)^2 / (K - 3)

The first term is the variance component from y_i, the second term is the contribution from the mean of all y_i, and the third term is the contribution from the uncertainty in the sum of squared deviations of y_i from the mean of all y_i.

For more information, see https://fburl.com/empirical_bayes.

Parameters

• means (Union[ndarray, List[float]]) – Means of each arm

• sems (Union[ndarray, List[float]]) – Standard errors of each arm

Returns

Empirical Bayes estimate of each arm’s mean sem_i: Empirical Bayes estimate of each arm’s sem

Return type

mu_hat_i

ax.utils.stats.statstools.relativize(means_t, sems_t, mean_c, sem_c, bias_correction=True, cov_means=0.0, as_percent=False)[source]¶

Ratio estimator based on the delta method.

This uses the delta method (i.e. a Taylor series approximation) to estimate the mean and standard deviation of the sampling distribution of the ratio between test and control – that is, the sampling distribution of an estimator of the true population value under the assumption that the means in test and control have a known covariance:

(mu_t / mu_c) - 1.

Under a second-order Taylor expansion, the sampling distribution of the relative change in empirical means, which is m_t / m_c - 1, is approximately normally distributed with mean

[(mu_t - mu_c) / mu_c] - [(sigma_c)^2 * mu_t] / (mu_c)^3

and variance

(sigma_t / mu_c)^2 - 2 * mu_t _ sigma_tc / mu_c^3 + [(sigma_c * mu_t)^2 / (mu_c)^4]

as the higher terms are assumed to be close to zero in the full Taylor series. To estimate these parameters, we plug in the empirical means and standard errors. This gives us the estimators:

[(m_t - m_c) / m_c] - [(s_c)^2 * m_t] / (m_c)^3

and

(s_t / m_c)^2 - 2 * m_t * s_tc / m_c^3 + [(s_c * m_t)^2 / (m_c)^4]

Note that the delta method does NOT take as input the empirical standard deviation of a metric, but rather the standard error of the mean of that metric – that is, the standard deviation of the metric after division by the square root of the total number of observations.

Parameters

• means_t (Union[ndarray, List[float], float]) – Sample means (test)

• sems_t (Union[ndarray, List[float], float]) – Sample standard errors of the means (test)

• mean_c (float) – Sample mean (control)

• sem_c (float) – Sample standard error of the mean (control)

• cov_means (Union[ndarray, List[float], float]) – Sample covariance between test and control

• as_percent (bool) – If true, return results in percent (* 100)

Returns

Inferred means of the sampling distribution of

the relative change (mean_t / mean_c) - 1

sem_hat: Inferred standard deviation of the sampling

distribution of rel_hat – i.e. the standard error.

Return type

rel_hat

ax.utils.stats.statstools.total_variance(means, variances, sample_sizes)[source]¶

Compute total variance.

Return type

float