Compare equality of two datetimes, ignoring microseconds.
A decorator to wrap all __eq__ methods to ensure that the inputs are of the right type.
Compare equality of two lists of core Ax objects.
- – The contents of each list are types that implement __eq__ – The lists do not contain duplicates
Checking equality is then the same as checking that the lists are the same length, and that one is a subset of the other.
Support functions for sphinx et. al
A decorator that copies the docstring of another object
sphinxactually loads the python modules to grab the docstrings this works with both
class Cat(Mamal): @property @copy_doc(Mamal.is_feline) def is_feline(self) -> true: ...
Return type: ~_T
Cast a value to a type (with a runtime safety check).
Returns the value unchanged and checks its type at runtime. This signals to the typechecker that the value has the designated type.
check_castperforms no runtime conversion on its argument, but, unlike
checked_castwill throw an error if the value is not of the expected type. The type passed as an argument should be a python class.
- typ (
Type[~V]) – the type to cast to
- val (~V) – the value that we are casting
- typ (
Calls checked_cast on all items in a list.
Calls checked_cast only if value is not None.
Unbox an optional type.
Parameters: val (
Optional[~T]) – the value to cast to a non
- ValueError if
Return type: ~T
render_report_elements(experiment_name, html_elements, header=True, offline=False)¶
Generate Ax HTML report for a given experiment from HTML elements.
Uses Jinja2 for template. Injects Plotly JS for graph rendering.
html_elements = [ h2_html("Subsection with plot"), p_html("This is an example paragraph."), plot_html(plot_fitted(gp_model, 'perf_metric')), h2_html("Subsection with table"), pandas_html(data.df), ] html = render_report_elements('My experiment', html_elements)
- experiment_name (
str) – the name of the experiment to use for title.
- html_elements (
str]) – list of HTML strings to render in report body.
- header (
bool) – if True, render experiment title as a header. Meant to be used for standalone reports (e.g. via email), as opposed to served on the front-end.
- offline (
bool) – if True, entire Plotly library is bundled with report.
- experiment_name (
agresti_coull_sem(n_numer, n_denom, prior_successes=2, prior_failures=2)¶
Compute the Agresti-Coull style standard error for a binomial proportion.
Reference: Agresti, Alan, and Brent A. Coull. Approximate Is Better than ‘Exact’ for Interval Estimation of Binomial Proportions.” The American Statistician, vol. 52, no. 2, 1998, pp. 119-126. JSTOR, www.jstor.org/stable/2685469.
inverse_variance_weight(means, variances, conflicting_noiseless='warn')¶
Perform inverse variance weighting.
- means (
ndarray) – The means of the observations.
- variances (
ndarray) – The variances of the observations.
- conflicting_noiseless (
str) – How to handle the case of multiple observations with zero variance but different means. Options are “warn” (default), “ignore” or “raise”.
- means (
This method calculates the relative (in %) change in the outcome achieved by using any individual factor level versus randomizing across all factor levels. It does this by estimating a baseline under the experiment by marginalizing over all factors/levels. For each factor level, then, it conditions on that level for the individual factor and then marginalizes over all levels for all other factors.
Parameters: df (
DataFrame) – Dataframe containing columns named mean and sem. All other columns are assumed to be factors for which to calculate marginal effects.
- A dataframe containing columns “Name”, “Level”, “Beta” and “SE”
- corresponding to the factor, level, effect and standard error. Results are relativized as percentage changes.
Estimation method for Positive-part James-Stein estimator.
This method takes a vector of K means (y_i) and standard errors (sigma_i) and calculates the positive-part James Stein estimator.
Resulting estimates are the shrunk means and standard errors. The positive part James-Stein estimator shrinks each constituent average to the grand average:y_i - phi_i * y_i + phi_i * ybar
The variable phi_i determines the amount of shrinkage. For phi_i = 1, mu_hat is equal to ybar (the mean of all y_i), while for phi_i = 0, mu_hat is equal to y_i. It can be shown that restricting phi_i <= 1 dominates the unrestricted estimator, so this method restricts phi_i in this manner. The amount of shrinkage, phi_i, is determined by:(K - 3) * sigma2_i / s2
That is, less shrinkage is applied when individual means are estimated with greater precision, and more shrinkage is applied when individual means are very tightly clustered together. We also restrict phi_i to never be larger than 1.
The variance of the mean estimator is:(1 - phi_i) * sigma2_i + phi * sigma2_i / K + 2 * phi_i ** 2 * (y_i - ybar)^2 / (K - 3)
The first term is the variance component from y_i, the second term is the contribution from the mean of all y_i, and the third term is the contribution from the uncertainty in the sum of squared deviations of y_i from the mean of all y_i.
For more information, see https://fburl.com/empirical_bayes.
Empirical Bayes estimate of each arm’s mean sem_i: Empirical Bayes estimate of each arm’s sem
relativize(means_t, sems_t, mean_c, sem_c, bias_correction=True, cov_means=0.0, as_percent=False)¶
Ratio estimator based on the delta method.
This uses the delta method (i.e. a Taylor series approximation) to estimate the mean and standard deviation of the sampling distribution of the ratio between test and control – that is, the sampling distribution of an estimator of the true population value under the assumption that the means in test and control have a known covariance:(mu_t / mu_c) - 1.
Under a second-order Taylor expansion, the sampling distribution of the relative change in empirical means, which is m_t / m_c - 1, is approximately normally distributed with mean[(mu_t - mu_c) / mu_c] - [(sigma_c)^2 * mu_t] / (mu_c)^3
and variance(sigma_t / mu_c)^2 - 2 * mu_t _ sigma_tc / mu_c^3 + [(sigma_c * mu_t)^2 / (mu_c)^4]
as the higher terms are assumed to be close to zero in the full Taylor series. To estimate these parameters, we plug in the empirical means and standard errors. This gives us the estimators:[(m_t - m_c) / m_c] - [(s_c)^2 * m_t] / (m_c)^3
and(s_t / m_c)^2 - 2 * m_t * s_tc / m_c^3 + [(s_c * m_t)^2 / (m_c)^4]
Note that the delta method does NOT take as input the empirical standard deviation of a metric, but rather the standard error of the mean of that metric – that is, the standard deviation of the metric after division by the square root of the total number of observations.
- means_t (
float]) – Sample means (test)
- sems_t (
float]) – Sample standard errors of the means (test)
- mean_c (
float) – Sample mean (control)
- sem_c (
float) – Sample standard error of the mean (control)
- cov_means (
float]) – Sample covariance between test and control
- as_percent (
bool) – If true, return results in percent (* 100)
- Inferred means of the sampling distribution of
the relative change (mean_t / mean_c) - 1
- sem_hat: Inferred standard deviation of the sampling
distribution of rel_hat – i.e. the standard error.
- means_t (