Abacus Core Mixins: Diagnostics (diagnostics.py)

This module provides the DiagnosticsMixin class, intended to be inherited by Marketing Mix Model (MMM) classes in Abacus. It offers methods for assessing model fit and predictive performance using standard Bayesian diagnostic tools.

DiagnosticsMixin Class

This mixin assumes the inheriting class has an idata attribute which is an ArviZ InferenceData object containing the results of the model fitting process (including the posterior and log-likelihood).

Methods

compute_elpd

def compute_elpd(self, model_name: str = "MMM", file_path: str = "ELPD.txt"):

Computes the Expected Log Pointwise Predictive Density (ELPD) using Leave-One-Out Cross-Validation (LOO-CV).

ELPD is a measure of a model’s out-of-sample predictive accuracy. Higher values generally indicate better predictive performance. This method uses ArviZ’s az.loo() function to perform the calculation based on the model’s idata.

The computed ELPD value and the detailed summary provided by az.loo() (which includes estimates of the effective number of parameters p_loo and warnings about Pareto k diagnostic values) are saved to the specified text file.

Parameters:

  • model_name (str, optional): A name for the model being evaluated, included in the output file. Defaults to "MMM".

  • file_path (str, optional): The path to the output text file where the ELPD results will be saved. Defaults to "ELPD.txt" in the current working directory. The directory will be created if it doesn’t exist.

Raises:

  • RuntimeError: If the model has not been fitted yet (i.e., self.idata is None or does not contain posterior samples).

Notes:

  • LOO-CV can be computationally intensive, especially for large datasets or complex models.

  • The output file contains the ELPD point estimate and the full ArviZ LOO summary, which should be inspected for potential issues indicated by high Pareto k values.