Guide: Interpreting Model Results

This guide helps you understand the various outputs generated by an Abacus MMM run, focusing on the visualisations and key diagnostic metrics.

(For detailed descriptions of specific output file formats, see the Output Files Reference.)

Key Areas of Analysis

  1. Model Fit & Convergence: Did the model fit the data well and did the MCMC sampler converge reliably?

  2. Channel Performance: Which channels are most effective? What is their ROI? How do they respond to spend?

  3. Contribution Breakdown: What drives the target variable (baseline vs. media vs. controls)?

  4. Optimisation (Optional): If the optimisation step is run, what budget shifts are recommended and what is the potential impact?

Interpreting Visualisations

Channel Performance & Optimisation Plots

  • budget_optimisation.png (If Optimisation Run):

    • Purpose: Shows the optimiser’s recommended budget shifts and the expected impact, if the optimisation step was performed after model fitting.

    • Interpretation: Compare “Initial” bars (historical average spend) to “Optimised” bars. Look for channels where the optimised budget is significantly higher (suggesting underspend) or lower (suggesting overspend/inefficiency). Compare contribution bars to see the estimated lift from reallocation.

  • channel_contribution_as_function_of_cost_share.png (Potentially Not Generated):

    • Purpose: (If generated) Shows how the total contribution from a channel changes as its total spend is scaled up or down (relative to the historical total spend for that channel, where delta=1 represents the historical total).

    • Interpretation: (If generated) Observe the shape of the mean curve and the width of the HDI band. Steep initial slope indicates high efficiency at low spend. Flattening indicates saturation. The HDI shows the uncertainty in the contribution estimate at different spend levels. The vertical line at delta=1 shows the contribution at the historical total spend level. (Note: The code generating this plot might be commented out in standard runs).

  • media_contribution_mean.png / media_contribution_median.png:

    • Purpose: Shows the average percentage contribution of each channel to the total target variable over the modelled period. See also media_performance_effect.csv.

    • Interpretation: Provides a high-level view of channel importance. Compare the mean/median values and the HDI error bars. Channels with higher bars contribute more on average. Wide error bars indicate more uncertainty about that channel’s contribution.

  • media_contribution_share.png:

    • Purpose: Shows the distribution of the share of contribution for each channel (relative to the total contribution from all channels).

    • Interpretation: Similar to the mean/median plots but focuses on the distribution of relative importance. Useful for understanding the range of likely contribution shares.

  • media_roi_mean.png / media_roi_median.png:

    • Purpose: Shows the average Return on Investment (Contribution / Spend) for each channel. See also media_performance_roi.csv.

    • Interpretation: Key metric for channel efficiency. Higher bars mean more contribution generated per unit of spend. Compare channels directly. Consider both mean and median, as ROI distributions can be skewed.

  • roi_distribution.png:

    • Purpose: Shows the full posterior distribution of ROI for each channel.

    • Interpretation: Provides a much richer view than just mean/median. Look at the shape, peak, and width of each distribution. A wide distribution indicates high uncertainty in the ROI estimate. Compare where the bulk of the probability mass lies for different channels. See also media_performance_cost_per_target.csv for the inverse metric (Cost Per Target).

Model Fit & Diagnostics Plots

  • model_fit_predictions.png:

    • Purpose: Assesses how well the model’s posterior predictions match the actual observed data. If a train/test split was used (train_test_ratio < 1.0), this plot typically shows the fit on the test set.

    • Interpretation: Look for the predicted line (median or mean) closely tracking the actual data points. Check if actual data mostly falls within the HDI bands (e.g., 95%). Systematic deviations indicate potential issues. R-squared values (often calculated separately for train/test) provide quantitative fit measures.

  • model_priors_and_posteriors.png:

    • Purpose: Diagnoses MCMC sampler convergence and shows prior vs. posterior distributions for key parameters. Generated by az.plot_trace.

    • Interpretation: Left side shows the prior (if available) and posterior distribution; right side shows the trace plot (parameter value across iterations). Chains in the trace plot should mix well (look like “fuzzy caterpillars”) and be stationary. The posterior distributions should be unimodal and reasonably well-defined. Compare posterior to prior to see how much the data updated beliefs. Bimodal distributions or wandering traces indicate convergence problems. Check R-hat and ESS values in model_summary.csv.

Response Curves and Decomposition Plots

  • response_curves.png:

    • Purpose: Visualises the estimated direct contribution curves for each channel based on spend/input, incorporating adstock and saturation effects. Generated by mmm.plot_direct_contribution_curves.

    • Interpretation: Shows diminishing returns. Analyse the shape: how quickly does it rise? Where does it start to flatten (saturate)? Helps understand marginal ROI at different spend levels. HDI bands show uncertainty in the curve shape.

  • waterfall_plot_components_decomposition.png:

    • Purpose: Breaks down the average prediction into the additive contributions from each model component (baseline, channels, controls).

    • Interpretation: Shows the relative magnitude of each component’s average contribution to the overall target variable value. Helps understand the key drivers on average.

  • components_contributions.png:

    • Purpose: Shows the time-varying contributions of different model component groups (e.g., baseline, total channel effect, total control effect) over the entire data period. See also all_decomp.csv.

    • Interpretation: Helps understand how the influence of different factors changes over time. Observe trends and the relative size of the HDI bands (uncertainty) for each component group.

Key Metrics & Diagnostics

  • R-hat: Should be close to 1.0 (e.g., < 1.01 or < 1.05) for all parameters, indicating MCMC chains have converged to the same distribution. Check the r_hat column in model_summary.csv.

  • ESS (Effective Sample Size): Should be sufficiently high (e.g., > 400 per chain, or > 10% of total samples) indicating enough independent samples were drawn from the posterior. Check ess_bulk and ess_tail columns in model_summary.csv.

  • LOO / ELPD: Calculate using az.loo(idata) (see Cross-Validation Guide). Compare ELPD values between different models fitted on the same data. Higher ELPD is better. Consider p_loo (complexity) and Pareto k diagnostics.

  • Posterior Predictive Checks: Visually inspect model_fit_predictions.png. Do the predictions look reasonable? Do the actual data points fall within the model’s prediction intervals (HDIs)? Check R-squared values if calculated.

  • Prior/Posterior Updates: Check model_priors_and_posteriors.png. Did the data inform the parameters (posterior narrower than prior)? Are posteriors sensible?

  • ROI & Contribution: Examine model_summary.csv (for parameter means/medians/HDIs), all_decomp.csv (for time-series contributions), media_performance_effect.csv, media_performance_roi.csv, media_performance_cost_per_target.csv, and related plots (media_contribution_mean.png, roi_distribution.png, etc.). Do the estimated ROIs and contributions make business sense? Are the uncertainty intervals (HDIs) reasonably narrow?

General Interpretation Notes

  • HDI (Highest Density Interval): Represents the range where a parameter or prediction is most likely to fall (e.g., a 90% HDI contains 90% of the posterior probability mass). Wider intervals mean more uncertainty.

  • Mean vs. Median: For skewed distributions (like ROI), the median can be a more robust measure of central tendency than the mean. Check both where available.

  • Context is Key: Always interpret results in the context of business knowledge and the specific data used. Statistical significance doesn’t always equal practical importance.

  • Assumptions: Be aware of the model’s assumptions (e.g., chosen adstock/saturation forms, linearity of controls, likelihood distribution). Check if these assumptions seem reasonable given the diagnostic plots.