Run from YAML

Objective

Execute a complete dsambayesplus model run from a YAML configuration file and verify the staged output artefacts.

Prerequisites

• dsambayesplus installed locally (see Install and Setup).
• A YAML config file (see Config Schema for structure).
• Data file(s) referenced by the config are accessible.

Steps

1. Set up the environment

Complete the environment setup in Install and Setup (selecting the host library, creating the cache directory, and exporting XDG_CACHE_HOME) before continuing.

2. Validate the configuration (dry run)

Rscript scripts/dsambayesplus.R validate --config config/blm_timeseries.yaml

Expected outcome:

• Exit code 0.
• No Stan compilation or sampling occurs.
• Metadata artefacts are written only if you supply --run-dir or set outputs.run_dir.

If validation fails:

• Check the error message for missing data paths, invalid YAML keys, or formula errors.
• Remember that the authored v2 schema does not expose model.formula; the runner compiles it from target, media, controls, and optional hierarchy / effects.
• Fix the config and re-run validate before proceeding.

3. Run the model

Rscript scripts/dsambayesplus.R run --config config/cre_geo_panel.yaml

Expected outcome:

• Exit code 0.
• Full staged artefact tree under the run directory.

For the bounded fixed-effects runner path:

Rscript scripts/dsambayesplus.R run --config config/fe_panel.yaml

This runs the coefficient-only FE MCMC estimator and writes only the dedicated FE files documented under Output Artefacts. It does not run generic level-scale diagnostics, PSIS model selection, scenario analysis, optimisation, forecasting, or deployment. To add seen-unit expanding-window validation, enable diagnostics.time_series_selection; every fold must contain only holdout units already seen in training and at least two training rows per unit. A completed FE run is not a production qualification; its diagnostics summary records qualification_status: not_assessed.

4. Locate the run directory

The runner prints the run directory path during execution. It follows the pattern:

results/YYYYMMDD_HHMMSS_<run_label>/

5. Verify artefacts

Check that the following stage folders are populated:

The runner files retain legacy decomp_* names, but they are not the mapping/reference-point result returned by interactive decomp(). They fail closed for hierarchical, offset-bearing, and probabilistic-media-transform models, with the reason recorded in artifact_status.csv.

Optional deployment artifact:

• Set outputs.save_deployment_model_rds: true to write 20_model_fit/deployment_model.rds.
• This artifact is a compact deployment package for explicit predict(newdata = ...) and explicit-data decomposition; it does not replace model.rds.
• Supported for model.type: blm, model.type: pooled with fit.method: mcmc, or hierarchical model.type: re / cre with fit.method: mcmc.
• Pooled deployment artifacts score on authored terms and do not require pooling columns in deployment-time newdata / data = ... unless those columns are also ordinary formula terms.
• Hierarchical deployment artifacts are seen-groups-only. Explicit newdata / data = ... must include the raw grouping columns, and decomposition also requires the response source column(s).
• FE does not support deployment_model.rds. Its optional posterior.rds is a compact extracted posterior table, while model.rds is the fitted same-environment analysis object.

6. Quick verification commands

# Check diagnostics overall status
head -1 results/<run_dir>/40_diagnostics/diagnostics_report.csv

# View posterior summary
head results/<run_dir>/30_post_run/posterior_summary.csv

# Count artefact files
find results/<run_dir> -type f | wc -l

Failure handling

For FE runs, missing generic observed.csv, fitted.csv, diagnostics_report.csv, PSIS model-selection, scenario, optimisation, forecast, or deployment files is expected. Use within_design.csv, contrast_residuals.csv, contrast_ppc.csv, level_diagnostics.csv, and diagnostics_summary.txt for the bounded FE runner reporting contract. level_diagnostics.csv is an in-sample conditional diagnostic for retained units, not held-out validation. When FE time-series validation is enabled, use 50_model_selection/tscv_summary.csv and, if requested, tscv_pointwise.csv for held-out seen-unit scoring. The direct R API separately supports conditional retained-unit intercepts, fitted level values, observed level data, and retained-unit prediction after MCMC.

Programmatic API note:

• dsambayesplus::run_from_yaml() can return outcome: completed_with_scenario_analysis_fail when the fit succeeded but the requested scenario analysis failed, or completed_with_artifact_write_fail when a later artefact-writing step failed, including TSCV artefact writes.
• For automation, inspect outcome, postfit_issue, and postfit_message instead of assuming every non-error return is fully successful.
• Diagnostics publish-gate failures still raise dsambayesplus_runtime_error, with the same runner_result attached as condition$result.

Related pages

• CLI Usage, command flags and exit codes
• Config Schema, YAML configuration reference
• Output Artefacts, stage folder contract
• Quickstart, minimal first run

Interpret Diagnostics

Objective

Read and act on the diagnostics report produced by a dsambayesplus runner execution, understanding which checks matter most and what remediation steps to take.

This is the operational triage guide. For the methodological meaning of the gates themselves, see Stage 4: Computation and Sampler and Stage 5: Model Adequacy.

Prerequisites

• A completed runner run execution with artefacts under 40_diagnostics/.
• Familiarity with Diagnostics Gates definitions.

Steps

1. Open the diagnostics report

cat results/<run_dir>/40_diagnostics/diagnostics_report.csv

Each row is one diagnostic check. The key columns are:

2. Check the overall status

The overall status follows a simple rule:

• Any fail → overall fail.
• Any warn (no fails) → overall warn.
• All pass → overall pass.

If the overall status is pass, no enabled check breached its configured threshold. Continue with substantive fit, predictive, sensitivity, identifiability and intended-use review. A pass is not model approval.

3. Triage failing checks

Focus on fail rows first, then warn rows. Use the check phase to prioritise:

4. Common diagnostics and actions

The phase assignments below match the emitted phase field:

Design integrity (P0)

Design conditioning (P1)

Sampler quality (P0, MCMC only)

Residual behaviour (P1)

Boundary and variation checks (P1)

Identifiability gate (P1)

5. Review diagnostic plots

Cross-reference the numeric report with visual diagnostics in 40_diagnostics/:

• Residual diagnostics plot: check for patterns in residuals over time.
• Boundary hits plot: identify which parameters are constrained.
• Latent residual ACF plot: confirm autocorrelation structure.

See Diagnostics Plots for interpretation guidance.

6. Decide on next steps

7. Change policy mode if appropriate

If you are in early model development, consider switching to explore mode to relax thresholds:

diagnostics:
  policy_mode: explore

For production or audit runs, use publish (default) or strict.

Related pages

• Diagnostics Gates, full threshold tables and policy modes
• Diagnostics Plots, visual diagnostic outputs
• Run from YAML, end-to-end runner workflow

Compare Runs

Objective

Compare multiple dsambayesplus runner executions and select a candidate model for reporting or decision-making, using predictive scoring and diagnostic summaries.

This page is a late-stage selection aid, not a full workflow. Use it only after the candidate runs are computationally trustworthy enough to compare. In the principled workflow, that means Stage 4 and Stage 5 work has already been done: the sampler is behaving acceptably, and the model is at least adequate enough to remain a candidate. For the surrounding methodology, see Stage 4: Computation and Sampler and Stage 5: Model Adequacy.

Prerequisites

• Two or more completed runner run executions (MCMC fit method).
• Artefacts under 50_model_selection/ for each run (LOO summary, ELPD outputs).
• Familiarity with Diagnostics Gates and Model Selection Plots.
• Candidate runs that are still eligible after basic diagnostic review. Do not compare obviously broken runs just because they produced LOO outputs.

Steps

1. Collect run directories

Identify the run directories to compare:

results/20260228_083808_blm_synth_kpi_os_hfb01/
results/20260228_084410_blm_synth_kpi_os_hfb01/
results/20260228_084602_blm_synth_kpi_os_hfb01/

2. Verify score comparability

Use identical response transformations and observation sets, and require likelihood_scale = unstandardised_model_response in newly generated loo_summary.csv files. Regenerate schema 3 and earlier scores from retained full fits; refit affected scaled no-intercept models first.

compare_runs() requires matching model_comparison metadata in artifact_schema.yaml: response expression, response scale, observation count, density units and a fingerprint of the ordered raw model response plus date and group labels. Numerical scaling and predictor selection do not change that fingerprint when the retained response rows remain identical. Changing the response, retained rows, row order or grouping labels prevents ranking.

The table reports comparison_status: compatible, missing_metadata, incompatible_data, inconsistent_metadata, or unscored. If any otherwise eligible candidate lacks compatible provenance, all candidate ranks and deltas remain NA. Restrict run_dirs to a compatible comparison set. Failed or unscored runs do not receive a rank or delta.

The writer needs the optional digest package and raw-response metadata retained by current fit preparation. Older fits without that metadata remain readable and scoreable, but regenerating their scores alone does not make them eligible for automatic ranking. Refit to establish current provenance; do not add a schema version or fingerprint by hand. loo_compare_models() applies the same provenance requirement to fitted models and precomputed LOO objects.

3. Compare ELPD scores

The compare_runs() helper ranks runs by expected log predictive density (ELPD):

library(dsambayesplus)
comparison <- compare_runs(
  run_dirs = c(
    "results/20260228_083808_blm_synth_kpi_os_hfb01",
    "results/20260228_084410_blm_synth_kpi_os_hfb01"
  )
)
print(comparison)

The output ranks runs by ELPD (higher is better) and reports Pareto-k diagnostics. When TSCV summaries are present, the table also carries tscv_method, tscv_horizon_weeks, tscv_stride_weeks, tscv_min_train_weeks, and tscv_gap_weeks so you can see whether holdout policies actually match. When 00_run_metadata/artifact_schema.yaml is present, the table also carries artifact_schema_version. Treat this as ranking among plausible candidates, not as an automatic winner-selection rule.

4. Check Pareto-k reliability

Examine the loo_summary.csv in each run’s 50_model_selection/ folder:

cat results/<run_dir>/50_model_selection/loo_summary.csv

Key metrics:

Default PSIS-LOO uses chain-aware likelihood relative efficiency. Public loo(model, r_eff = ...) accepts an explicit override; WAIC needs no relative efficiency. Runner diagnostics.model_selection.max_draws is an upper bound: it retains balanced chronological prefixes from every chain and requires at least six draws per chain. An insufficient budget produces a skipped diagnostic. See the loo relative-efficiency reference.

If many observations have high Pareto-k values, the LOO approximation is unreliable for that run. Consider time-series cross-validation as an alternative.

5. Review time-series CV (if available)

If diagnostics.time_series_selection.enabled: true was configured, check:

cat results/<run_dir>/50_model_selection/tscv_summary.csv

This provides blocked-CV or leave-future-out scores (holdout ELPD, RMSE, SMAPE), optionally with an embargo gap when gap_weeks is configured, and is usually more appropriate for time-series data than standard LOO.

compare_runs() ranks a TSCV result only when every scheduled fold succeeded. It conservatively requires the same full retained response dataset across candidates, as well as the same method, horizon_weeks, stride_weeks, min_train_weeks, gap_weeks, number of folds, and realised holdout start and end dates. If these conditions do not hold, rank_tscv and delta_tscv_elpd remain NA; the helper warns when otherwise eligible runs are not comparable.

compare_runs() also warns when explicit artifact_schema_version values differ across runs. The version warning is separate from the provenance guard: matching version numbers alone never establish comparability.

Time-series selection is advisory in the current runner contract. It is useful for model comparison, but it does not change publish-gate status.

6. Cross-reference diagnostics

For each candidate run, check the diagnostics overall status:

head -1 results/<run_dir>/40_diagnostics/diagnostics_report.csv

A model with better ELPD but failing diagnostics should not be preferred over a model with slightly lower ELPD and passing diagnostics.

If a run is computationally untrustworthy, remove it from contention before you start arguing about small predictive-score differences.

7. Compare fit quality visually

Review the fit time series and scatter plots in 20_model_fit/ for each run:

• Fit time series: does the model track the observed KPI?
• Fit scatter: is the predicted-vs-observed relationship close to the diagonal?
• Posterior forest: are coefficient estimates reasonable and well-identified?

8. Selection decision matrix

Use the matrix in order:

1. Remove runs that are not computationally trustworthy enough to compare.
2. Rank the remaining candidates by predictive evidence.
3. Prefer the run whose coefficients, decomposition, and fit behaviour remain most defensible for the business question.

9. Record the selection

Document the selected run directory and rationale. If using the runner for release evidence, the selected run’s artefacts form part of the evidence pack.

Caveats

• ELPD is not causal validation. Predictive scoring measures in-sample predictive quality, not whether the model identifies causal media effects correctly.
• ELPD is not a substitute for adequacy. Stronger predictive ranking does not rescue a run that is diagnostically broken or substantively implausible.
• Pooled models do not support time-series CV (rejected by config validation).
• Adstock/Hill media transforms are not supported by time-series CV; lower-level scoring aborts if transformed-media paths are used.
• MAP-fitted models do not produce LOO diagnostics. Use MCMC for model comparison.

Related pages

• Diagnostics Gates, diagnostic quality checks
• Model Selection Plots, Pareto-k and ELPD influence plots
• Run from YAML, executing runs for comparison
• Config Schema, model_selection.* YAML keys
• Stage 4: Computation and Sampler, when a run is computationally trustworthy enough to compare
• Stage 5: Model Adequacy, why predictive ranking is only one part of model selection

Debug Run Failures

Objective

Diagnose and resolve the most common failure modes encountered when running dsambayesplus via the YAML/CLI runner.

Prerequisites

• A failed runner execution (non-zero exit code or missing artefacts).
• Access to the terminal output or log from the failed run.
• Familiarity with CLI Usage and Config Schema.

Triage by failure stage

Stage 0: Config resolution failures

Symptoms: runner exits immediately after validate or at the start of run; no run directory created or only 00_run_metadata/ is present.

Quick check:

Rscript scripts/dsambayesplus.R validate --config config/blm_timeseries.yaml

If validate passes, the config is structurally valid.

Stage 1: Stan compilation failures

Symptoms: runner reports compilation errors after “Compiling model”; may reference C++ or Stan syntax errors.

Quick check:

mkdir -p .cache
export XDG_CACHE_HOME="$PWD/.cache"

Stage 2: Data preparation failures

Symptoms: runner fails after compilation but before sampling; error messages reference prep_data_for_fit, model.frame, or scaling.

Stage 3: Sampling failures

Symptoms: runner fails during rstan::sampling() or rstan::optimizing(); may report Stan runtime errors.

Stage 4: Post-fit artefact failures

Symptoms: runner completes sampling but some artefact folders are empty or missing files.

Quick check:

find results/<run_dir> -type f | sort

Compare against the expected artefact list in Output Artefacts.

Stage 5: Plot generation failures

Symptoms: CSV artefacts are present but PNG plot files are missing.

General debugging steps

1. Read the full error message. dsambayesplus uses cli::cli_abort() with descriptive messages that identify the failing function and parameter.

2. Check the terminal run status first. If a run directory was created, inspect 00_run_metadata/run_status.yaml if present to see whether the run ended as fit_failed, completed, completed_with_scenario_analysis_fail, completed_with_publish_gate_fail, or completed_with_artifact_write_fail.

3. Check the resolved and compiled configs. Inspect 00_run_metadata/config.resolved.yaml to see what defaults were applied and 00_run_metadata/config.compiled.yaml to see the internal runner config that was actually passed downstream.

4. Check session info. Inspect 00_run_metadata/session_info.txt for package version mismatches.

5. Clear the Stan cache. Stale compiled models can cause unexpected failures:

   rm -rf .cache/dsambayesplus/

6. Run validate before run. Always validate first to catch config errors before committing to a full MCMC run.

7. Reduce iterations for debugging. Use a small fit.mcmc block or switch temporarily to fit.method: optimise to iterate quickly on schema and data issues.

Related pages

• CLI Usage, exit codes and error modes
• Config Schema, YAML key reference
• Output Artefacts, expected artefact contract
• Install and Setup, environment prerequisites
• Diagnostics Gates, post-fit quality checks