Author: Sebastian Weber

# Simulation based calibration for RBesT

#### Sebastian Weber

#### Tue Aug 4 12:38:36 2026

This report documents the results of a simulation based calibration (SBC) run for `RBesT`. The calibration data will be generated whenever relevant changes to the `gMAP` function were made. The calibration runs are performed for typical use cases of `gMAP`. These include the three likelihoods (binomial, gaussian & Poisson), a sparse (\(2\) trials) and dense (\(10\) trials) data situation and finally a run with a very/less conservative prior choice for between-trial heterogeniety parameter.

The calibration data presented here has been generated at and with the `RBesT` git version as:

    ## Created:  2026-08-04 08:56:39 UTC
    ## git hash: 6a422b34105f2a5bc00fee4f94bee9d3660abb00
    ## MD5:      3ae7e33cdfdbd010bc2452fb3a6e0541

The MD5 hash of the calibration data file presented here must match the above listed MD5:

    ## /home/runner/work/RBesT/RBesT/inst/sbc/calibration.rds
    ##                     "3ae7e33cdfdbd010bc2452fb3a6e0541"

# Introduction

Simulation based calibration (SBC) is a necessary condition which must be met for any Bayesian analysis with proper priors. The details are presented in Talts, et. al (see https://arxiv.org/abs/1804.06788).

Self-consistency of any Bayesian analysis with a proper prior:

\[ p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta|\tilde{y}) \, p(\tilde{y}|\tilde{\theta}) \, p(\tilde{\theta}) \] \[ \Leftrightarrow p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta,\tilde{y},\tilde{\theta}) \]

SBC procedure:

Repeat \(s=1, ..., S\) times:

1. Sample from the prior \[\tilde{\theta} \sim p(\theta)\]

2. Sample fake data \[\tilde{y} \sim p(y|\tilde{\theta})\]

3. Obtain \(L\) posterior samples \[\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\]

4. Calculate the rank \(r_s\) of the prior draw \(\tilde{\theta}\) wrt to the posterior sample \(\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\) which falls into the range \([0,L]\) out of the possible \(L+1\) ranks. The rank is calculated as \[r_s = \sum_{l=1}^L \mathbb{I}[ \theta_l < \tilde{\theta}]\]

The \(S\) ranks then form a uniform \(0-1\) density and the count in each bin has a binomial distribution with probability of \[p(r \in \mbox{Any Bin}) =\frac{(L+1)}{S}.\]

## Hierarchical intercept only (random-effects intercept) model for binomial, gaussian and Poisson likelihood

Likelihood:

- Binary \[y_i|\theta_{j} \sim \mbox{Bernoulli}(\theta_j), \] \[g(\theta) = \mbox{logit}(\theta)\]
- Normal \[y_i|\theta_{j} \sim \mbox{Normal}(\theta_j, \sigma^2), \] \[g(\theta) = \theta\]
- Poisson \[y_i|\theta_{j} \sim \mbox{Poisson}(\theta_j), \] \[g(\theta) = \log(\theta)\]

Hierarchical prior:

\[ g(\theta_j)|\mu,\tau \sim \mbox{Normal}(\mu, \tau^2)\]

\[\mu \sim \mbox{Normal}(m_\mu, s^2_\mu)\] \[\tau \sim \mbox{Normal}^+(0, s^2_\tau)\]

The fake data simulation function returns for binomial and Poisson data the sum of the responses while for normal the mean summary is used. Please refer to the `sbc_tools.R` and `make_reference_rankhist.R` R programs for the implementation details.

The reference runs are created with \(L=1023\) posterior draws for each replication and a total of \(S=10^4\) replications are run per case. For the evaluation here the results are reduced to \(B=L'+1=64\) bins to ensure a sufficiently large sample size per bin.

# SBC results

## Sampler Diagnostics Overview

family | data_scenario | sd_tau | N | total_divergent | total_divergent_sim_fraction | min_ess | max_Rhat | total_large_Rhat | min_lp_ess_bulk | min_lp_ess_tail
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
binomial | dense | 0.5 | 10000 | 330 | 0.025 | 213 | 1.012 | 0 | 517.597 | 512.380
binomial | dense | 1.0 | 10000 | 434 | 0.029 | 128 | 1.021 | 0 | 558.833 | 545.517
binomial | sparse | 0.5 | 10000 | 1059 | 0.060 | 61 | 1.046 | 0 | 144.899 | 59.269
binomial | sparse | 1.0 | 10000 | 3402 | 0.114 | 5 | 1.367 | 1 | 42.591 | 55.864
gaussian | dense | 0.5 | 10000 | 445 | 0.030 | 167 | 1.019 | 0 | 463.103 | 289.523
gaussian | dense | 1.0 | 10000 | 426 | 0.024 | 83 | 1.027 | 0 | 354.980 | 467.374
gaussian | sparse | 0.5 | 10000 | 20851 | 0.459 | 5 | 1.367 | 1 | 34.611 | 44.792
gaussian | sparse | 1.0 | 10000 | 37103 | 0.402 | 4 | 1.669 | 1 | 8.742 | 40.653
poisson | dense | 0.5 | 10000 | 843 | 0.060 | 184 | 1.026 | 0 | 421.486 | 394.058
poisson | dense | 1.0 | 10000 | 2174 | 0.112 | 14 | 1.136 | 0 | 23.628 | 24.697
poisson | sparse | 0.5 | 10000 | 21120 | 0.389 | 5 | 1.431 | 1 | 8.719 | 62.147
poisson | sparse | 1.0 | 10000 | 37694 | 0.406 | 3 | 2.255 | 1 | 4.745 | 2.309

Note: Large Rhat is defined as exceeding 1.2.

# Summary Statistics

## \(\chi^2\) Statistic, \(\mu\)

| data_scenario | likelihood | sd_tau | parameter | statistic | df | p.value
--- | --- | --- | --- | --- | --- | --- | ---
1 | dense | binomial | 0.5 | mu | 65.587 | 63 | 0.387
13 | dense | binomial | 1 | mu | 45.734 | 63 | 0.950
25 | dense | gaussian | 0.5 | mu | 61.978 | 63 | 0.513
37 | dense | gaussian | 1 | mu | 73.920 | 63 | 0.163
49 | dense | poisson | 0.5 | mu | 43.904 | 63 | 0.968
61 | dense | poisson | 1 | mu | 68.506 | 63 | 0.296
73 | sparse | binomial | 0.5 | mu | 53.005 | 63 | 0.811
78 | sparse | binomial | 1 | mu | 62.016 | 63 | 0.511
83 | sparse | gaussian | 0.5 | mu | 56.922 | 63 | 0.691
88 | sparse | gaussian | 1 | mu | 50.099 | 63 | 0.881
93 | sparse | poisson | 0.5 | mu | 81.165 | 63 | 0.061
98 | sparse | poisson | 1 | mu | 73.165 | 63 | 0.179

## \(\chi^2\) Statistic, \(\tau\)

| data_scenario | likelihood | sd_tau | parameter | statistic | df | p.value
--- | --- | --- | --- | --- | --- | --- | ---
2 | dense | binomial | 0.5 | tau | 74.035 | 63 | 0.161
14 | dense | binomial | 1 | tau | 56.730 | 63 | 0.698
26 | dense | gaussian | 0.5 | tau | 54.874 | 63 | 0.757
38 | dense | gaussian | 1 | tau | 72.883 | 63 | 0.185
50 | dense | poisson | 0.5 | tau | 74.982 | 63 | 0.143
62 | dense | poisson | 1 | tau | 76.045 | 63 | 0.125
74 | sparse | binomial | 0.5 | tau | 71.360 | 63 | 0.220
79 | sparse | binomial | 1 | tau | 65.190 | 63 | 0.400
84 | sparse | gaussian | 0.5 | tau | 62.950 | 63 | 0.478
89 | sparse | gaussian | 1 | tau | 48.166 | 63 | 0.916
94 | sparse | poisson | 0.5 | tau | 60.301 | 63 | 0.573
99 | sparse | poisson | 1 | tau | 60.275 | 63 | 0.574

## \(\chi^2\) Statistic, group estimates \(\theta\)

| data_scenario | likelihood | sd_tau | parameter | statistic | df | p.value
--- | --- | --- | --- | --- | --- | --- | ---
3 | dense | binomial | 0.5 | theta[10] | 51.955 | 63 | 0.838
4 | dense | binomial | 0.5 | theta[1] | 77.453 | 63 | 0.104
5 | dense | binomial | 0.5 | theta[2] | 49.446 | 63 | 0.894
6 | dense | binomial | 0.5 | theta[3] | 65.485 | 63 | 0.391
7 | dense | binomial | 0.5 | theta[4] | 59.405 | 63 | 0.605
8 | dense | binomial | 0.5 | theta[5] | 47.693 | 63 | 0.924
9 | dense | binomial | 0.5 | theta[6] | 63.923 | 63 | 0.444
10 | dense | binomial | 0.5 | theta[7] | 55.642 | 63 | 0.733
11 | dense | binomial | 0.5 | theta[8] | 56.627 | 63 | 0.701
12 | dense | binomial | 0.5 | theta[9] | 70.515 | 63 | 0.241
15 | dense | binomial | 1 | theta[10] | 69.018 | 63 | 0.281
16 | dense | binomial | 1 | theta[1] | 72.666 | 63 | 0.190
17 | dense | binomial | 1 | theta[2] | 50.675 | 63 | 0.868
18 | dense | binomial | 1 | theta[3] | 45.274 | 63 | 0.955
19 | dense | binomial | 1 | theta[4] | 62.029 | 63 | 0.511
20 | dense | binomial | 1 | theta[5] | 92.390 | 63 | 0.009
21 | dense | binomial | 1 | theta[6] | 70.592 | 63 | 0.239
22 | dense | binomial | 1 | theta[7] | 51.584 | 63 | 0.847
23 | dense | binomial | 1 | theta[8] | 52.058 | 63 | 0.836
24 | dense | binomial | 1 | theta[9] | 48.589 | 63 | 0.909
27 | dense | gaussian | 0.5 | theta[10] | 59.110 | 63 | 0.616
28 | dense | gaussian | 0.5 | theta[1] | 80.435 | 63 | 0.068
29 | dense | gaussian | 0.5 | theta[2] | 67.046 | 63 | 0.340
30 | dense | gaussian | 0.5 | theta[3] | 86.131 | 63 | 0.028
31 | dense | gaussian | 0.5 | theta[4] | 65.344 | 63 | 0.395
32 | dense | gaussian | 0.5 | theta[5] | 60.582 | 63 | 0.563
33 | dense | gaussian | 0.5 | theta[6] | 65.869 | 63 | 0.378
34 | dense | gaussian | 0.5 | theta[7] | 84.314 | 63 | 0.038
35 | dense | gaussian | 0.5 | theta[8] | 53.632 | 63 | 0.794
36 | dense | gaussian | 0.5 | theta[9] | 44.442 | 63 | 0.963
39 | dense | gaussian | 1 | theta[10] | 100.390 | 63 | 0.002
40 | dense | gaussian | 1 | theta[1] | 56.755 | 63 | 0.697
41 | dense | gaussian | 1 | theta[2] | 48.870 | 63 | 0.904
42 | dense | gaussian | 1 | theta[3] | 48.013 | 63 | 0.919
43 | dense | gaussian | 1 | theta[4] | 71.488 | 63 | 0.217
44 | dense | gaussian | 1 | theta[5] | 58.086 | 63 | 0.652
45 | dense | gaussian | 1 | theta[6] | 48.410 | 63 | 0.912
46 | dense | gaussian | 1 | theta[7] | 51.008 | 63 | 0.861
47 | dense | gaussian | 1 | theta[8] | 69.018 | 63 | 0.281
48 | dense | gaussian | 1 | theta[9] | 66.035 | 63 | 0.372
51 | dense | poisson | 0.5 | theta[10] | 43.085 | 63 | 0.974
52 | dense | poisson | 0.5 | theta[1] | 84.390 | 63 | 0.037
53 | dense | poisson | 0.5 | theta[2] | 56.256 | 63 | 0.714
54 | dense | poisson | 0.5 | theta[3] | 75.763 | 63 | 0.130
55 | dense | poisson | 0.5 | theta[4] | 65.779 | 63 | 0.381
56 | dense | poisson | 0.5 | theta[5] | 63.040 | 63 | 0.475
57 | dense | poisson | 0.5 | theta[6] | 73.677 | 63 | 0.168
58 | dense | poisson | 0.5 | theta[7] | 55.782 | 63 | 0.729
59 | dense | poisson | 0.5 | theta[8] | 56.538 | 63 | 0.704
60 | dense | poisson | 0.5 | theta[9] | 56.512 | 63 | 0.705
63 | dense | poisson | 1 | theta[10] | 66.176 | 63 | 0.368
64 | dense | poisson | 1 | theta[1] | 49.920 | 63 | 0.884
65 | dense | poisson | 1 | theta[2] | 54.477 | 63 | 0.769
66 | dense | poisson | 1 | theta[3] | 63.386 | 63 | 0.463
67 | dense | poisson | 1 | theta[4] | 62.886 | 63 | 0.480
68 | dense | poisson | 1 | theta[5] | 87.181 | 63 | 0.024
69 | dense | poisson | 1 | theta[6] | 100.493 | 63 | 0.002
70 | dense | poisson | 1 | theta[7] | 56.768 | 63 | 0.697
71 | dense | poisson | 1 | theta[8] | 55.923 | 63 | 0.724
72 | dense | poisson | 1 | theta[9] | 71.578 | 63 | 0.215
75 | sparse | binomial | 0.5 | theta[1] | 61.043 | 63 | 0.546
76 | sparse | binomial | 0.5 | theta[2] | 48.179 | 63 | 0.916
77 | sparse | binomial | 0.5 | theta[3] | 55.731 | 63 | 0.730
80 | sparse | binomial | 1 | theta[1] | 64.576 | 63 | 0.421
81 | sparse | binomial | 1 | theta[2] | 60.518 | 63 | 0.565
82 | sparse | binomial | 1 | theta[3] | 67.264 | 63 | 0.333
85 | sparse | gaussian | 0.5 | theta[1] | 61.133 | 63 | 0.543
86 | sparse | gaussian | 0.5 | theta[2] | 44.954 | 63 | 0.958
87 | sparse | gaussian | 0.5 | theta[3] | 57.587 | 63 | 0.669
90 | sparse | gaussian | 1 | theta[1] | 68.134 | 63 | 0.307
91 | sparse | gaussian | 1 | theta[2] | 79.706 | 63 | 0.076
92 | sparse | gaussian | 1 | theta[3] | 64.678 | 63 | 0.418
95 | sparse | poisson | 0.5 | theta[1] | 63.654 | 63 | 0.453
96 | sparse | poisson | 0.5 | theta[2] | 73.997 | 63 | 0.162
97 | sparse | poisson | 0.5 | theta[3] | 76.762 | 63 | 0.114
100 | sparse | poisson | 1 | theta[1] | 71.680 | 63 | 0.212
101 | sparse | poisson | 1 | theta[2] | 58.099 | 63 | 0.651
102 | sparse | poisson | 1 | theta[3] | 59.277 | 63 | 0.610

## Session Info

    ## R version 4.6.1 (2026-06-24)
    ## Platform: x86_64-pc-linux-gnu
    ## Running under: Ubuntu 24.04.4 LTS
    ##
    ## Matrix products: default
    ## BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
    ## LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so;  LAPACK version 3.12.0
    ##
    ## locale:
    ##  [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8
    ##  [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8
    ##  [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C
    ## [10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C
    ##
    ## time zone: UTC
    ## tzcode source: system (glibc)
    ##
    ## attached base packages:
    ## [1] tools     stats     graphics  grDevices utils     datasets  methods
    ## [8] base
    ##
    ## other attached packages:
    ##  [1] purrr_1.2.2         rstan_2.32.7        StanHeaders_2.32.10
    ##  [4] here_1.0.2          ggplot2_4.0.3       broom_1.0.13
    ##  [7] tidyr_1.3.2         dplyr_1.2.1         assertthat_0.2.1
    ## [10] knitr_1.51
    ##
    ## loaded via a namespace (and not attached):
    ##  [1] sass_0.4.10        generics_0.1.4     digest_0.6.39      magrittr_2.0.5
    ##  [5] evaluate_1.0.5     grid_4.6.1         RColorBrewer_1.1-3 fastmap_1.2.0
    ##  [9] rprojroot_2.1.1    jsonlite_2.0.0     pkgbuild_1.4.8     backports_1.5.1
    ## [13] gridExtra_2.3.1    QuickJSR_1.10.0    scales_1.4.0       codetools_0.2-20
    ## [17] jquerylib_0.1.4    cli_3.6.6          rlang_1.3.0        withr_3.0.3
    ## [21] cachem_1.1.0       yaml_2.3.12        otel_0.2.0         parallel_4.6.1
    ## [25] inline_0.3.21      vctrs_0.7.3        R6_2.6.1           matrixStats_1.5.0
    ## [29] stats4_4.6.1       lifecycle_1.0.5    pkgconfig_2.0.3    RcppParallel_6.2.0
    ## [33] pillar_1.11.1      bslib_0.12.0       gtable_0.3.6       loo_2.10.1
    ## [37] glue_1.8.1         Rcpp_1.1.2         xfun_0.60          tibble_3.3.1
    ## [41] tidyselect_1.2.1   farver_2.1.2       htmltools_0.5.9    rmarkdown_2.31
    ## [45] compiler_4.6.1     S7_0.2.2