# srsbench

Evaluation metrics for spaced-repetition schedulers in R: the sample-weighted
binned RMSE (`RMSE(bins)`) used to rank algorithms in the open spaced repetition
benchmark, together with log loss, the area under the ROC curve, and calibration
curves. The metrics take plain vectors of predicted recall probabilities and
observed outcomes, so they can benchmark any scheduler's predictions.

## Install

```r
# install.packages("remotes")
remotes::install_github("chrislongros/srsbench")
```

## Use

Each metric takes one row per review as parallel vectors: the predicted recall
probability `p`, the observed outcome `y` (1 recalled, 0 forgotten), and -- for
`rmse_bins()` -- the interval in days, the review number, and the lapse count.

```r
library(srsbench)

p            <- c(0.95, 0.86, 0.72, 0.60, 0.91)
y            <- c(1,    1,    0,    1,    1)
elapsed_days <- c(1,    6,    20,   4,    15)
i            <- c(2,    3,    4,    2,    3)
lapse        <- c(0,    0,    1,    0,    0)

rmse_bins(p, y, elapsed_days, i, lapse)   # calibration, lower is better
log_loss(p, y)                            # cross-entropy, lower is better
srs_auc(p, y)                             # discrimination, higher is better
calibration_bins(p, y)                    # predicted vs observed by bin
```

To benchmark two schedulers, evaluate each one's predictions on the same reviews
and read off the numbers -- lower `rmse_bins`/`log_loss`, higher `srs_auc` wins.

## Getting the predictions

`srsbench` benchmarks predictions; it does not generate them or read Anki files.
Any algorithm that predicts a recall probability per review can be benchmarked
directly: FSRS (via the `rfsrs` package), Ebisu, half-life regression, DASH,
ACT-R style activation models, neural models, or a plain logistic regression.
Interval schedulers such as SM-2, Leitner or Anki's default do not provide
probabilities.

Produce `p` by running the algorithm over a review log with a time-based
train/test split so the predictions are out-of-sample. Read Anki collections
with `ankiR`. `srsbench` then evaluates the output `(p, y)`.

## Metrics

- `rmse_bins()` -- sample-weighted binned RMSE. Reviews are grouped along three
  log-quantised axes (interval, review number, lapse count) and the mean
  predicted recall is compared with the observed recall rate in each bin.
- `log_loss()` -- binary cross-entropy.
- `srs_auc()` -- rank-based area under the ROC curve.
- `calibration_bins()` -- predicted probability versus observed recall in
  equal-width bins, for calibration plots.

## Attribution

`RMSE(bins)` is the metric defined by the
[open spaced repetition benchmark](https://github.com/open-spaced-repetition/srs-benchmark);
`rmse_bins()` is an independent R implementation of its `rmse_matrix` definition.
The log loss, ROC AUC and calibration metrics are standard.

## License

MIT.
