# Automatic Selection and Ensembling

Code

    library(knitr)
    library(tsissm)
    library(xts)
    library(tsaux)
    library(data.table)
    library(zoo)

## Introduction

The `issm_modelspec()` function is the primary entry point for modeling in the `tsissm package`. It defines a model specification with support for automatic selection across multiple configurations. In addition to selecting the single best model (e.g., by AIC), it can optionally return the top N ranked models, enabling downstream model ensembling for filtering, forecasting and simulation.

This vignette demonstrates that functionality using U.S. Advanced Retail Sales data available in the package.

## Advanced Retail Sales

This dataset represents the monthly, non-seasonally adjusted advance estimate of retail trade sales, based on a sub-sample of firms from the larger Monthly Retail Trade Survey.

As shown in the plot below, the series exhibits two prominent structural breaks: one around 2008 during the Global Financial Crisis, and another during the COVID-19 pandemic. To address this, we use the `auto_regressors()` function from the tsaux package, which detects and encodes three types of anomalies: additive outliers, temporary changes, and structural breaks (see this blog post for details).

Modeling such anomalies is important. When ignored, they are often absorbed by other components of the model, leading to biased coefficient estimates and inflated forecast variance.

Code

    data("us_retail_sales")
    opar <- par(mfrow = c(1,1))
    y <- as.xts(us_retail_sales)
    par(mar = c(2,2,2,2))
    plot(as.zoo(y), ylab = "Sales (Mil's S)", xlab = "", main = "Advance Retail Sales")
    grid()

Code

    par(opar)

## Model Selection

For this demonstration, we reserve the last 38 months of data for forecasting evaluation and select the top 4 models for ensembling.

Code

    train <- y["/2021"]
    test <- y["2022/"]
    lambda_pre_estimate <- box_cox(lambda = NA)$transform(train) |> attr("lambda")
    xreg <- auto_regressors(train, frequency = 12, lambda = lambda_pre_estimate, sampling = "months", h = nrow(test), method = "sequential",
                            check.rank = TRUE, discard.cval = 3.5, maxit.iloop = 10, maxit.oloop = 10,
                            forc_dates = index(test))
    spec <- issm_modelspec(train, auto = TRUE, slope = c(TRUE, FALSE), seasonal = TRUE,
                                 seasonal_harmonics = list(c(3,4,5)), xreg = xreg$xreg[index(train), ],
                                 seasonal_frequency = 12, ar = 0:2, ma = 0:2, lambda = lambda_pre_estimate, top_n = 4)
    mod <- spec |> estimate()
    #> Warning in eval_f(x, ...): NaN or Infinite value in likelihood
    #> Warning in eval_f(x, ...): NaN or Infinite value in likelihood
    #> Warning in eval_f(x, ...): NaN or Infinite value in likelihood
    #> Warning in eval_f(x, ...): NaN or Infinite value in likelihood
    #> Warning in eval_f(x, ...): NaN or Infinite value in likelihood
    #> Warning in eval_f(x, ...): NaN or Infinite value in likelihood
    #> Warning in eval_f(x, ...): NaN or Infinite value in likelihood

We first inspect the top selected model:

Code

    summary(mod$models[[1]])
    #> Warning in sqrt(diag(V)): NaNs produced
    #> ISSM Model: AAA(12{5})+ARMA(1,1)+X(8)
    #> Coefficients:
    #>             Estimate Std. Error t value Pr(>|t|)
    #> alpha      3.122e-01  3.878e-02   8.051 8.88e-16 ***
    #> beta       1.048e-08        NaN     NaN      NaN
    #> gamma12.1 -3.452e-03  2.663e-03  -1.296   0.1950
    #> gamma12.2  3.574e-03  1.520e-03   2.352   0.0187 *
    #> theta1    -9.900e-01  1.229e-02 -80.580  < 2e-16 ***
    #> psi1       9.339e-01  3.791e-02  24.633  < 2e-16 ***
    #> kappa.1   -8.376e-01  4.109e-01  -2.039   0.0415 *
    #> kappa.2   -2.707e+00  4.216e-01  -6.420 1.36e-10 ***
    #> kappa.3   -5.139e+00  4.180e-01 -12.292  < 2e-16 ***
    #> kappa.4   -1.785e+00  4.255e-01  -4.194 2.74e-05 ***
    #> kappa.5    3.085e+00  4.538e-01   6.798 1.06e-11 ***
    #> kappa.6    3.676e+00  4.621e-01   7.954 1.78e-15 ***
    #> kappa.7   -2.721e+00  3.464e-01  -7.856 4.00e-15 ***
    #> kappa.8    2.284e+00  3.750e-01   6.090 1.13e-09 ***
    #> ---
    #> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
    #>
    #> N: 360
    #> sigma^2: 0.4512
    #> LogLik: -3614.34,  AIC:  3672.34,  BIC: 7399.38
    #> DAC (%): 83.844
    #> MAPE (%): 1.51402

This model uses 5 harmonics and an ARMA(2,1) structure. Most coefficients appear statistically significant. We can also summarize the top 4 selected models:

Code

    mod$table |> kable()

iter | slope | slope_damped | seasonal | ar | ma | Seasonal12 | variance | distribution | lambda | AIC | MAPE
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
45 | TRUE | FALSE | TRUE | 1 | 1 | 5 | constant | norm | 0.2521608 | 3672.342 | 0.0151402
51 | TRUE | FALSE | TRUE | 1 | 2 | 5 | constant | norm | 0.2521608 | 3674.406 | 0.0150257
47 | TRUE | FALSE | TRUE | 2 | 1 | 5 | constant | norm | 0.2521608 | 3677.111 | 0.0153474
53 | TRUE | FALSE | TRUE | 2 | 2 | 5 | constant | norm | 0.2521608 | 3677.701 | 0.0151564

All top models use 5 harmonics. The main differences lie in the ARMA specification, and one model does not include a slope term. The pairwise correlation among model predictions is very high — expected, given the small structural differences:

Code

    mod$correlation |> kable()

| model_1 | model_2 | model_3 | model_4
--- | --- | --- | --- | ---
model_1 | 1.0000000 | 0.9946482 | 0.9870880 | 0.9934973
model_2 | 0.9946482 | 1.0000000 | 0.9816921 | 0.9930069
model_3 | 0.9870880 | 0.9816921 | 1.0000000 | 0.9884887
model_4 | 0.9934973 | 0.9930069 | 0.9884887 | 1.0000000

## Prediction and Ensembling

The object returned by `estimate()` when top_n > 1 is of class `tsissm.selection`. This class supports `tsfilter()`, `predict()`, and `simulate()` methods, which apply operations to each retained model, returning individual results. These can then be ensembled using the `tsensemble()` function.

Below, we generate forecasts from the top model and from all 4 models, and then ensemble them using equal weights:

Code

    p_top <- mod$models[[1]] |> predict(h = nrow(test), seed = 200, nsim = 4000, newxreg = xreg$xreg[index(test),])
    p_all <- mod |> predict(h = nrow(test), seed = 200, nsim = 4000, newxreg = xreg$xreg[index(test),])
    p_ensemble <- p_all |> tsensemble(weights = rep(1/4,4))

We visualize the top model’s forecast and overlay the ensembled forecast distribution. The actual data is also shown for reference.

Code

    opar <- par(mfrow = c(1,1))
    par(mar = c(2,2,2,2))
    p_top |> plot(n_original = 50, gradient_color =  "aliceblue", interval_type = 2, interval_color = "steelblue", interval_width = 1, median_width = 1.5, xlab = "")
    p_ensemble$distribution |> plot(gradient_color =  "aliceblue", interval_type = 3, interval_color = "green", interval_width = 1, median_width = 1.5,
                                    median_type = 1, median_color = "green", add = TRUE)
    lines(index(test), as.numeric(test), col = 2, lwd = 1.5)
    legend("topleft", c("Historical","Actual (Forecast)", "Top (Forecast)", "Ensemble (Forecast)"), col = c("red","red","black","green"), lty = c(1,1.5,1.5,1.5), bty = "n")

Code

    par(opar)

Note: You can overlay multiple predictive distributions using the `add = TRUE` argument in `plot()` for class `tsmodel.distribution`.

The following table compares performance metrics between the top model and the ensemble forecast. The ensemble consistently outperforms the top model across all metrics:

Code

    tab <- rbind(tsmetrics(p_top, actual = test, alpha = 0.1),
                 tsmetrics(p_ensemble, actual = test, alpha = 0.1))
    rownames(tab) <- c("Top","Ensemble")
    tab |> kable()

| h | MAPE | MASE | MSLRE | BIAS | MIS | CRPS
--- | --- | --- | --- | --- | --- | --- | ---
Top | 38 | 0.0209574 | 0.7466499 | 0.0007027 | -8691.808 | 84544.84 | 9228.015
Ensemble | 38 | 0.0204617 | 0.7295871 | 0.0006565 | -7274.506 | 83071.71 | 8969.007

## Conclusion

The `tsissm` package provides a flexible and robust framework for modeling, forecasting, and simulating time series data using the innovations state space model. By supporting full model enumeration, automatic selection, and simulation-based prediction, it offers a rich toolkit for handling complex seasonal patterns and heteroscedasticity, and the availability of regressors allows the handling of identified outliers and structural breaks.

This vignette demonstrated how to apply automatic model selection with support for retaining the top N models and performing ensemble forecasting. By leveraging both predictive distributions and weighted ensembling, users can obtain more stable and accurate forecasts — particularly valuable when models are structurally similar but individually uncertain.