% Generated by roxygen2: do not edit by hand
% Please edit documentation in R/tuning_spaces_rtdl.R
\name{mlr_tuning_spaces_rtdl}
\alias{mlr_tuning_spaces_rtdl}
\alias{mlr_tuning_spaces_classif.mlp.rtdl}
\alias{mlr_tuning_spaces_classif.tab_resnet.rtdl}
\alias{mlr_tuning_spaces_classif.ft_transformer.rtdl}
\alias{mlr_tuning_spaces_regr.mlp.rtdl}
\alias{mlr_tuning_spaces_regr.tab_resnet.rtdl}
\alias{mlr_tuning_spaces_regr.ft_transformer.rtdl}
\title{Deep Learning Tuning Spaces from Yandex's RTDL}
\source{
Gorishniy Y, Rubachev I, Khrulkov V, Babenko A (2021).
\dQuote{Revisiting Deep Learning  for Tabular Data.}
\emph{arXiv}, \bold{2106.11959}.
}
\description{
Tuning spaces for deep neural network architectures from the Gorishniy (2021) article.

These tuning spaces require optimizers that have a \code{weight_decay} parameter, such as AdamW or any of the other optimizers built into \code{mlr3torch}.

When the article suggests multiple ranges for a given hyperparameter, these tuning spaces choose the widest range.

The FT-Transformer tuning space disables weight decay for all bias parameters, matching the implementation provided by the authors in the rtdl-revisiting-models package.
However, this differs from the experiments described in the article, which states that the

For the FT-Transformer, if training is unstable, consider a combination of standardizing features, using an adaptive optimizer (e.g. Adam), reducing the learning rate,
and using a learning rate scheduler.
}
\section{MLP tuning space}{

\itemize{
\item n_layers \eqn{[1, 16]}{[1, 16]}
\item neurons -
\item p \eqn{[0, 0.5]}{[0, 0.5]}
\item opt.lr \eqn{[1e-05, 0.01]}{[1e-05, 0.01]} Logscale
\item opt.weight_decay \eqn{[1e-06, 0.001]}{[1e-06, 0.001]} Logscale
\item epochs \eqn{[1, 100]}{[1, 100]}
\item patience 17
}
}

\section{Tabular ResNet tuning space}{

\itemize{
\item n_blocks \eqn{[1, 16]}{[1, 16]}
\item d_block \eqn{[64, 1024]}{[64, 1024]}
\item d_hidden_multiplier \eqn{[1, 4]}{[1, 4]}
\item dropout1 \eqn{[0, 0.5]}{[0, 0.5]}
\item dropout2 \eqn{[0, 0.5]}{[0, 0.5]}
\item opt.lr \eqn{[1e-05, 0.01]}{[1e-05, 0.01]} Logscale
\item opt.weight_decay \eqn{[1e-06, 0.001]}{[1e-06, 0.001]} Logscale
\item epochs \eqn{[1, 100]}{[1, 100]}
\item patience 17
}
}

\section{FT-Transformer tuning space}{

\itemize{
\item n_blocks \eqn{[1, 6]}{[1, 6]}
\item d_token \eqn{[8, 64]}{[8, 64]}
\item attention_n_heads 8
\item residual_dropout \eqn{[0, 0.2]}{[0, 0.2]}
\item attention_dropout \eqn{[0, 0.5]}{[0, 0.5]}
\item ffn_dropout \eqn{[0, 0.5]}{[0, 0.5]}
\item ffn_d_hidden_multiplier \eqn{[0.666666666666667, 2.66666666666667]}{[0.666666666666667, 2.66666666666667]}
\item opt.lr \eqn{[1e-05, 1e-04]}{[1e-05, 1e-04]} Logscale
\item opt.weight_decay \eqn{[1e-06, 0.001]}{[1e-06, 0.001]} Logscale
\item opt.param_groups -
\item epochs \eqn{[1, 100]}{[1, 100]}
\item patience 17
}

In the FT-Transformer, the validation-related parameters must still be set manually, via e.g. \code{lts("regr.ft_transformer.rtdl")$get_learner(validate = 0.2, measures_valid = msr("regr.rmse"))}.
}

