% Generated by roxygen2: do not edit by hand
% Please edit documentation in R/fuzz.R
\name{fuzz}
\alias{fuzz}
\title{Fuzz-test the specified functions}
\usage{
fuzz(
  funs,
  what = test_inputs(),
  args = NULL,
  package = NULL,
  listify_what = FALSE,
  ignore_patterns = "",
  ignore_warnings = FALSE,
  daemons = 2L,
  timeout = 2
)
}
\arguments{
\item{funs}{A character vector of function names to test. If the vector has
a \code{"package"} attribute and no \code{package} argument is given, functions
are loaded from the namespace specified in that attribute.}

\item{what}{A list of objects; each is used, in turn, to modify the list of
arguments in \code{args} before calling each of the functions in \code{funs}.
If no inputs are provided, a default set of inputs generated by
\link{test_inputs} will be used. If set to \code{NULL}, then \code{args} must be
specified, and all functions will be called with that exact list of
arguments with no fuzzing occurring.}

\item{args}{A list of arguments to pass to the functions being fuzzed. Each
element in the list is in turn replaced by each object in \code{what},
then each modified argument list is used to fuzz the functions in
\code{funs}. Argument names are preserved and used as named arguments in
the fuzzed functions. If \code{NULL} (default), only the first argument of
each function is fuzzed.}

\item{package}{Name of the package where functions are searched. If \code{NULL}
(default), the function first checks the \code{"package"} attribute of
\code{funs}, and if that is not set, names are searched in the global
namespace.}

\item{listify_what}{Whether each input in \code{what} should also be tested
in its listified version (\code{FALSE} by default). When set to \code{TRUE},
if \code{what} is \code{list(x = x)}, the function will operate as if it
were \code{list(x = x, "list(x)" = list(x))}, for any input object \code{x}.}

\item{ignore_patterns}{One or more strings containing regular expressions
to match the errors to ignore. The strings "unused argument" and
"is missing, with no default" are always ignored.}

\item{ignore_warnings}{Whether warnings should be ignored (\code{FALSE} by
default).}

\item{daemons}{Number of daemons to use (2 by default). As many \code{mirai}
daemons as specified will be started when entering the function and
closed at the end, unless active daemons are already available, in
which case the argument is ignored and the active daemons are used.}

\item{timeout}{Number of seconds (2 by default) after which the function
being fuzzed is interrupted with result status set to "OK".}
}
\value{
An object of class \code{cbtf} that stores the results obtained for each of the
functions tested. This contains the following fields:
\item{runs}{a list of data frames, each containing the results of fuzzing
all the functions in \code{funs} with one of the inputs in \code{what}. The
data frame contains the following columns and attributes:\cr
- \code{res}: The result of the fuzz test, see below for the possible
values.\cr
- \code{msg}: The error or warning message returned by the function, if
any.\cr
- \verb{attr(*, "what")}: The character representation of the input
tested.
}
\item{funs}{a vector of names of the functions tested.}
\item{args}{a named list of arguments, with names generated by deparsing
the \code{args} argument if not already provided.}
\item{package}{a character string specifying the package name where
function names were searched, or \code{NA} if none was provided.}
\item{ignore_patterns}{The value of the \code{ignore_patterns} argument.}
\item{ignore_warnings}{The value of the \code{ignore_warnings} argument.}

The \code{res} column in each of the data frames in the \verb{$runs} field can
contain the following values:
\itemize{
\item \strong{OK}: either no error or warning was produced (in which case, the \code{msg}
entry is left blank), or it was whitelisted (in which case, the message
received is stored in \code{msg}), or it was timed out (in which case, \code{msg}
records that a timeout was applied).
\item \strong{SKIP}: no test was run, either because the given name cannot be found, or
it doesn't correspond to a function, or the function accepts no arguments,
or more arguments were provided than the function accepts;
the exact reason is given in \code{msg}.
\item \strong{WARN}: a warning was thrown for which no whitelisting occurred and
\code{ignore_warnings = FALSE}; its message is stored in \code{msg}.
\item \strong{FAIL}: an error was thrown for which no whitelisting occurred; its message
is stored in \code{msg}.
}
}
\description{
The fuzzer calls each function in \code{funs} with the argument list provided in
\code{args} (each of its elements in turn modified by each object in \code{what}) and
records any errors or warnings that are thrown. If no error occurs within
the first \code{timeout} seconds, the execution of the function being fuzzed is
interrupted and the next one is started.
}
\details{
\subsection{Multiple arguments}{

A list of arguments to be passed to the functions being fuzzed can be
provided via the \code{args} argument. Each element in that list is modified in
turn by each object in \code{what} and the resulting list of arguments is then
passed to each function via \code{do.call()}. If more arguments are given than
the number of formal arguments accepted by a function, that function will
produce a "SKIP" result.

If arguments are named, they will be passed with their names to the fuzzed
functions. If a function doesn't have a formal argument of that name and
doesn't accept \code{...}, then the standard R behaviour is to return an "unused
argument" error. This is whitelisted by default in \code{fuzz()}, and the
corresponding result status is set to "OK".

It is possible to define arguments that should remain unchanged while
fuzzing by prefixing their names with \code{..}. These arguments will use the
values assigned in \code{args} without modification. For example, to ensure that
the argument \code{na.rm} is always set to \code{TRUE}, it should be specified as
\code{..na.rm = TRUE} in \code{args}. If all elements in \code{args} are fixed, \code{what} is
ignored and all functions in \code{funs} will be called with the provided \code{args}
list.
}

\subsection{Parallel execution}{

The implementation uses \code{mirai} as a backend to execute tasks asynchronously
in parallel worker processes. The function can start a pool of persistent
background processes (daemons) of size given by the \code{daemons} argument
(note that starting more daemons than available cores yields no benefit).
Alternatively, the function can also make use of already active daemons
started with the \link[mirai:daemons]{mirai::daemons} function: this allows to control in
greater detail the number of processes to use, which can also be remote.
}

\subsection{Whitelisting}{

In order to reduce the number of false positive results produced, this
function applies the following set of rules, to establish if an error or
warning condition should be ignored (whitelisting):
\itemize{
\item If the name of the function appears in the error or warning message, as
it is considered that the condition has been handled by the developer.
\item If the error or warning message contains the text "is missing, with no
default", which is produced when a missing argument is used without a
value being assigned to it.
\item If the error or warning message contains any of the patterns specified
in \code{ignore_patterns}. In case of multiple warnings being captured, the
result is whitelisted only if \emph{all} warnings match any of the patterns.
\item If a warning is thrown but \code{ignore_warnings = TRUE} is set.
}

In all whitelisted cases, the result is "OK", and the message that
was received is stored in the \verb{$msg} field (see the \emph{Value} section).

\emph{Note:} Whitelisting can also be applied post-hoc on the results of a fuzz
run using the \link{whitelist} function.
}
}
\examples{
## set up persistent background processes
mirai::daemons(2L)

## this should produce no errors
res <- fuzz(funs = c("list", "matrix", "mean"),
            what = test_inputs(c("numeric", "raw")))
summary(res)

## display all results even for successful tests
print(res, show = "all")

## this will catch an error (false positive)
fuzz(funs = "matrix",  what = test_inputs("scalar"))

## apply a whitelist pattern to remove the false positive
fuzz(funs = "matrix",  what = test_inputs("scalar"),
     ignore_patterns = "'data' must be of a vector type")

## close the background processes
mirai::daemons(0L)

}
\seealso{
\link{get_exported_functions}, \link{test_inputs}, \link{whitelist},
\link{summary.cbtf}, \link{print.cbtf}
}
