% Generated by roxygen2: do not edit by hand
% Please edit documentation in R/chunk.R
\name{chunk_feeder}
\alias{chunk_feeder}
\title{Turn a query into a resettable chunk generator}
\usage{
chunk_feeder(.source)
}
\arguments{
\item{.source}{Either a function of no arguments returning a fresh
\code{vectra_node} each time it is called (e.g. \code{function() tbl_csv("occ.csv") |> select(presence, bio1, bio12)}), or an offloaded node from \code{\link[=offload]{offload()}}.
Every chunk must contain all variables the consumer's formula references.}
}
\value{
A function \verb{function(reset = FALSE)}. With \code{reset = TRUE} it rewinds
and returns \code{invisible(NULL)}; with \code{reset = FALSE} it returns the next
chunk as a data.frame, or \code{NULL} at end of stream.
}
\description{
Wraps a query so a pull-based consumer can read it one chunk at a time and
re-read it from the start as many times as needed. The returned closure
follows the \code{data(reset)} protocol that \code{biglm::bigglm()} expects: called
with \code{reset = TRUE} it rewinds to the beginning of the data, and called with
\code{reset = FALSE} it returns the next chunk as a data.frame, or \code{NULL} once the
data is exhausted. This lets \code{bigglm()} fit a generalized linear model on a
dataset larger than RAM, streaming each iteratively reweighted pass through
the engine without ever holding the full design matrix.
}
\details{
Because a vectra node is consumed as it streams, re-reading requires a fresh
node on each pass. \code{chunk_feeder()} accepts either form: a \emph{factory}, a
function of no arguments that returns a new node each time it is called; or an
offloaded node from \code{\link[=offload]{offload()}}, which is backed by a file and replays from
disk directly. On every \code{reset = TRUE} a fresh stream is started, so the same
query is replayed on each pass.

Prefer feeding an \code{\link[=offload]{offload()}} of the prepared query: the pipeline (scan,
joins, mutate) runs once into the spill, and every reweighted pass is then a
disk scan of the prepared columns rather than a re-run of the pipeline.
}
\examples{
f <- tempfile(fileext = ".vtr")
write_vtr(mtcars, f)

feed <- chunk_feeder(function() tbl(f) |> select(mpg, wt, hp))
feed(reset = TRUE)       # rewind to the start of the stream
first <- feed()          # first chunk as a data.frame
head(first)

\donttest{
# Out-of-core GLM: prepare once with offload(), then bigglm() replays it.
if (requireNamespace("biglm", quietly = TRUE)) {
  s <- offload(tbl(f) |> select(mpg, wt, hp))
  fit <- biglm::bigglm(mpg ~ wt + hp, data = chunk_feeder(s),
                       family = gaussian())
  coef(fit)
}
}
unlink(f)

}
\seealso{
\code{\link[=offload]{offload()}} for the replay cache, and \code{\link[=collect_chunked]{collect_chunked()}} for
single-pass reductions that vectra drives.
}
