% Generated by roxygen2: do not edit by hand
% Please edit documentation in R/offload.R
\name{offload}
\alias{offload}
\title{Spill a query to disk and stream it back (the offload functor)}
\usage{
offload(
  x,
  by = NULL,
  n = NULL,
  method = c("auto", "level", "range", "hash"),
  path = NULL,
  compress = c("fast", "small", "none")
)
}
\arguments{
\item{x}{A \code{vectra_node} to materialize.}

\item{by}{Optional name (string) of a partition key column. When supplied,
the result is a partition rather than a single node.}

\item{n}{Number of buckets for \code{method = "range"} or \code{"hash"}. Ignored for a
one-shard-per-value partition.}

\item{method}{Partition strategy: \code{"auto"} (default; one shard per value for
a discrete key, \code{n} ranges for a numeric key), \code{"level"} (one shard per
distinct value), \code{"range"} (\code{n} equal-width value ranges), or \code{"hash"}
(\code{n} buckets by a stable hash of the key, co-locating each key).}

\item{path}{Optional file path for a durable replay-cache spill (used only
when \code{by} is \code{NULL}). When \code{NULL} a temporary file is used and removed when
the returned node is garbage-collected.}

\item{compress}{Compression for spill files, passed to \code{\link[=write_vtr]{write_vtr()}}:
\code{"fast"} (default), \code{"small"}, or \code{"none"}.}
}
\value{
A \code{vectra_node} (no \code{by}) or a \code{vectra_partition} (with \code{by}), each
carrying a cost grade shown by \code{\link[=print]{print()}} and \code{\link[=explain]{explain()}}.
}
\description{
Materializes a query once to disk and returns a stream that holds the same
rows, so every later pass is a disk scan instead of a re-run of the upstream
pipeline. The materialization streams batch by batch, so peak memory stays at
one batch regardless of result size. This is the bridge from the bounded
single-pass world of \code{\link[=collect_chunked]{collect_chunked()}} to out-of-core fits.
}
\details{
With no \code{by}, \code{offload()} returns a \strong{replay cache}: a \code{vectra_node} backed
by one \code{.vtr} file. Feed it to a pull-based consumer such as
\code{biglm::bigglm()} through \code{\link[=chunk_feeder]{chunk_feeder()}}, which accepts an offloaded node
directly, so each iteratively reweighted pass reads the prepared columns from
disk rather than rebuilding them. Bake the selects and mutates into the query
you offload, and replay does no further work.

With \code{by}, \code{offload()} returns a \strong{partition}: the rows split into disjoint
shards, one per key value (discrete key) or per value range (\code{method = "range"}, or any numeric key), written in a single streaming pass. A
partition prints as a list of shards and behaves like one: \code{length()},
\code{names()} (the keys), \code{p[["key"]]} (a shard node), and \code{lapply(p, ...)} all
work. Fold it with \code{\link[=collect_chunked]{collect_chunked()}} (supplying \code{combine}). The union of
the shards reproduces the input; row totals are checked.
}
\examples{
f <- tempfile(fileext = ".vtr")
write_vtr(mtcars, f)

# Replay cache: same rows, now on disk.
s <- offload(tbl(f) |> filter(cyl > 4) |> select(mpg, wt, hp))
nrow(collect(s))

# Partition by a key: a list of per-shard nodes.
p <- offload(tbl(f), by = "cyl")
names(p)
length(p)
nrow(collect(p[[1]]))
unlink(f)

}
\seealso{
\code{\link[=chunk_feeder]{chunk_feeder()}} (accepts an offloaded node), \code{\link[=collect_chunked]{collect_chunked()}}
for the partitioned monoidal reduce, and \code{\link[=arrange]{arrange()}} for the external-sort
instance.
}
