% Generated by roxygen2: do not edit by hand
% Please edit documentation in R/collect_content.R
\name{collect_content}
\alias{collect_content}
\title{Fetch homepage HTML and text for domains}
\usage{
collect_content(
  domains = NULL,
  delay = 1,
  timeout = 10,
  max_bytes = 2 * 1024^2,
  obey_robots = TRUE,
  max_crawl_delay = 30,
  max_redirects = 5,
  user_agent = rdomains_user_agent()
)
}
\arguments{
\item{domains}{Character vector of domains or URLs.}

\item{delay}{Minimum seconds between requests to the same host. `Crawl-delay` overrides
this upward.}

\item{timeout}{Per-request timeout, seconds.}

\item{max_bytes}{Cap on the response body actually read.}

\item{obey_robots}{Whether to fetch and honour robots.txt. Turning this off is
discouraged and is your responsibility, not the package's.}

\item{max_crawl_delay}{Skip a host that asks for a longer delay than this rather than
sleeping on it.}

\item{max_redirects}{Maximum redirect hops to follow. Every hop is re-validated, so a
redirect cannot be used to reach an address the first check refused.}

\item{user_agent}{Override the identifying user-agent.}
}
\value{
A tibble with one row per input: `domain_name`, `status`, `stage`, `error_code`,
  `retryable`, `http_status`, `final_url`, `fetched_at`, `content_bytes`, `title`,
  `description`, `lang`, `text`, `n_tokens`, `page_state`, `block_vendor`,
  `robots_allowed`, `source_last_published`.
}
\description{
Every requested domain comes back, in order, whether or not it was reachable. Failures
carry a code from [fetch_error_codes()] rather than a bare `NA`.
}
\details{
The crawler identifies itself as `rdomains/<version>`, obeys `robots.txt` including
`Crawl-delay`, spaces requests to the same host, caps the response body, and refuses to
fetch hosts that resolve to private or link-local addresses.
}
\examples{
\dontrun{
res <- collect_content(c("example.com", "wikipedia.org"))
fetch_report(res)

# retry only what is worth retrying
again <- collect_content(res$domain_name[res$retryable])
}
}
\seealso{
[fetch_report()] to summarise the run, [page_signals()] for what the page
  states are, [source_vintage()] for how a live fetch compares with the static lists.
}
