# Crawling a website

    library(crawlee)

This article follows the same path as the Crawlee fundamentals: start from a single page, then teach the crawler to follow links, control its scope, route different page types and discover URLs from a sitemap. The examples target `books.toscrape.com`, a public sandbox built for practising web scraping.

## The model

A crawler owns three things:

- a request queue — a deduplicating, resumable list of URLs to visit;
- one or more handlers — functions run on each fetched page;
- a dataset — the structured records your handlers produce.

You build a crawler with `crawler()` and configure it with `cr_*` verbs that compose through the native pipe (`|>`), then run it with `cr_run()`.

## Your first crawler

Fetch a single page and extract a couple of fields. The handler receives a context object (`ctx`) exposing the parsed page and the action `push_data()`.

    result <- crawler("https://books.toscrape.com/") |>
      cr_on_html(function(ctx) {
        ctx$push_data(list(
          url   = ctx$request$url,
          title = ctx$page |> rvest::html_element("title") |> rvest::html_text2()
        ))
      }) |>
      cr_run() |>
      cr_collect()

    result

## Following links

Real crawls discover new URLs as they go. `ctx$enqueue_links()` extracts links from the current page and adds them to the queue; the crawler keeps going until the queue drains. Because the queue deduplicates by a normalised URL, each page is visited at most once.

    crawler("https://books.toscrape.com/") |>
      cr_on_html(function(ctx) {
        ctx$push_data(list(url = ctx$request$url))
        ctx$enqueue_links() # follow every same-domain link
      }) |>
      cr_options(max_requests = 50) |>
      cr_run()

`enqueue_links()` only follows same-domain links by default, so a crawl cannot wander off across the whole web.

## Controlling scope

You rarely want every link. `enqueue_links()` takes `glob` (a shorthand for `include`), `include`/`exclude` patterns and a `same_domain` flag; the crawler itself enforces `max_depth` and `max_requests`.

    crawler("https://books.toscrape.com/") |>
      cr_options(max_depth = 3, max_requests = 200) |>
      cr_on_html(function(ctx) {
        ctx$push_data(list(url = ctx$request$url, depth = ctx$request$depth))
        ctx$enqueue_links(
          glob    = "*/catalogue/*", # only follow catalogue pages
          exclude = "*/category/*"
        )
      }) |>
      cr_run() |>
      cr_collect()

## Routing different page types

Most sites have a few kinds of page — listings vs. detail pages, say. Give a `label` when enqueuing and register a handler for that label. Listing pages enqueue detail pages; detail pages extract the data.

    books <- crawler("https://books.toscrape.com/") |>
      # listing pages: enqueue book detail pages, labelled "book"
      cr_on_html(function(ctx) {
        ctx$enqueue_links(glob = "*/catalogue/*index.html", label = "book")
        ctx$enqueue_links(glob = "*/page-*.html") # pagination, default handler
      }) |>
      # detail pages
      cr_on_html(label = "book", function(ctx) {
        ctx$push_data(list(
          title = ctx$page |> rvest::html_element("h1") |> rvest::html_text2(),
          price = ctx$page |> rvest::html_element(".price_color") |> rvest::html_text2()
        ))
      }) |>
      cr_run() |>
      cr_collect()

    books

A request’s `label` always wins over the content-kind default, so labelled routing and `cr_on_html()`/`cr_on_pdf()` defaults compose cleanly.

## Crawling from a sitemap

When a site publishes a `sitemap.xml`, you can seed the queue directly from it instead of discovering links page by page — `cr_from_sitemap()` handles sitemap indexes and gzipped sitemaps, and can filter by glob or by `<lastmod>` date.

    crawler() |>
      cr_from_sitemap("https://books.toscrape.com/sitemap.xml", label = "book") |>
      cr_on_html(label = "book", function(ctx) {
        ctx$push_data(list(url = ctx$request$url))
      }) |>
      cr_run() |>
      cr_collect()

The companion `cr_from_rss()` does the same for RSS and Atom feeds.

## Rendering JavaScript pages

If a page builds its content with JavaScript, the plain HTTP backend sees an empty shell. Switch to the headless-browser backend with `cr_use_browser()` (requires the package and a Chrome/Chromium install). Handlers are unchanged; you additionally get `ctx$screenshot()`.

    crawler("https://example.com") |>
      cr_use_browser(wait_selector = ".content") |>
      cr_on_html(function(ctx) {
        ctx$push_data(list(url = ctx$request$url))
        ctx$screenshot()
      }) |>
      cr_run()

## Where next

- Politeness & speed — `robots.txt` is respected by default; `cr_options(delay = )` rate-limits, and `cr_parallel()` fetches concurrently.
- Documents — `cr_on_pdf()` extracts text from PDFs; `ctx$save_body()` stores raw files in a key-value store.
- Reproducible, resumable runs — `cr_persist(dir)` checkpoints the queue and persists the dataset, so an interrupted crawl continues where it left off.
- RAG — `cr_chunk()`, `cr_embed()` and `cr_export()` turn crawled text into a retrieval-ready table.