# scopusflow 0.4.0

A release about reporting a search and about trusting the cache. The package now
writes a search up as the reproducible record a systematic review has to publish,
and a family of silent cache defects that served the wrong records is fixed.

## Writing up a search

* `scopus_search_report()` assembles the reproducible search-strategy record a
  systematic review has to report, from a harvest or from a plan not yet run. It
  prints as a readable report, formats as a methods paragraph fit to paste into a
  manuscript (`format(report, style = "paragraph")`), and writes as Markdown when
  `file` is supplied. It carries the database and platform, the field-wrapped
  query of every cell, the year limits, view, page size and paging mode, the date
  searched, the records retrieved against the number the API reported as matching,
  per-cell completeness, duplicates removed, the software versions, a runnable
  snippet that rebuilds the plan, and an explicit map of which PRISMA-S items the
  package can supply and which only you can. The reporting standard is PRISMA-S
  (Rethlefsen et al., 2021, Systematic Reviews, 10, 39), with the identification
  counts of the PRISMA 2020 flow diagram.

  Because the output is written for a published methods section, the report states
  only what the objects record. An absent retrieval time is never replaced by the
  current one, a completeness figure is never given for a harvest whose reported
  total is unknown, duplicates are counted only where a merge recorded removing
  them, and every unverifiable field says it is unrecorded, in words. Ten of the
  sixteen PRISMA-S items, among them peer review of the strategy, grey literature
  and any other database searched, are listed as yours to supply, since the
  package has no evidence for them.

* `scopus_fetch_plan()` now carries the per-cell accounting onto its result, as
  the `cell_totals` attribute, and sums it into `total_results` when (and only
  when) every cell reported a total. A cell that comes back shorter than the API
  said it should now warns, which the Python twin has done since 0.3.0, and the
  warning text is byte-identical between the two. A cell stopped by `max_results`
  is short by request and does not warn.

* `scopus_combine()` records the merge in the `combined` attribute: how many
  records went in, how many were kept, how many were removed and whether
  de-duplication ran. That count exists only at the moment of the merge, and
  PRISMA-S asks for it.

* A retrieval also records how it was paged, as the `paging` attribute, since an
  offset-paged query stops at the API's ceiling where a cursor-paged one does not.

## Cache defects that returned the wrong records

All were silent, and all are fixed.

* A year-partitioned plan could be served another plan's years, because checkpoints
  were keyed by the cell's position alone and every cell of such a plan carries the
  same query (the year travels separately, as the API's `date` parameter), so the guard
  meant to reject a foreign checkpoint could not tell 2015's from 2016's. A plan over
  2016:2017 pointed at a cache written by a plan over 2015:2016 returned the 2015
  records and reported "loaded from cache". Checkpoints are now named
  `cell-NNN-YYYY.rds` and carry a manifest (query, date, view, page size and record
  cap) that resume compares against. Existing caches become invisible and refetch once,
  which spends a little quota to buy back correctness.
* A cell cached under a `max_results` cap was served to a later request asking for
  more. The manifest now records the cap and whether it actually bit, so a truncated
  checkpoint is a miss for a wider request while an untruncated one stays usable for any
  request. The Shiny app's cache key gained the record cap and now hashes the query,
  where it used to truncate it at 80 characters, so two long queries sharing a prefix no
  longer share a directory.
* A zero-row checkpoint answered for any query. The guard that recognises a foreign
  checkpoint reads the cached rows' `query` column, which an empty cell has no rows to
  carry, and the manifest comparison never consulted its own copy of the query, so a
  plan that legitimately found nothing left a checkpoint that a different plan pointed
  at the same `cache_dir` would load as its own empty result. The manifest's query must
  now match as well.
* A failed abstract retrieval was checkpointed as if it were data. A rate-limited,
  server-error or offline attempt in `scopus_abstract()` degrades to an all-NA row so
  the batch survives, but that row was written to the cache, so a resumed run read the
  failure back from disk and never retried it. Only rows parsed from successful
  responses are checkpointed now.
* Two identifiers could share one abstract checkpoint, because the cache filename
  reduces every non-alphanumeric character to `_`, so distinct identifiers such as
  `10.1/a.b` and `10.1/a-b` collide on the same file, and resume never asked which
  identifier a cached row belonged to. The cached row's own id must now match, and a
  collision warns and refetches.
* A checkpoint holding more records than the current `max_results` asks for is now
  served trimmed to that cap, with the fuller set left on disk for a later, wider
  request. The opposite direction, a checkpoint truncated below what is asked for,
  already refetched with a warning.

A rejected or damaged checkpoint now warns, where it used to be mentioned only under
`verbose`, and checkpoints are written to a sibling temporary file and renamed into
place, so an interrupted run can no longer leave a half-written file that breaks every
later resume.

## Other fixes

* `scopus_combine()` handed the merged set one input's own retrieval attributes.
  `rbind()` keeps those of its first argument, so the result inherited that harvest's
  `plan`, `cell_totals` and `total_results` even though it held the rows of several,
  contradicting the documented promise that per-retrieval attributes are not carried
  over. A search record read off the inherited total then called the union of two
  harvests complete against a figure belonging to one of them. Only what survives a
  merge is kept now: the earliest retrieval time, every contributing version, the
  paging mode where the inputs agree, and the merge counts themselves.
* The PRISMA 2020 identification block counted the records identified after
  de-duplication, so the two figures it gives could not both be right: the diagram
  subtracts the duplicates removed from the records identified to reach the records
  screened, and 138 identified less 11 removed is not the 138 rows the set holds. Where
  a merge recorded how many records went into it, that is now the identification count.
* A plan not yet run was described in the past tense, and its date line said the set did
  not carry a retrieval time, when the truth was that no retrieval had happened.
* `average_comparison_percentage` summed the numerator over years the denominator
  excluded, so a year with a missing reference count inflated it, giving a figure that
  contradicted the per-year shares printed beside it and that drives the topic ordering
  in `plot_scopus_comparison()`. It is now computed over the years where both counts are
  available.
* `scopus_abstract(include = "references")` aborted the whole batch when one document's
  response omitted its reference-count attribute. Because that arrived as a bare base
  error, untyped and so outside the `scopus_error` hierarchy, the per-identifier handler
  could not catch it, defeating the documented promise that one bad identifier does not
  lose a batch.
* The app's demo banner still called the demo data synthetic, which was true of the
  six invented rows that shipped before 0.3.0. Demo mode has replayed 138 real
  published articles since then, and only the topic comparison is simulated, so the
  banner now says which is which.
* A record count past 2^31 was coerced to `NA` at three points, each of which
  accepted any finite whole number and then reached for a 32-bit integer. `scopus_top(n = )`
  left `head.data.frame()` raising an untyped error about its own argument;
  `scopus_fetch(max_results = )` and `scopus_fetch_plan(max_results = )` turned every
  later `n >= max_results` into `NA`, so the retrieval died on a bare "missing value
  where TRUE/FALSE needed"; and the app's reproducible-code panel wrote
  `max_results = NA` into a script it promises is runnable. All three now carry the
  value as a double, so a cap wider than the result set simply never bites, which is
  what the Python twin has always done. A cap that fits stays an integer.
* The 5000-record cap warning fired even when `max_results` had already asked for fewer
  records than the ceiling, advising a remedy for a problem the caller did not have.
* `scopus_compare_topics()` carries its counts as doubles, matching `scopus_trend()` and
  `scopus_intersections()`, so billion-scale totals no longer coerce to `NA`.
* Every request carries a 60-second transfer timeout, tunable through
  `getOption("scopusflow.timeout")`, so a stalled connection cannot hang a session.
* Retrievals record when they were taken and by which package version, as the
  `retrieved_at` and `scopusflow_version` attributes, preserved through the `.rds` round
  trip. Both are attributes, so the documented schema and the CSV round trip are
  unchanged.
* Every text file the package writes now carries LF line endings on all platforms: the
  BibTeX and RIS exports, the DOI and record CSVs, and the app's script and comparison
  downloads previously arrived with CRLF endings when written on Windows.
* Every `print()` method now sends its whole output to standard output. The header line
  went through cli, which writes to the message stream, so `capture.output()` returned
  the table without its header, and a knitted document showed one printed object as two
  separate blocks.

## Continuous integration and metadata

* The dependency canary could finish green having exercised nothing, when no
  development head was actually installed. It now says so, in a warning and in the step
  summary, and it installs the ggplot2 development head alongside the Imports.
* The declared dependency floors are now installed and tested by a `min-deps` job, and
  the check matrix reaches back to roughly R 4.3.
* `rlang`'s floor is corrected to 1.1.0. The evidence for it is httr2 1.0.0's own
  DESCRIPTION, which requires `rlang (>= 1.1.0)`, so the old floor described a
  combination no user could have installed. It is CRAN-visible metadata.
* The live API check's documented request cost, the cache guard's description in the
  documentation and vignette, and the path to the Python twin's copy of the example
  records were all corrected.

# scopusflow 0.3.0

A data and documentation release. The bundled corpus becomes a real harvest, and
every example and article is rebuilt on it.

## The bundled example records

The dataset the package ships for offline work was replaced outright.

* `example_records` is now a worked example harvest of 138 real journal articles
  on graphene supercapacitors published between 2015 and 2024, carrying their
  real titles, DOIs, source titles, first authors and citation counts. It
  replaces the six invented records shipped previously.
* The records are not a 'Scopus' harvest and are not described as one. The
  Elsevier API terms do not permit redistributing retrieved records, so they come
  from OpenAlex, whose metadata is released under CC0, reshaped into the schema
  [`scopus_fetch()`] returns. The reasoning is recorded in the design notes.
* The harvest is complete, so the rows per year are the real publications per year
  for that query. Eleven records carry no DOI and two no source title, kept as they
  arrive because a real harvest has such gaps.
* `scopus_id` is empty throughout, these records not having come from 'Scopus',
  so de-duplication falls back to the DOI as it does for any record whose
  identifier is missing.

## Documentation

The material a reader meets was rebuilt on the new corpus, and one misleading
fixture was replaced.

* Every vignette and example runs on that corpus, paired with the key-gated live
  call a reader would write, and the figures quoted in the prose were
  recomputed against the new data.
* The demo mode of `run_app()` draws on the same corpus, so a first look at the
  app opens on real articles, where it used to show invented rows.
* The parser fixture in `inst/extdata` moves onto the reserved 10.5555 example
  prefix. It previously paired genuine, resolving DOIs with invented titles and
  authors, so a reader who checked one found a real paper mislabelled.
* Every vignette now turns console colour off and fixes the console width while
  it renders. pkgdown enables colour for its own build, and the escape sequences
  were reaching the reader as literal text in the middle of the tables.
* `vignette("comparing-topics")` shows one year of its illustrative comparison
  across every topic, in place of the whole table, and leaves out the `query`
  column, which in a real comparison holds the whole query string and so reads
  badly in a table.
* The README no longer opens with a link to the documentation site, which on the
  site's own home page pointed the reader at the page in front of them.

# scopusflow 0.2.1

A documentation release. The vignettes now demonstrate several features that
0.2.0 shipped but did not show.

* `vignette("designing-queries")` shows the `AND NOT` operator in
  [`scopus_query()`], excluding a dominant homonym from a search.
* `vignette("analysing-a-literature")` passes [`scopus_intersections()`] a
  concept that is already a complete field-tagged expression and so is used as
  given, letting a concept be a whole synonym set.
* `vignette("scopusflow")` introduces [`scopus_top()`] on a record set, and
  points to the analysis article for the plots and trends built on it.
* `vignette("plans-and-quota")` covers `verbose = TRUE` in
  [`scopus_fetch_plan()`], which reports a line as each cell is fetched or
  loaded from cache.
* `vignette("building-a-reference-set")` shows [`scopus_extract_dois()`] on a
  plain vector of DOIs, both with the default deduplication (which ignores case
  and resolver prefixes) and with `dedupe = FALSE`.
* `vignette("keywords-and-references")` tallies author keywords across a
  [`scopus_corpus()`] result, the per-keyword document count the article is
  named for.
* [`scopus_abstract()`]'s help page no longer describes the Python twin's
  reference fields.

# scopusflow 0.2.0

This release reaches further into the API, adds an analysis and export layer on
top of a retrieval, and introduces a local, code-free app.

## Deeper retrieval

A search now reaches further into the API, past the offset ceiling and beyond
the fields the Search endpoint returns.

* [`scopus_fetch()`] gains `cursor = TRUE`, cursor-based pagination that
  retrieves a whole large query without the 5000-record ceiling of offset
  paging. The warning on a query that exceeds the ceiling suggests this
  alongside partitioning with [`scopus_plan()`].
* [`scopus_fetch()`] and [`scopus_fetch_plan()`] add an `authkeywords` column
  when `view = "COMPLETE"` is requested, at no cost beyond that view's own
  smaller page size. The `view = "STANDARD"` output is unchanged, and
  [`read_scopus_records()`] keeps the column across a CSV round-trip.
* [`scopus_abstract()`] retrieves the abstract and fuller metadata for one or
  many records from the 'Scopus' Abstract Retrieval API, resilient to an
  identifier that cannot be found. Through `view` and `include =
  c("references", "keywords")` it also retrieves a document's own reference
  list, as a structured, per-citation data frame, and author keywords, with
  per-identifier caching keyed by the requested view and extras, an
  `n_requests`/`quota` attribute, and a clear, actionable error on an
  entitlement 403 that stops the batch, so the same failure is not repeated for
  every identifier. `include = "keywords"` without `view = "FULL"` is rejected
  up front, since the `REF` response carries no author keywords.
* [`scopus_corpus()`] combines a search result with abstract retrieval into a
  minimal `id`/`title`/`year`/`keywords`/`references` shape for downstream
  tools such as keyword co-occurrence or citation-network analysis, without
  replacing [`as_bibliometrix()`]. A new vignette, *Author keywords and
  references*, walks through all of this with real DOIs.
* [`scopus_fetch_plan()`] compares each checkpoint's recorded query with the
  plan cell before loading it, refetching and overwriting the checkpoint on a
  mismatch, so two different plans pointed at the same `cache_dir` cannot
  serve each other's records. A checkpoint that carries no query information
  (a zero-row cell, or one written by scopusflow 0.1.0) loads as before, and a
  cache directory is still best kept to a single plan.

## Analysis and plots

The package gains a layer for summarising a literature and drawing the result.

* [`scopus_trend()`] reports annual record counts for a query (the size of a
  literature over time), with [`plot_scopus_trend()`].
* [`scopus_top()`] tallies the most frequent sources or authors in a record
  set, with [`plot_scopus_top()`], which draws whole-number axis breaks (so a
  tally of small counts shows no fractional ticks) and derives the count
  axis's headroom from the widest end-of-bar label (so a wide count, say five
  figures on a top-authors bar, does not clip at the panel edge). An
  `autoplot()` method draws a record set's publications per year.
* [`scopus_intersections()`] counts a named set of concepts and any requested
  intersections of them, sizing where a study or a niche sits within the
  surrounding literature at one count request per row. Concept values that are
  already complete field-tagged expressions are used exactly as given, with no
  second wrap. [`plot_scopus_intersections()`] draws the result as a
  lollipop chart on a log-scale axis, with an `autoplot()` method and an
  optional highlight (for example the niche itself) whose legend label is
  derived from what is highlighted: 'Focal intersection' for intersections,
  'Focal concept' for concepts and 'Focal set' for a mixture. An explicit
  `highlight_label` still wins.
* [`plot_scopus_comparison()`] gains `legend_inside`. When set, and a legend is
  drawn, it is placed inside the panel in whichever corner has the most free
  space, on a small semi-transparent background, where the default places it
  above the panel. That default is unchanged.
* [`plot_scopus_comparison()`] now spreads the direct line labels vertically so
  topics that converge near the final year no longer overlap, and falls back to
  a legend when there are too many topics to label legibly. The labels are
  spread when the figure is drawn, against the rendered text height, so they
  stay legible at any figure size, including a short panel such as the app's
  result card. They carry no leader lines. The labels are colour-matched to
  their lines and spread in the same order as the line ends, so the link is
  clear without a leader that would otherwise cut across neighbouring labels.

## Export

A retrieval can now leave the package in the formats other tools read.

* [`as_bibtex()`] and [`as_ris()`] export a record set to the BibTeX and RIS
  interchange formats, so a search can be carried into Zotero, EndNote,
  Mendeley or a LaTeX bibliography.

## A code-free app

The whole workflow is now available without writing any R.

* [`run_app()`] launches a local, code-free Shiny app for building a search,
  retrieving records with a live progress terminal, and exporting them. A panel
  mirrors every choice as a runnable R script, so the app is an on-ramp to the
  package. It runs on your own machine, so the API key never leaves it. The app
  also has a *Compare topics* tab (with highlight, stability-band and
  counts-in-label controls, a per-term progress indicator, a quota estimate and
  a CSV export) and a *Demo mode*, on by default, that synthesises records and
  a comparison so the whole workflow can be explored with no key and no
  network. A new vignette, *Using the code-free app*, walks through every
  panel.
* The app holds steady under stress. It refuses to start a comparison while a
  harvest is running, turns any comparison failure into a notification, floors a
  fractional maximum-records entry, drops duplicate comparison terms, and tells
  you when there is nothing to cancel.

## Other improvements

One rough edge in the package's messaging was smoothed.

* The no-key error renders its guidance (the option name, the `api_key`
  argument and the key-request URL) through cli, where it used to leak raw
  markup.

# scopusflow 0.1.0

First release.

* Reproducible search plans with [`scopus_plan()`], and cheap sizing with
  [`scopus_count()`].
* Quota-aware, paginated retrieval through [`scopus_fetch()`], with the largest
  page each view allows requested by default to keep request counts low, and
  resumable, cached, partitioned retrieval through [`scopus_fetch_plan()`].
* A stable normalised record schema from [`scopus_records()`], with a
  `summary()` method that gives a quick overview.
* DOI extraction and change tracking with [`scopus_extract_dois()`] and
  [`scopus_diff_dois()`].
* Topic-trend comparison with [`scopus_compare_topics()`], and a plot from
  [`plot_scopus_comparison()`] or `autoplot()`.
* Interoperability and I/O through [`as_bibliometrix()`],
  [`write_scopus_records()`] and [`read_scopus_records()`].
* A reference to the common 'Scopus' field tags in [`scopus_field_tags()`], a
  safe query composer in [`scopus_query()`], and a bundled [`example_records`]
  dataset for offline exploration.
* Safe merging of record sets with [`scopus_combine()`] (and a `c()` method), plus
  `as_tibble()` and `as.data.frame()` coercion.
* A typed condition system (`scopus_error` and its subclasses) and quota-header
  parsing with [`scopus_quota()`].
* The comparison plot uses whole-number year breaks, a colour-blind-safe palette,
  direct line labels, an optional `highlight` argument and a shaded Wilson
  stability band (an illustrative range, switchable with `interval`).
* The bundled `example_records` spans several disciplines, and the examples and
  five workflow vignettes draw on a wide range of fields.
* The `authors` column retains every author a record lists. Very large result
  totals are handled without overflow, and DOI cleaning copes with `www.doi.org`
  hosts and `DOI:` labels.
