---
title: "Comparing topics over time"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Comparing topics over time}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(collapse = FALSE, comment = "")
# Console colour carries no meaning on a rendered page. pkgdown turns it on for
# its own build, and the escape sequences then reach the reader as literal text,
# so colour is switched off here for a plain vignette render and a site build
# alike. The fixed width keeps tibbles inside the documentation column.
options(cli.num_colors = 1, cli.hyperlink = FALSE, crayon.enabled = FALSE,
        width = 80)
# Print data frames and tibbles as formatted tables.
local({
  kp <- function(x, ...) {
    if (any(vapply(x, is.list, logical(1)))) return(knitr::normal_print(x))
    knitr::knit_print(knitr::kable(x))
  }
  for (cls in c("data.frame", "tbl_df", "tbl")) {
    registerS3method("knit_print", cls, kp, envir = asNamespace("knitr"))
  }
})
# The plotting chunks below need ggplot2, which is only a suggested package.
# Skip them when it is not installed.
has_ggplot2 <- requireNamespace("ggplot2", quietly = TRUE)
# Draw the figures on a transparent background so they sit on whatever colour
# the page behind them happens to be. An opaque white matte reads as a white
# slab on the site's dark theme, and worse once pkgdown's dark-mode filter
# inverts it into a black one. Both halves below are needed: the device option
# gives the file an alpha channel, and the theme override clears the white
# rectangle that ggplot2's complete themes paint over it regardless. The
# override belongs here, in the vignette, because a figure saved for a paper
# usually does want a background of its own, so the package's plotting
# functions leave it alone.
knitr::opts_chunk$set(dev.args = list(bg = "transparent"))
if (has_ggplot2) {
  sf_on_page <- ggplot2::theme(
    plot.background  = ggplot2::element_rect(fill = "transparent", colour = NA),
    panel.background = ggplot2::element_rect(fill = "transparent", colour = NA)
  )
  knitr::opts_chunk$set(render = function(x, ...) {
    if (inherits(x, "ggplot")) x <- x + sf_on_page
    knitr::knit_print(x, ...)
  })
}
```

```{r setup}
library(scopusflow)
```

A common bibliometric question is not how large a literature is, but how its
internal emphasis shifts over time. Within deep-learning research, say, is the
share of work that also concerns medical imaging growing faster than the share
about computer vision? `scopus_compare_topics()` answers exactly this, and
`plot_scopus_comparison()` shows the answer. The comparison itself contacts the
API, so it is shown but not run. The plotting is reproduced offline from an
object of the same shape.

## What the comparison measures

For each year and each comparison term, the function counts the records matching
the reference topic *and* that term, and expresses it as a percentage of the
records matching the reference *alone*. A value of 30% for 'computer vision' in
2020 means that 30% of the deep-learning records that year also mention computer
vision. The reference is the denominator, so it sits at 100% by construction and
is not drawn.

```{r eval = FALSE}
cmp <- scopus_compare_topics(
  reference_query  = "deep learning",
  comparison_terms = c("computer vision", "natural language processing",
                       "medical imaging", "drug discovery"),
  years            = 2013:2021,
  field            = "TITLE-ABS-KEY"
)
```

## The shape of the result

The result is a tidy table with one row per topic and year. A comparison counts
whole literatures, so unlike a trend it cannot be derived from the corpus the
package bundles for its other examples. The table below is rebuilt in the same
shape, with illustrative counts, so the rest of the article runs without a key. The reference set grows over the period, which the
uncertainty band will reflect.

```{r}
years <- 2013:2021
ref_n <- round(seq(400, 1600, length.out = length(years)))
mk <- function(from, to) round(seq(from, to, length.out = length(years)))
counts <- list(
  "computer vision" = mk(140, 720),
  "natural language processing" = mk(90, 540),
  "medical imaging" = mk(15, 260),
  "drug discovery" = mk(8, 170)
)
cmp <- tibble::tibble(
  query = "q",
  query_type = c(rep("reference", length(years)),
                 rep("comparison", length(counts) * length(years))),
  abridged_query = c(rep("deep learning", length(years)),
                     rep(names(counts), each = length(years))),
  year = rep(years, length(counts) + 1),
  n = c(ref_n, unlist(counts, use.names = FALSE)),
  reference_n = rep(ref_n, length(counts) + 1),
  comparison_percentage = 100 * c(ref_n, unlist(counts, use.names = FALSE)) /
    rep(ref_n, length(counts) + 1),
  average_comparison_percentage = c(rep(100, length(years)),
                                    rep(c(40, 33, 15, 9), each = length(years)))
)
class(cmp) <- c("scopus_comparison", class(cmp))

# The whole table is too long to read here, so show its first year across every
# topic. The `query` column is left out because a real comparison carries the
# whole query string sent to the API in it, which is too long for a table. The
# illustrative table built above holds a placeholder there instead.
cmp[cmp$year == min(cmp$year), setdiff(names(cmp), "query")]
```

Those are the five rows for the first year, one for the reference and one for
each comparison term. The whole table has `r nrow(cmp)` rows on the same
pattern. The `query` column left out above is still in the object. A real
comparison carries the whole query behind each count there, where this
illustrative table carries a placeholder.

The `comparison_percentage` column is the per-year share, and
`average_comparison_percentage` is the same ratio computed over the whole period,
which is what orders the topics. A year in which the reference has no records has
no defined share, so it is recorded as `NA`. A zero there would be read as a real
observation.

## A first plot

Drawing the comparison takes one call on the result.

```{r eval = has_ggplot2, fig.alt = "Four application areas' share of the deep-learning literature from 2013 to 2021, with shaded uncertainty bands and an in-panel legend", fig.width = 8, fig.height = 4.6}
plot_scopus_comparison(cmp, legend_inside = TRUE)
```

Here `legend_inside = TRUE` places the topic key inside the panel, in whichever
corner has the most free space. Left at its default the chart labels each line
at its end, and uses whole-number year breaks and a colour-blind-safe palette,
so with only a few topics the reader never has to match colours to a legend at
all. Each label carries the topic's total record count. The shaded band around
each line is a Wilson stability range. It is wide in the early years, when the
reference set is small and the share would move easily, and narrows as the
literature grows. 'Scopus' returns exact counts, so nothing here is a sample
from which an interval could be estimated. The band is illustrative, a point the
`plot_scopus_comparison()` help page sets out.

## When lines converge at the right end

Direct labels are legible only if they do not overlap, and topics sometimes end
the period at nearly the same share. `plot_scopus_comparison()` spreads
converging labels apart automatically, at the point the figure is actually
drawn, so they stay readable at any figure size and never stack into an
unreadable pile. Here six sub-areas of materials-science research all end
2013–2021 within three points of one another.

```{r}
years <- 2013:2021
ends <- c(18, 18.6, 19.2, 19.8, 20.4, 21)
names(ends) <- c(
  "graphene", "perovskites", "MXenes", "COFs", "MOFs", "aerogels"
)
ref_n <- round(seq(500, 2000, length.out = length(years)))
converge <- function(end) round(end *
  (0.5 + 0.5 * (0:(length(years) - 1)) / (length(years) - 1)) * ref_n / 100)
counts <- lapply(ends, converge)

cmp_converging <- tibble::tibble(
  query = "q",
  query_type = c(rep("reference", length(years)),
                 rep("comparison", length(counts) * length(years))),
  abridged_query = c(rep("energy materials", length(years)),
                     rep(names(counts), each = length(years))),
  year = rep(years, length(counts) + 1),
  n = c(ref_n, unlist(counts, use.names = FALSE)),
  reference_n = rep(ref_n, length(counts) + 1),
  comparison_percentage = 100 * c(ref_n, unlist(counts, use.names = FALSE)) /
    rep(ref_n, length(counts) + 1),
  average_comparison_percentage = c(rep(100, length(years)),
                                    rep(ends, each = length(years)))
)
class(cmp_converging) <- c("scopus_comparison", class(cmp_converging))
```

```{r eval = has_ggplot2, fig.alt = "Six materials-science sub-areas converging to similar shares by 2021, with end labels automatically spread apart so that none overlaps", fig.width = 8, fig.height = 4.6}
plot_scopus_comparison(cmp_converging)
```

Without this, six labels ending within three points of each other would print
on top of one another. Here every one is still readable, each colour-matched
to its own line and spread in the same order as the line ends.

## Drawing the eye to one topic

When one topic is the focus of a figure, `highlight` draws it in an accent colour
and greys the rest, which keeps the context visible without letting it compete.

```{r eval = has_ggplot2, fig.alt = "The same chart with the medical-imaging topic highlighted against the others in grey", fig.width = 8, fig.height = 4.6}
plot_scopus_comparison(cmp, highlight = "medical imaging")
```

## Adjusting the labels

The count suffix on each label can be turned off, and the uncertainty band can be
removed, when a cleaner look is wanted.

```{r eval = has_ggplot2, fig.alt = "The comparison chart without record counts or bands", fig.width = 8, fig.height = 4.6}
plot_scopus_comparison(cmp, pub_count_in_legend = FALSE, interval = FALSE)
```

The return value is an ordinary [ggplot2](https://ggplot2.tidyverse.org) object,
so any further adjustment, a different theme or a saved file, is one `+` or one
`ggplot2::ggsave()` away.

## Reading the result as a table

Sometimes the numbers matter more than the picture. Because the output is a
tibble, the usual tools apply. Here are the topics ranked by their average
share.

```{r}
comp <- cmp[cmp$query_type == "comparison", ]
unique(comp[, c("abridged_query", "average_comparison_percentage")])
```
