---
title: "Summarise clinical tables records"
output: 
  html_document:
    pandoc_args: [
      "--number-offset=1,0"
      ]
    number_sections: yes
    toc: yes
vignette: >
  %\VignetteIndexEntry{summarise_clinical_tables_records}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

# Introduction

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

In this vignette, we will explore the *OmopSketch* functions designed to provide an overview of the clinical tables within a CDM object (e.g. *visit_occurrence*, *condition_occurrence*, *drug_exposure*, *procedure_occurrence*, *device_exposure*, *measurement*, *observation*, and *death*). Specifically, there are two key functions that facilitate this:

-   `summariseClinicalRecords()`: creates a summary statistics with key basic information about the clinical table (e.g., number of records, records per person, etc.), some quality checks (e.g, missingness, correct filling of date columns, etc.) and a summary of the concepts used in the table (domains, source vocabularies, etc.)

-   `tableClinicalRecords()`: helps visualising the results in a formatted table.

## Create a mock cdm

Let's see an example of its functionalities. To start with, we will load essential packages and create a mock cdm using the R package [omock](https://ohdsi.github.io/omock/)

```{r, warning=FALSE}
library(dplyr)
library(OmopSketch)
library(omock)

# Connect to mock database
cdm <- mockCdmFromDataset(datasetName = "GiBleed", source = "duckdb")

cdm
```

# Summarise clinical tables

Let's now use `summariseClinicalTables()`from the OmopSketch package to help us have an overview of one of the clinical tables of the cdm (i.e., **condition_occurrence**).

```{r, warning=FALSE}
summarisedResult <- summariseClinicalRecords(
  cdm = cdm, 
  omopTableName = "condition_occurrence"
)

summarisedResult
```

Notice that the output is in the [summarised result](https://darwin-eu.github.io/omopgenerics/articles/summarised_result.html) format.
## Records per person
We can use the arguments to specify which statistics we want to perform. For example, use the argument `recordsPerPerson` to indicate which estimates you are interested regarding the number of records per person.

```{r, warning=FALSE}
summarisedResult <- summariseClinicalRecords(
  cdm = cdm,
  omopTableName = "condition_occurrence",
  recordsPerPerson = c("mean", "sd", "q05", "q95")
)

summarisedResult |>
  filter(variable_name == "records_per_person") |>
  select(variable_name, estimate_name, estimate_value)
```

## Quality

When the argument `quality = TRUE` is set, the results will include a quality assessment of the data.  
This assessment provides information such as:

- The proportion of records that fall outside the subjects’ observation periods.  
- Issues with date columns (e.g., start dates occurring after end dates, or dates preceding a subject’s birth date).  
- The presence of `person_id` values that do not exist in the `person` table.

```{r, warning=FALSE}
summarisedResult <- summariseClinicalRecords(
  cdm = cdm,
  omopTableName = "condition_occurrence",
  recordsPerPerson = NULL, 
  conceptSummary = FALSE,
  missing = FALSE,
  quality = TRUE
)

summarisedResult |>
  select(variable_name, estimate_name, estimate_value) 
```
## Concept Summary

When the argument `conceptSummary = TRUE` is set, the results will also include information about the concepts contained in the table, such as:

- The domain to which each concept belongs.  
- Whether each concept is a standard concept.  
- The type and source vocabulary associated with each concept.

```{r, warning=FALSE}
summarisedResult <- summariseClinicalRecords(
  cdm = cdm,
  omopTableName = "drug_exposure",
  recordsPerPerson = NULL, 
  conceptSummary = TRUE,
  missing = FALSE,
  quality = FALSE
)

summarisedResult |>
  select(variable_name, variable_level, estimate_name, estimate_value) 
```

## Missingness

When the argument `missing = TRUE` is set, the results will include a summary of missing data in the table, including the number of `0`s in the concept columns.  
This output is analogous to the results produced by the OmopSketch function `summariseMissingData()`.

```{r, warning=FALSE}
summarisedResult <- summariseClinicalRecords(
  cdm = cdm,
  omopTableName = "condition_occurrence",
  recordsPerPerson = NULL, 
  conceptSummary = FALSE,
  missing = TRUE,
  quality = FALSE
)

summarisedResult |>
  select(variable_name, variable_level, estimate_name, estimate_value) 
```

## Strata

It is also possible to stratify the results by sex and age groups:

```{r, warning=FALSE}
summarisedResult <- summariseClinicalRecords(
  cdm = cdm,
  omopTableName = "condition_occurrence",
  recordsPerPerson = c("mean", "sd", "q05", "q95"),
  quality = TRUE,
  conceptSummary = TRUE,
  sex = TRUE,
  ageGroup = list("<35" = c(0, 34), ">=35" = c(35, Inf))
)

summarisedResult |>
  select(variable_name, strata_level, estimate_name, estimate_value) 
```

Notice that, by default, the "overall" group will also be included, as well as crossed strata (that means, `sex == "Female"` and `ageGroup == "\>35"`).

Also, see that the analysis can be conducted for multiple OMOP tables at the same time:

```{r, warning=FALSE}
summarisedResult <- summariseClinicalRecords(
  cdm = cdm,
  omopTableName = c("visit_occurrence", "drug_exposure"),
  recordsPerPerson = c("mean", "sd"),
  quality = FALSE,
  conceptSummary = FALSE,
  missingData = FALSE
)

summarisedResult |>
  select(group_level, variable_name, estimate_name, estimate_value)
```

## Date Range

We can also filter the clinical table to a specific time window by setting the `dateRange` argument.

```{r}

summarisedResult <- summariseClinicalRecords(
  cdm = cdm, 
  omopTableName ="drug_exposure",
  dateRange = as.Date(c("1990-01-01", "2010-01-01"))
) 

summarisedResult |>
  settings() |>
  glimpse()
```


# Tidy the summarised object

`tableClinicalRecords()` will help you to tidy the previous results and create a formatted table of type [gt](https://gt.rstudio.com/), [reactable](https://glin.github.io/reactable/) or [datatable](https://rstudio.github.io/DT/). By default it creates a [gt](https://gt.rstudio.com/) table.

```{r, warning=FALSE}
summarisedResult <- summariseClinicalRecords(cdm,
  omopTableName = "condition_occurrence",
  recordsPerPerson = c("mean", "sd", "q05", "q95"),
  quality = TRUE, 
  conceptSummary = TRUE,
  sex = TRUE
)

tableClinicalRecords(result = summarisedResult, type = "gt")
```

# Disconnect from CDM

Finally, disconnect from the mock CDM.

```{r}
cdmDisconnect(cdm = cdm)
```
