# A Complete Incrementality Analysis

## Overview

Incrementality asks what happened because a treatment was applied, beyond what would have happened under control. `IncrementalityTEST` analyzes a collection of experiment-level treatment and control measurements.

The workflow has four steps:

1. calculate business metrics where necessary;
2. validate and pair treatment and control rows;
3. calculate one effect per experiment;
4. summarize those effects and quantify uncertainty.

## Prepare group-level data

Each experiment needs exactly one row for control and one for treatment.

    library(IncrementalityTEST)

    results <- data.frame(
      experiment = rep(paste0("test_", 1:6), each = 2),
      group = rep(c("control", "treatment"), 6),
      transactions = c(100, 112, 80, 87, 130, 141, 92, 96, 110, 121, 70, 79),
      users = c(1000, 1005, 800, 804, 1200, 1208, 900, 902, 1050, 1053, 700, 704),
      revenue = c(2500, 2860, 1920, 2140, 3300, 3690, 2200, 2350, 2750, 3100, 1680, 1950),
      buyers = c(90, 99, 72, 78, 115, 124, 83, 86, 101, 109, 63, 70)
    )

Calculate the four supported commerce metrics and attach them to the data.

    metrics <- calculate_metrics(
      results$transactions,
      results$users,
      results$revenue,
      results$buyers
    )
    results <- cbind(results, metrics)
    head(results)
    #>   experiment     group transactions users revenue buyers      RPU         BR
    #> 1     test_1   control          100  1000    2500     90 2.500000 0.09000000
    #> 2     test_1 treatment          112  1005    2860     99 2.845771 0.09850746
    #> 3     test_2   control           80   800    1920     72 2.400000 0.09000000
    #> 4     test_2 treatment           87   804    2140     78 2.661692 0.09701493
    #> 5     test_3   control          130  1200    3300    115 2.750000 0.09583333
    #> 6     test_3 treatment          141  1208    3690    124 3.054636 0.10264901
    #>        AOV      TPB
    #> 1 25.00000 1.111111
    #> 2 25.53571 1.131313
    #> 3 24.00000 1.111111
    #> 4 24.59770 1.115385
    #> 5 25.38462 1.130435
    #> 6 26.17021 1.137097

## Inspect experiment-level effects

`metric_differences()` uses treatment minus control by default. A positive number therefore means that the metric was higher under treatment.

    effects <- metric_differences(results, metric = "RPU")
    effects
    #>   experiment  control treatment difference
    #> 1     test_1 2.500000  2.845771  0.3457711
    #> 2     test_2 2.400000  2.661692  0.2616915
    #> 3     test_3 2.750000  3.054636  0.3046358
    #> 4     test_4 2.444444  2.605322  0.1608771
    #> 5     test_5 2.619048  2.943970  0.3249220
    #> 6     test_6 2.400000  2.769886  0.3698864

Missing groups, duplicate groups, and nonnumeric metrics produce explicit errors. Use `na_action = "omit"` only when dropping incomplete experiments is methodologically defensible.

## Estimate the overall effect

    analysis <- analyze_incrementality(
      results,
      metric = "RPU",
      conf_level = 0.95,
      bootstrap_times = 2000,
      seed = 2026
    )
    analysis
    #> Incrementality analysis
    #> Metric: RPU
    #> Direction: treatment-control
    #> Experiments: 6
    #>
    #>     method      mean standard_error conf_level     lower     upper
    #>  Student t 0.2946306     0.03069823       0.95 0.2157183 0.3735430
    #>  Bootstrap 0.2946306     0.02819784       0.95 0.2358360 0.3433737

The object contains all intermediate and final results:

    analysis$differences
    #>   experiment  control treatment difference
    #> 1     test_1 2.500000  2.845771  0.3457711
    #> 2     test_2 2.400000  2.661692  0.2616915
    #> 3     test_3 2.750000  3.054636  0.3046358
    #> 4     test_4 2.444444  2.605322  0.1608771
    #> 5     test_5 2.619048  2.943970  0.3249220
    #> 6     test_6 2.400000  2.769886  0.3698864
    analysis$t_interval
    #>   n      mean standard_error conf_level     lower    upper
    #> 1 6 0.2946306     0.03069823       0.95 0.2157183 0.373543
    analysis$bootstrap_interval
    #>        mean standard_error conf_level    lower     upper
    #> 1 0.2946306     0.02819784       0.95 0.235836 0.3433737

The t interval assumes that experiment-level effects are independent and that their sampling distribution is reasonably approximated by a normal distribution. The percentile bootstrap makes fewer distributional assumptions, but a small or unrepresentative set of experiments still limits inference.

## Report the result

A useful report states:

- the metric and effect direction;
- the number and selection of experiments;
- the mean experiment-level incremental effect;
- the confidence level and interval method;
- the treatment/control unit and any filtering decisions.

The package estimates an unweighted mean across experiments. If experiments have materially different precision or target populations, a hierarchical model or justified weighting strategy may be more appropriate.

## Working with legacy data

For existing datasets containing `iabtest_id` and numeric `abt_group` values, the compatibility function remains available:

    legacy <- transform(
      results,
      iabtest_id = experiment,
      abt_group = ifelse(group == "control", 0, 1)
    )
    test_metric(legacy, "RPU")
    #>   iabtest_id    inc_RPU
    #> 1     test_1 -0.3457711
    #> 2     test_2 -0.2616915
    #> 3     test_3 -0.3046358
    #> 4     test_4 -0.1608771
    #> 5     test_5 -0.3249220
    #> 6     test_6 -0.3698864

For historical compatibility, `test_metric()` uses control minus treatment. New analyses should use `metric_differences()` so the direction is explicit.