Creating

data_frame() is a nice way to create data frames. It encapsulates best practices for data frames:

Coercion

To complement data_frame(), dplyr provides as_data_frame() for coercing lists into data frames. It does two things:

This is much simpler than as.data.frame(). It’s hard to explain precisely what as.data.frame() does, but it’s similar to do.call(cbind, lapply(x, data.frame)) - i.e. it coerces each component to a data frame and then cbinds() them all together. Consequently as_data_frame() is much faster than as.data.frame():

l2 <- replicate(26, sample(100), simplify = FALSE)
names(l2) <- letters
microbenchmark::microbenchmark(
  as_data_frame(l2),
  as.data.frame(l2)
)
#> Unit: microseconds
#>               expr      min        lq   median        uq      max neval
#>  as_data_frame(l2)   99.288  108.7505  115.799  127.1665  326.978   100
#>  as.data.frame(l2) 1402.052 1459.5375 1495.812 1546.2840 3197.099   100

The speed of as.data.frame() is not usually a bottleneck in interatively use, but can be a problem when combining thousands of messy inputs into one tidy data frame.

Memory

One of the reasons that dplyr is fast is that it is very careful about when it makes copies of columns. This section describes how this works, and gives you some useful tools for understanding the memory usage of data frames in R.

The first tool we’ll use is dplyr::location(). It tells us three things about a data frame:

location(iris)
#> <0x7f8a5e8eea88>
#> Variables:
#>  * Sepal.Length: <0x7f8a5e998200>
#>  * Sepal.Width:  <0x7f8a5b450200>
#>  * Petal.Length: <0x7f8a5b44f400>
#>  * Petal.Width:  <0x7f8a5b5ab200>
#>  * Species:      <0x7f8a5a598de0>
#> Attributes:
#>  * names:        <0x7f8a5e8eeaf0>
#>  * row.names:    <0x7f8a5a541d50>
#>  * class:        <0x7f8a5e8166c8>

It’s useful to know the memory address, because if the address changes, then you know R has made a copy. Copies are bad because it takes time to copy a vector. This isn’t usually a bottleneck if you have a few thousand values, but if you have millions or tens of millions it starts to take up a significant amount of time. Unnecessary copies are also bad because they take up memory.

R tries to avoid making copies where possible. For example, if you just assign iris to another variable, it continues to the point same location:

iris2 <- iris
location(iris2)
#> <0x7f8a5e8eea88>
#> Variables:
#>  * Sepal.Length: <0x7f8a5e998200>
#>  * Sepal.Width:  <0x7f8a5b450200>
#>  * Petal.Length: <0x7f8a5b44f400>
#>  * Petal.Width:  <0x7f8a5b5ab200>
#>  * Species:      <0x7f8a5a598de0>
#> Attributes:
#>  * names:        <0x7f8a5e8eeaf0>
#>  * row.names:    <0x7f8a5d070ad0>
#>  * class:        <0x7f8a5e8166c8>

Rather than carefully comparing long memory locations, we can instead use the dplyr::changes() function to highlights changes between two versions of a data frame. This shows us that iris and iris2 are identical: both names point to the same location in memory.

changes(iris2, iris)
#> <identical>

What do you think happens if you modify a single column of iris2? In R 3.1.0 and above, R knows enough to only modify one column and leave the others pointing to the existing location:

iris2$Sepal.Length <- iris2$Sepal.Length * 2
changes(iris, iris2)
#> Changed variables:
#>              old            new           
#> Sepal.Length 0x7f8a5e998200 0x7f8a5b29a000
#> 
#> Changed attributes:
#>              old            new           
#> row.names    0x7f8a5a595d70 0x7f8a5a58f050

(This was not the case prior to R 3.1.0: R created a deep copy of the entire data frame.)

dplyr is similarly smart:

iris3 <- mutate(iris, Sepal.Length = Sepal.Length * 2)
changes(iris3, iris)
#> Changed variables:
#>              old            new           
#> Sepal.Length 0x7f8a5b452800 0x7f8a5e998200
#> 
#> Changed attributes:
#>              old            new           
#> class        0x7f8a5f8567f8 0x7f8a5e8166c8
#> names        0x7f8a5b34be88 0x7f8a5e8eeaf0
#> row.names    0x7f8a5d08e940 0x7f8a5d073bc0

It’s smart enough to create only one new column: all the other columns continue to point at their old locations. You might notice that the attributes have still been copied. This has little impact on performance because the attributes are usually short vectors and copying makes the internal dplyr code considerably simpler.

dplyr never makes copies unless it has to:

This means that dplyr lets you work with data frames with very little memory overhead.

data.table takes this idea one step further than dplyr, and provides functions that modify a data table in place. This avoids the need to copy the pointers to existing columns and attributes, and provides speed up when you have many columns. dplyr doesn’t do this with data frames (although it could) because I think it’s safer to keep data immutable: all dplyr data frame methods return a new data frame, even while they share as much data as possible.