---
title: "Connect to a Databricks Workspace"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Connect to a Databricks Workspace}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  eval = FALSE
)
```

## Defining Credentials

The `{brickster}` package connects to a Databricks workspace in three ways:

1.  [OAuth user-to-machine (U2M) authentication](https://docs.databricks.com/en/dev-tools/auth/oauth-u2m.html#oauth-user-to-machine-u2m-authentication), either directly or through credentials managed by the Databricks CLI
2.  [OAuth machine-to-machine (M2M) authentication](https://docs.databricks.com/en/dev-tools/auth/oauth-m2m.html)
3.  [Personal Access Tokens (PAT)](https://docs.databricks.com/en/dev-tools/auth/pat.html)

It's recommended to use option (1) when using `{brickster}` interactively. If you need to run code via an automated process, use option (2) or (3).

`{brickster}` will automatically detect when a session has [Posit Workbench managed Databricks OAuth credentials](https://docs.posit.co/ide/server-pro/integration/databricks.html) enabled. For more information about this authentication flow see the section [Posit Workbench Managed Databricks OAuth Credentials](#posit-workbench-managed-databricks-oauth-credentials).

Personal Access Tokens can be generated in a few steps, for a step-by-step breakdown [refer to the documentation](https://docs.databricks.com/aws/en/dev-tools/auth).

Once you have a token you'll be able to store it alongside the workspace URL in an `.Renviron` file. The `.Renviron` is used for storing the variables, such as those which may be sensitive (e.g. credentials) and de-couple them from the code [additional reading](https://CRAN.R-project.org/package=startup/vignettes/startup-intro.html).

To get started add the following to your `.Renviron`:

-   `DATABRICKS_HOST`: The workspace URL

-   `DATABRICKS_TOKEN`: Personal access token (*not required if using OAuth U2M or M2M*)

-   `DATABRICKS_CLIENT_ID`: OAuth M2M client id (*only required for OAuth M2M*)

-   `DATABRICKS_CLIENT_SECRET`: OAuth M2M client secret (*only required for OAuth M2M*)

-   `ARM_CLIENT_ID`: Azure AD service principal application id (*only required for Azure service principal OAuth M2M*)

-   `ARM_CLIENT_SECRET`: Azure AD service principal client secret (*only required for Azure service principal OAuth M2M*)

-   `ARM_TENANT_ID`: Azure AD tenant id (*only required for Azure service principal OAuth M2M*)

-   `DATABRICKS_WSID`: The workspace ID ([docs](https://docs.databricks.com/workspace/workspace-details.html#workspace-instance-names-urls-and-ids))

`DATABRICKS_WSID` is only required for the RStudio IDE integration with the connection pane.

Example of entries in `.Renviron`:

```         
DATABRICKS_HOST=xxxxxxx.cloud.databricks.com
DATABRICKS_TOKEN=dapi123456789012345678a9bc01234defg5
DATABRICKS_WSID=123123123123123
```

For OAuth M2M:

```
DATABRICKS_HOST=xxxxxxx.cloud.databricks.com
DATABRICKS_CLIENT_ID=11111111-2222-3333-4444-555555555555
DATABRICKS_CLIENT_SECRET=abcdefg1234567890
DATABRICKS_WSID=123123123123123
```

For Azure service principal OAuth M2M:

```
DATABRICKS_HOST=adb-xxxxxx.xx.azuredatabricks.net
ARM_CLIENT_ID=11111111-2222-3333-4444-555555555555
ARM_CLIENT_SECRET=abcdefg1234567890
ARM_TENANT_ID=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee
```

With no `auth_type` in a selected `.databrickscfg` profile, `{brickster}` attempts Databricks OAuth M2M (`DATABRICKS_CLIENT_*`), then Azure service principal OAuth M2M (`ARM_*`), then OAuth U2M. When using `.databrickscfg`, set `auth_type = azure-client-secret` in the profile to force Azure service principal authentication.

**Note**: Recommend creating an `.Renviron` for each project. You can create `.Renviron` within your user home directory if required.

Restarting your R session will allow those variable to be picked up via the `{brickster}` package.

## Using Credentials with `{brickster}`

Authentication should now be possible without specifying the credentials in your R code. You can load `{brickster}` and list the clusters within the workspace using `db_cluster_list()`, to access the host/token use `db_host()`/`db_token()` respectively.

```{r setup}
library(brickster)

# using db_host() and db_token() to get credentials
clusters <- db_cluster_list(host = db_host(), token = db_token())
```

All `{brickster}` functions have their host/token parameters default to calling `db_host()`/`db_token()` therefore we can omit explicit calls to the functions.

```{r}
# all host/token parameters default to db_host()/db_token()
clusters <- db_cluster_list()
```

When using OAuth U2M or OAuth M2M authentication you don't define a token in `.Renviron` and therefore `db_token()` will return `NULL`.

## Managing Multiple Credentials

There are two methods that `{brickster}` supports to simplify switching of credentials within an R project/session:

1.  Adding multiple credentials to `.Renviron`, each additional set of credentials is differentiated via a suffix (e.g. `DATABRICKS_TOKEN_DEV`)
2.  Using a `.databrickscfg` file (primary method in [Databricks CLI](https://docs.databricks.com/dev-tools/cli/index.html#set-up-authentication))

The `use_databrickscfg` option selects which configuration source `{brickster}`
reads. It does not sign in to the Databricks CLI or select a named profile.
The following example switches the session from `.Renviron` to
`.databrickscfg`:

```{r}
# uses the CLI-selected default profile, or `DEFAULT` when none is selected
options(use_databrickscfg = TRUE)

# values are read from `.databrickscfg`
db_host()
db_token()
```

With its default value of `FALSE`, `{brickster}` does not discover profiles
created by `databricks auth login` and does not enable the `databricks-cli`
authentication provider. Set the option in `.Rprofile` when `.databrickscfg`
should be the normal configuration source. Posit Workbench managed OAuth
sessions enable `.databrickscfg` automatically.

### Switching Between Credentials

When `.databrickscfg` is enabled, the selected profile controls the host,
credentials, authentication type, and the profile passed to `databricks auth
token`. `{brickster}` resolves the profile in this order:

1. `DATABRICKS_CONFIG_PROFILE`
2. The `db_profile` option
3. The CLI default selected by `databricks auth switch`
4. The `DEFAULT` profile

`DATABRICKS_CONFIG_PROFILE` is the standard Databricks unified authentication
setting. `db_profile` is the `{brickster}` session-level equivalent.

Profiles enable you to switch contexts between:

-   Different workspaces (e.g. development or production)

-   Different permissions (e.g. admin or restricted user)

This behaviour works when using credentials specified in either `.Renviron` or `.databrickscfg`:

```{r}
# using .Renviron
db_host() # returns `DATABRICKS_HOST` (.Renviron)

# switch profile to 'prod'
options(db_profile = "prod")
db_host() # returns `DATABRICKS_HOST_PROD` (.Renviron)

# clear the session-specific profile selection
options(db_profile = NULL)
# use .databrickscfg
options(use_databrickscfg = TRUE)
db_host() # returns the CLI-selected default, otherwise `DEFAULT` (.databrickscfg)

options(db_profile = "prod")
db_host() # returns host from `prod` profile (.databrickscfg)
```

It is expected that profiles in `.Renviron` will adhere to the same naming convention as default but add an additional suffix.

Here is an example of an `.Renviron` file that has three profiles (default, dev, prod):

```         
# default
DATABRICKS_HOST=xxxxxxx.cloud.databricks.com
DATABRICKS_TOKEN=dapixxxxxxxxxxxxxxxxxxxxxxxxx
DATABRICKS_WSID=123123123123123
# dev
DATABRICKS_HOST_DEV=xxxxxxx-dev.cloud.databricks.com
DATABRICKS_TOKEN_DEV=dapixxxxxxxxxxxxxxxxxxxxxxxxx
DATABRICKS_WSID_DEV=123123123123124
# prod
DATABRICKS_HOST_PROD=xxxxxxx-prod.cloud.databricks.com
DATABRICKS_TOKEN_PROD=dapixxxxxxxxxxxxxxxxxxxxxxxxx
DATABRICKS_WSID_PROD=123123123123125
```

### Configuring `.databrickscfg`

For details on configuring please refer to [documentation from Databricks CLI](https://docs.databricks.com/dev-tools/cli/index.html#connection-profiles).

There is only one `{brickster}` specific feature and it is the inclusion of `wsid` alongside `host`/`token`.

Supported OAuth fields in a `.databrickscfg` profile include:

- Databricks service principal fields: `client_id`, `client_secret`
- Azure service principal fields: `azure_client_id`, `azure_client_secret`, `azure_tenant_id`
- Optional auth mode override: `auth_type` (`oauth-m2m`, `azure-client-secret`, `oauth-u2m`, `databricks-cli`)

#### Databricks CLI OAuth profiles

Profiles created by `databricks auth login` use `auth_type = databricks-cli`.
The login can be performed from any terminal; the credentials are not scoped to
that terminal session. `{brickster}` must run as the same operating-system user
and use the same `.databrickscfg` and CLI credential storage.

The `databricks-cli` provider is intentionally available only when the selected
`.databrickscfg` profile contains `auth_type = databricks-cli`. Environment
configuration cannot activate this provider independently.

Enable `.databrickscfg` and select the profile before making requests:

```{r}
options(
  use_databrickscfg = TRUE,
  db_profile = "e2-demo"
)

# The host and CLI token are resolved from the same profile.
db_sql_warehouse_list()
```

Instead of `db_profile`, set `DATABRICKS_CONFIG_PROFILE`, or select the CLI
default outside R:

``` sh
databricks auth switch --profile e2-demo
```

When a profile is selected, `{brickster}` calls `databricks auth token
--profile <profile>`. Without a selected profile, `{brickster}` follows the
Databricks SDK behavior and asks the CLI for the resolved host. If multiple
profiles contain the same host, the CLI refuses to guess; select a profile
rather than signing in again. Avoid supplying a different `host` argument while
a profile is selected, because the request host and CLI credential would no
longer describe the same configuration.

Token acquisition is deferred until an API request is performed. `{brickster}`
keeps only the short-lived access token in memory until it expires, then calls
the CLI again. The Databricks CLI remains responsible for durable credential
storage and refresh. The CLI must be available on `PATH`, or its executable can
be provided with `DATABRICKS_CLI_PATH`. `DATABRICKS_CONFIG_FILE` is inherited by
the CLI when using a non-default config location.

The `oauth-u2m` mode remains `{brickster}`'s direct browser-based OAuth flow;
it does not reuse credentials created by `databricks auth login`.

`wsid` is used by the connections pane integration in RStudio as the underlying API's require it.

### Posit Workbench Managed Databricks OAuth Credentials

Posit Workbench has a [managed Databricks OAuth credentials](https://docs.posit.co/ide/server-pro/integration/databricks.html) feature, which allows users to sign into a Databricks workspace from the home page of Workbench when launching a session and then access Databricks resources as their own identity. When in an RStudio Pro session running on Posit Workbench with managed Databricks OAuth credentials selected, `{brickster}` functions using `db_host()`/`db_token()` respectively should just work without needing to specify any credentials in your R code. See the code below as an example.

```{r}
library(brickster)
db_cluster_list()
```

`{brickster}` will automatically detect when a session has Workbench managed OAuth credentials and then use the `workbench` profile defined in a `.databrickscfg` file at the `DATABRICKS_CONFIG_FILE` specified location. Workbench generates this `.databrickscfg` file in a temporary directory and should not be modified directly.

To use an alternative `.databrickscfg` file, a different `profile`, an alternative env variable `DATABRICKS_HOST` or set an env variable `DATABRICKS_TOKEN`, launch an RStudio Pro session without the Databricks managed credentials box selected.
