> For the complete documentation index, see [llms.txt](https://documentation.immuta.com/saas/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://documentation.immuta.com/saas/configuration/integrations/databricks/databricks-unity-catalog/how-to-guides/connect-unity-catalog.md).

# Register a Databricks Unity Catalog Connection

{% hint style="info" %}
**Preview documentation for upcoming feature**

This page currently reflects the UI that users will see in future updates.
{% endhint %}

## **Requirements**

* **Immuta user** with the `APPLICATION_ADMIN` Immuta permission
* [**Databricks service principal**](#user-content-fn-1)[^1] with the following privileges. For instructions on setting up this user, see the [Creating the Databricks service principal section](#creating-the-databricks-service-principal):
  * `USE CATALOG` and `MANAGE` on all catalogs containing securables you want registered as Immuta data sources.
  * `USE SCHEMA` on all schemas containing securables you want registered as Immuta data sources.
  * [`MODIFY`](#user-content-fn-2)[^2] and `SELECT` on all securables you want registered as Immuta data sources.
  * Workspace admin on the workspace configured as the host for the integration.
  * Additional privileges are required for query audit:
    * `USE CATALOG` on the `system` catalog
    * `USE SCHEMA` on the `system.access` and `system.query` schemas
    * `SELECT` on the following system tables:
      * `system.access.table_lineage`
      * `system.access.column_lineage`
      * `system.access.audit`
      * `system.query.history`
* **Databricks user to run the script to register the connection** with the following privileges:
  * [Metastore admin and account admin](#user-content-fn-3)[^3]
  * `CREATE CATALOG` privilege on the Unity Catalog metastore to create an Immuta-owned catalog and tables

See the [Databricks documentation](https://docs.databricks.com/en/data-governance/unity-catalog/manage-privileges/privileges.html) for more details about Unity Catalog privileges and securable objects.

#### Prerequisites

* Unity Catalog [metastore created](https://docs.databricks.com/data-governance/unity-catalog/create-metastore.html) and attached to a Databricks workspace.
* Unity Catalog enabled on your Databricks cluster or SQL warehouse. All SQL warehouses have Unity Catalog enabled if your workspace is attached to a Unity Catalog metastore. Immuta recommends linking a SQL warehouse to your Immuta tenant rather than a cluster for both performance and availability reasons.

## Register a connection

{% hint style="warning" %}
**Create a separate Immuta catalog for each Immuta tenant**

If multiple Immuta tenants are connected to your Databricks environment, create a separate **Immuta catalog** for each of those tenants. Having multiple Immuta tenants use the same Immuta catalog causes failures in policy enforcement.
{% endhint %}

1. Click <i class="fa-database">:database:</i> **Data** and select the **Connections** tab in the navigation menu.
2. Click the **+ Add Connection** button.
3. Select the **Databricks** data platform tile.
4. Enter the connection information:
   * **Display Name**: The display name represents the unique name of your connection and will be used as prefix in the name for all data objects associated with this connection. It will also appear as the display name in the UI and will be used in all API calls made to update or delete the connection. Avoid the use of periods (`.`) or [restricted words](#user-content-fn-4)[^4] in your connection name.
   * **Host**: The hostname of your Databricks workspace.
   * **Port**: Your Databricks port.
   * **HTTP Path**: The HTTP path of your Databricks cluster or SQL warehouse.
   * **Immuta Catalog**: The name of the catalog Immuta will create to store internal entitlements and other user data specific to Immuta. This catalog will only be readable for the Immuta service principal and should not be granted to other users. The catalog name may only contain letters, numbers, and underscores and cannot start with a number.
   * Opt to configure an HTTP proxy for this connection by selecting **Enable** and entering the required details:
     * **Proxy Host**: Enter the full proxy URL.
     * **Proxy Port**: Enter the proxy port.
     * **Proxy Username**: Enter the username for the proxy connection.
     * **Proxy Password**: Enter the password for the proxy connection.
5. Select your authentication method from the dropdown menu and enter the authentication information for the [Immuta service principal that you created](#creating-the-databricks-service-principal).

<details>

<summary>Personal Access Token</summary>

Enter the **Token**. This is the access token for the Immuta service principal, which can be [an on-behalf token created in Databricks](https://docs.databricks.com/api/workspace/tokenmanagement/createobotoken). This service principal must have the metastore [privileges listed](#setting-up-the-required-databricks-service-principal) for the metastore associated with the Databricks workspace. If this token is configured to expire, [update this field regularly](/saas/configuration/integrations/databricks/databricks-unity-catalog/how-to-guides/manage-databricks-unity-catalog-connection-settings.md) for the connection to continue to function. This authentication information will be included in the script populated to complete the connection setup.

</details>

<details>

<summary>OAuth M2M (Client Secret)</summary>

**AWS Databricks**:

1. Follow [Databricks documentation to create a client secret](https://docs.databricks.com/en/dev-tools/auth/oauth-m2m.html) for the Immuta service principal and assign this service principal the [privileges listed](#setting-up-the-required-databricks-service-principal) for the metastore associated with the Databricks workspace.
2. Fill out the **Client ID**. This is a combination of letters, numbers, or symbols, used as a public identifier and is the [client ID displayed in Databricks when creating the client secret for the service principal](https://docs.databricks.com/en/dev-tools/auth/oauth-m2m.html#step-3-create-an-oauth-secret-for-a-service-principal).
3. Fill out the **Authority URL** with the full URL of the identity provider. This is where the generated token is sent. The default value is `https://<your workspace name>.cloud.databricks.com/oidc/v1/token`.
4. Enter the **Scope** (string). The scope limits the operations and roles allowed in Databricks by the access token. See the [OAuth 2.0 documentation](https://oauth.net/2/scope/) for details about scopes.
5. Enter the **Client Secret**. Immuta uses this secret to authenticate with the authorization server when it requests a token.

**Azure Databricks:**

1. Follow [Databricks documentation](https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/oauth-m2m) to create a service principal within Azure and then populate to your Databricks account and workspace.
2. Assign this service principal the [privileges listed](#setting-up-the-required-databricks-service-principal) for the metastore associated with the Databricks workspace.
3. Within Databricks, [create an OAuth client secret for the service principal](https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/oauth-m2m#step-5-create-an-azure-databricks-oauth-secret-for-the-service-principal). This completes your Databricks-based service principal setup.
4. Fill out the **Client ID**. This is a combination of letters, numbers, or symbols, used as a public identifier and is the [client ID displayed in Databricks when creating the client secret for the service principal](https://docs.databricks.com/en/dev-tools/auth/oauth-m2m.html#step-3-create-an-oauth-secret-for-a-service-principal) (note that Azure Databricks uses the Azure SP Client ID; it will be identical).
5. Within Immuta, fill out the **Authority URL** with the full URL of the identity provider. This is where the generated token is sent. The default value is `https://<your workspace name>.azuredatabricks.net/oidc/v1/token`.
6. Enter the **Scope** (string). The scope limits the operations and roles allowed in Databricks by the access token. See the [OAuth 2.0 documentation](https://oauth.net/2/scope/) for details about scopes.
7. Enter the **Client Secret** you created above. Immuta uses this secret to authenticate with the authorization server when it requests a token.

</details>

<details>

<summary>OAuth M2M (Certificate)</summary>

1. Fill out the **Client ID**, which is the subject of the generated token. It is also known as `sub` (subject).
2. Fill out the **Authority URL**, which is the authority URL of your identity provider.
3. Fill out the **Scope**, which is the scope of operations and roles allowed by the secret.
4. Enter the **Certificate Thumbprint**. This identifies the corresponding key to the token and is often abbreviated as `x5t` or is called `kid` (key identifier).
5. Opt to fill out the **Resource** field with a URI of the resource where the requested token will be used.
6. Upload the **Private Key**, which is the client certificate that is used to sign the authorization request.

</details>

6. Click **Next**.
7. Configure the **Policy** setting:
   1. Opt to [exclude specific users from policy](/saas/configuration/integrations/snowflake/reference-guides/snowflake-overview.md#excepted-roles-users) by adding them to the **Exclude Group Pattern**. This field cannot be changed after the connection is created.
8. Click **Next**.
9. Configure the Audit settings:
   1. Opt to **Enable Audit**.
   2. If audit is enabled, opt to enter **Workspace IDs** to filter the audit ingest to just specific workspaces.
10. Click **Next**.
11. Opt to **Enable Object Sync**, which will keep your connection objects up to date with your Databricks Unity Catalog environment.
    1. Select the **Start Time** for when the first object sync job will run each day.
    2. Select the **Run Frequency** to determine how frequently object sync will run each day.
    3. Opt to **Run Object Sync on Create** to create all the data objects for your connection. If you do not select this option, you must run object sync manually or wait for the scheduled object sync for your connection's data objects.
12. Click **Save connection**.
13. Copy the provided script and run it in Databricks Unity Catalog as a user with the privileges [listed in the requirements section](#requirements).
14. Click **Validate Connection**.
15. If the connection is successful, click **Finish**. If there are any errors, check the connection details and credentials to ensure they are correct and try again.

## Opt to enable Databricks Unity Catalog tag ingestion

{% hint style="info" %}
**Private preview**: This feature is only available to select accounts. Contact your Immuta representative to enable this feature.
{% endhint %}

**Requirements**:

* A configured Databricks Unity Catalog connection
* Fewer than 10,000 Databricks Unity Catalog data sources registered in Immuta
* The following permissions must be added to your Immuta service principal:
  * `USE CATALOG` on the `system` catalog
  * `USE SCHEMA` on the `system.access` schema
  * `SELECT` on the following system table:
    * `system.access.audit`
  * If ingesting and applying tags to **foreign** **catalogs (created via Lakehouse Federation)**, the following additional permissions are required:
    * `USE SCHEMA` on the `system.information_schema` schema
    * `SELECT` on the following system tables:
      * `system.information_schema.column_tags`
      * `system.information_schema.table_tags`
      * `system.information_schema.tables`
      * `system.information_schema.columns`

To allow Immuta to automatically import table and column tags from Databricks Unity Catalog, enable Databricks Unity Catalog tag ingestion in the external catalog section of the Immuta app settings page.

1. Navigate to the <i class="fa-gear">:gear:</i> **App Settings** page.
2. Scroll to **2 External Catalogs**, and click **Add Catalog**.
3. Enter a **Display Name** and select **Databricks Unity Catalog** from the dropdown menu.
4. Click **Save** and confirm your changes.

## Setting up the required Databricks service principal

If you need instruction for setting up your Databricks service principal **before registering your connection**, see the steps below.

### Creating the Databricks service principal

In Databricks, [create a service principal](https://docs.databricks.com/aws/en/admin/users-groups/manage-service-principals) with the privileges listed below. Immuta uses this service principal continuously to orchestrate Unity Catalog policies and maintain state between Immuta and Databricks.

* `USE CATALOG` and `MANAGE` on all catalogs containing securables you want registered as Immuta data sources.
* `USE SCHEMA` on all schemas containing securables you want registered as Immuta data sources.
* `MODIFY` and `SELECT` on all securables you want registered as Immuta data sources. *The* `MODIFY` *privilege is not required for materialized views registered as Immuta data sources, since* `MODIFY` *is not a supported privilege on that object type in* [*Databricks*](https://docs.databricks.com/aws/en/data-governance/unity-catalog/manage-privileges/privileges#privilege-types-by-securable-object-in-unity-catalog)*.*
* [Workspace admin permissions](https://docs.databricks.com/aws/en/admin/users-groups/manage-service-principals#assign-the-workspace-admin-role-to-a-service-principal) (If your service principal is a regular user in Databricks and not an actual service principal in Databricks, follow the guidance for [managing users in Databricks documentation](https://docs.databricks.com/aws/en/admin/users-groups/users#assign-the-workspace-admin-role-to-a-user) to grant this user the workspace admin permission.)

{% hint style="info" %}
`MANAGE` and `MODIFY` are required so that the service principal can apply row filters and column masks on the securable; to do so, the service principal must also have `SELECT` on the securable as well as `USE CATALOG` on its parent catalog and `USE SCHEMA` on its parent schema. Since privileges are inherited, you can grant the service principal the `MODIFY` and `SELECT` privilege on all catalogs or schemas containing Immuta data sources, which automatically grants the service principal the `MODIFY` and `SELECT` privilege on all current and future securables in the catalog or schema. The service principal also inherits `MANAGE` from the parent catalog for the purpose of applying row filters and column masks, but that privilege must be set directly on the parent catalog in order for grants to be fully applied.
{% endhint %}

See the [Databricks documentation](https://docs.databricks.com/en/data-governance/unity-catalog/manage-privileges/privileges.html) for more details about Unity Catalog privileges and securable objects.

### Configuring query audit privileges

[Grant the service principal access to the Databricks Unity Catalog system tables](https://docs.databricks.com/en/administration-guide/system-tables/index.html#grant-access-to-system-tables). For Databricks Unity Catalog audit to work, Immuta must have, at minimum, the following access.

* `USE CATALOG` on the `system` catalog
* `USE SCHEMA` on the `system.access` and `system.query` schemas
* `SELECT` on the following system tables:

  * `system.access.table_lineage`
  * `system.access.column_lineage`
  * `system.access.audit`
  * `system.query.history`

  Access to system tables is governed by Unity Catalog. No user has access to these system schemas by default. To grant access, a user that is both a metastore admin and an account admin must grant `USE_SCHEMA` and `SELECT` privileges on the system schemas to the service principal. See [Manage privileges in Unity Catalog](https://docs.databricks.com/en/data-governance/unity-catalog/manage-privileges/index.html).

[^1]: A Databricks user authorized to create a Databricks service principal must create one for Immuta. This service principal is used continuously by Immuta to orchestrate Unity Catalog policies and maintain state between Immuta and Databricks.

[^2]: *The* `MODIFY` *privilege is not required for materialized views registered as Immuta data sources, since* `MODIFY` *is not a supported privilege on that object type in* [*Databricks*](https://docs.databricks.com/aws/en/data-governance/unity-catalog/manage-privileges/privileges#privilege-types-by-securable-object-in-unity-catalog)*.*

[^3]: These privileges are required to enable query audit. See [Manage privileges in Unity Catalog](https://docs.databricks.com/en/data-governance/unity-catalog/manage-privileges/index.html) for details.

[^4]: Your display name cannot be any of the following words: `data`, `connection`, `object`, `crawl`, `search`, `settings`, `metadata`, `permission`, `sync`, `bulk`, and `upgrade`.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://documentation.immuta.com/saas/configuration/integrations/databricks/databricks-unity-catalog/how-to-guides/connect-unity-catalog.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.