Register a Databricks Unity Catalog Connection
Last updated
Last updated
Public preview
This feature is public preview and available to select accounts. Reach out to your Immuta support professional to enable it on your tenant.
The following permissions and personas are used in the registration process.
An Immuta user with the CREATE_DATA_SOURCE
Immuta permission must register the Databricks Unity Catalog connection.
A Databricks user authorized to create a Databricks service principal must create one for Immuta. This service principal is used continuously by Immuta to orchestrate Unity Catalog policies and maintain state between Immuta and Databricks. This service principal needs the following Databricks privileges:
USE CATALOG
and MANAGE
on all catalogs containing securables registered as Immuta data sources and USE SCHEMA
on all schemas containing securables registered as Immuta data sources.
MODIFY
and SELECT
on all securables registered as Immuta data sources. MANAGE
and MODIFY
are required so that the service principal can apply row filters and column masks on the securable; to do so, the service principal must also have SELECT
on the securable as well as USE CATALOG
on its parent catalog and USE SCHEMA
on its parent schema. Since privileges are inherited, you can grant the service principal the MODIFY
and SELECT
privilege on all catalogs or schemas containing Immuta data sources, which automatically grants the service principal the MODIFY
and SELECT
privilege on all current and future securables in the catalog or schema. The service principal also inherits MANAGE
from the parent catalog for the purpose of applying row filters and column masks, but that privilege must be set directly on the parent catalog in order for grants to be fully applied.
See the Databricks documentation for more details about Unity Catalog privileges and securable objects.
Optionally, to include audit, the service principal needs the following additional privileges:
USE CATALOG
on system
catalog
USE SCHEMA
on system.access
schema
SELECT
on system.access.audit
table
SELECT
on system.access.table_lineage
table
SELECT
on system.access.column_lineage
table
Access to system tables is governed by Unity Catalog. No user has access to these system schemas by default. To grant access, a user that is both a metastore admin and an account admin must grant USE
and SELECT
permissions on the system schemas to the service principal. See Manage privileges in Unity Catalog. The system.access
schema must also be enabled on the metastore before it can be used.
Unity Catalog metastore created and attached to a Databricks workspace. See the Databricks Unity Catalog reference guide for information on workspaces and catalog isolation support with Immuta.
Unity Catalog enabled on your Databricks cluster or SQL warehouse. All SQL warehouses have Unity Catalog enabled if your workspace is attached to a Unity Catalog metastore. Immuta recommends linking a SQL warehouse to your Immuta tenant rather than a cluster for both performance and availability reasons.
Click Data and select the Infrastructure tab in the navigation menu.
Click the + Add Host button.
Select the Databricks data platform tile.
Enter the connection information:
Host: The hostname of your Databricks workspace.
Port: Your Databricks port.
HTTP Path: The HTTP path of your Databricks cluster or SQL warehouse.
Immuta Catalog: The name of the catalog Immuta will create to store internal entitlements and other user data specific to Immuta. This catalog will only be readable for the Immuta service principal and should not be granted to other users. The catalog name may only contain letters, numbers, and underscores and cannot start with a number.
Connection Key: The connection key represents the unique name of your connection and will be used as prefix in the name for all data objects associated with this connection. It will also appear as the display name in the UI and will be used in all API calls made to update or delete the connection.
Click Next.
Select Access Token authentication method from the dropdown menu.
Enter the Access Token in the Immuta System Account Credentials section. This is the access token for the Immuta service principal. This service principal must have the metastore privileges listed in the requirements section at the top of this page for the metastore associated with the Databricks workspace. If this token is configured to expire, update this field regularly for the integration to continue to function. This authentication information will be included in the script populated later on the page.
Copy the provided script and run it in Databricks as a user with the CREATE CATALOG
privilege on the Unity Catalog metastore.
Click Validate Connection.
If the connection is successful, click Next. If there are any errors, check the connection details and credentials to ensure they are correct and try again.
Ensure all the details are correct in the summary and click Complete Setup.