Configure a Databricks Unity Catalog Integration
Last updated
Was this helpful?
Last updated
Was this helpful?
Immuta’s integration with Unity Catalog allows you to manage multiple Databricks workspaces through Unity Catalog while protecting your data with Immuta policies. Instead of manually creating UDFs or granting access to each table in Databricks, you can author your policies in Immuta and have Immuta manage and enforce Unity Catalog access-control policies on your data in Databricks clusters or SQL warehouses.
Use the /integrations endpoint to
The following permissions and personas are used in the registration process.
Immuta user: An Immuta user with the APPLICATION_ADMIN Immuta permission must configure the Databricks Unity Catalog integration.
Databricks user: The Databricks user must have the following privileges.
Account admin
CREATE CATALOG privilege on the Unity Catalog metastore to create an Immuta-owned catalog and tables
(only required if enabling query audit)
:
USE CATALOG and MANAGE on all catalogs containing securables registered as Immuta data sources and USE SCHEMA on all schemas containing securables registered as Immuta data sources.
MODIFY and SELECT on all securables registered as Immuta data sources. MANAGE and MODIFY are required so that the service principal can apply row filters and column masks on the securable; to do so, the service principal must also have SELECT on the securable as well as USE CATALOG on its parent catalog and USE SCHEMA on its parent schema. Since privileges are inherited, you can grant the service principal the MODIFY and SELECT privilege on all catalogs or schemas containing Immuta data sources, which automatically grants the service principal the MODIFY and SELECT privilege on all current and future securables in the catalog or schema. The service principal also inherits MANAGE from the parent catalog for the purpose of applying row filters and column masks, but that privilege must be set directly on the parent catalog in order for grants to be fully applied.
Optionally, to include audit, the service principal needs the following additional privileges:
USE CATALOG on system catalog
USE SCHEMA on system.access schema
SELECT on system.access.audit table
SELECT on system.access.table_lineage table
SELECT on system.access.column_lineage table
Access to system tables is governed by Unity Catalog. No user has access to these system schemas by default. To grant access, a user that is both a metastore admin and an account admin must grant USE and SELECT permissions on the system schemas to the service principal. See . The system.access schema must also be on the metastore before it can be used.
Access token authentication: If using this method, generate a personal access token for the service principal that Immuta will use to manage policies in Unity Catalog. This service principal must have the privileges listed above for the metastore associated with the Databricks workspace.
Enable Databricks Unity Catalog on the Immuta app settings page:
Click the App Settings icon in the left sidebar.
Scroll to the Global Integrations Settings section and check the Enable Databricks Unity Catalog support in Immuta checkbox.
You have two options for configuring your Databricks Unity Catalog integration:
Copy the request example, and replace the values with your own as directed to configure the integration settings. The examples provided use JSON format, but the request also accepts YAML.
Change the config values to your own, where
workspaceUrl is your Databricks workspace URL.
httpPath is the HTTP path of your Databricks cluster or SQL warehouse.
token is the Databricks personal access token. This is the access token for the Immuta service principal.
catalog is the name of the Databricks catalog Immuta will create to store internal entitlements and other user data specific to Immuta. This catalog will only be readable for the Immuta service principal and should not be granted to other users. The catalog name may only contain letters, numbers, and underscores and cannot start with a number.
A successful response includes the validation tests statuses.
To manually configure the integration, complete the following steps:
Copy the request example, and replace the values with your own as directed to configure the integration settings. The examples provided use JSON format, but the request also accepts YAML.
Change the config values to your own, where
workspaceUrl is your Databricks workspace URL.
httpPath is the HTTP path of your Databricks cluster or SQL warehouse.
token is the Databricks personal access token. This is the access token for the Immuta service principal.
catalog is the name of the Databricks catalog Immuta will create to store internal entitlements and other user data specific to Immuta. This catalog will only be readable for the Immuta service principal and should not be granted to other users. The catalog name may only contain letters, numbers, and underscores and cannot start with a number.
Run the script returned in the response in your Databricks environment.
Response
The response returns the script for you to run in your environment.
workspaceUrl is your Databricks workspace URL.
httpPath is the HTTP path of your Databricks cluster or SQL warehouse.
token is the Databricks personal access token. This is the access token for the Immuta service principal.
catalog is the name of the Databricks catalog Immuta will create to store internal entitlements and other user data specific to Immuta. This catalog will only be readable for the Immuta service principal and should not be granted to other users. The catalog name may only contain letters, numbers, and underscores and cannot start with a number.
A successful response includes the validation tests statuses.
Copy the request example.
Copy the request example.
Copy the request example, and replace the values with your own as directed to configure the integration settings. The examples provided use JSON format, but the request also accepts YAML.
This example updates the access token.
Replace the {id} request parameter with the unique identifier of the integration you want to update.
Change the config values to your own, where
workspaceUrl is your Databricks workspace URL.
httpPath is the HTTP path of your Databricks cluster or SQL warehouse.
token is the Databricks personal access token. This is the access token for the Immuta service principal.
catalog is the name of the Databricks catalog Immuta will create to store internal entitlements and other user data specific to Immuta. This catalog will only be readable for the Immuta service principal and should not be granted to other users. The catalog name may only contain letters, numbers, and underscores and cannot start with a number.
A successful response includes the validation tests statuses.
Copy the request example.
Replace the {id} request parameter with the unique identifier of the integration you want to delete.
See the for more details about Unity Catalog privileges and securable objects.
OAuth machine-to-machine (M2M) authentication: If using this method, follow for the Immuta service principal. This service principal must have the privileges listed above for the metastore associated with the Databricks workspace.
: When performing an automatic setup, the Databricks personal access token you configure below must be attached to an account with for the metastore associated with the specified Databricks workspace. Immuta creates the catalogs, schemas, tables, and functions using the integration's configured personal access token.
: Run the Immuta script in Databricks yourself to create the catalog. You can also modify the script to customize your storage location for tables, schemas, or catalogs. The user running the script needs to have the CREATE CATALOG permission on the workspace metastore. The Databricks personal access token you configure must be attached to an account with the Databricks permissions listed in the .
Required permissions: When performing an automatic setup, the credentials provided must have the .
See the for parameter definitions, value types, and additional configuration options.
Replace the Immuta URL and with your own.
Replace the Immuta URL and with your own.
oAuthClientConfig specifies your client ID, client secret, and authority URL. See the for details about child parameters.
The response returns the status of the Databricks Unity Catalog integration configuration connection. See the for details about the response schema.
An unsuccessful request returns the status code and an error message. See the for a list of statuses, error messages, and troubleshooting guidance.
Required permissions: When performing a manual setup, the Databricks user running the script must have the .
See the for parameter definitions, value types, and additional configuration options.
Replace the Immuta URL and with your own.
Replace the Immuta URL and with your own.
oAuthClientConfig specifies your client ID, client secret, and authority URL. See the for details about child parameters.
Copy the request example, and replace the values with your own as directed to configure the integration settings. The examples provided use JSON format, but the request also accepts YAML. The payload you provide must match the payload sent when .
See the for parameter definitions, value types, and additional configuration options.
Replace the Immuta URL and with your own.
Pass the same payload you sent when , where
Replace the Immuta URL and with your own.
Pass the same payload you sent when , where
oAuthClientConfig specifies your client ID, client secret, and authority URL. See the for details about child parameters.
The response returns the status of the Databricks Unity Catalog integration configuration connection. See the for details about the response schema.
An unsuccessful request returns the status code and an error message. See the for a list of statuses, error messages, and troubleshooting guidance.
Replace the Immuta URL and with your own.
Replace the {id} request parameter with the unique identifier of the integration you want to get. Alternatively, you can get a list of all integrations and their IDs with the .
The response returns a Databricks Unity Catalog integration configuration. See the for details about the response schema. An unsuccessful request returns the status code and an error message. See the for a list of statuses, error messages, and troubleshooting guidance.
Replace the Immuta URL and with your own.
The response returns the configuration for all integrations. See the for details about the response schema. An unsuccessful request returns the status code and an error message. See the for a list of statuses, error messages, and troubleshooting guidance.
See the for parameter definitions, value types, and additional configuration options.
Replace the Immuta URL and with your own.
The response returns the status of the Databricks Unity Catalog integration configuration connection. See the for details about the response schema.
An unsuccessful request returns the status code and an error message. See the for a list of statuses, error messages, and troubleshooting guidance.
Replace the Immuta URL and with your own.
The response returns the status of the Databricks Unity Catalog integration configuration that has been deleted. See the for details about the response schema. An unsuccessful request returns the status code and an error message. See the for a list of statuses, error messages, and troubleshooting guidance.