Databricks Spark with Unity Catalog Pre-Configuration Details
Prerequisites
Databricks Runtime 11.3.
Unity Catalog enabled on your Databricks cluster.
Unity Catalog metastore created and attached to a Databricks workspace.
The metastore owner you are using to manage permissions has been granted access to all catalogs, schemas, and tables that will be protected by Immuta. Data protected by Immuta should only be granted to privileged users in Unity Catalog so that the only view of that data is through an Immuta-enabled cluster.
You have generated a personal access token for the metastore owner that Immuta can use to read data in Unity Catalog.
You do not plan to use non-Unity Catalog enabled clusters with Immuta data sources. Once enabled, all access to data source tables must be on Databricks clusters with Unity Catalog enabled on runtime 11.3.
Feature Availability
| Project Workspaces | Databricks Tag Ingestion | User Impersonation | Native Query Audit | Multiple Integrations |
|---|---|---|---|---|
For details about the supported features listed in the table above, see the pre-configuration details page for Databricks.
Supported Databricks Cluster Configurations
The table below outlines the integrations supported for various Databricks cluster configurations. For example, the only integration available to enforce policies on a cluster configured to run on Databricks Runtime 9.1 is the Databricks Spark integration.
| Example cluster | Databricks Runtime | Unity Catalog in Databricks | Databricks Spark integration | Databricks Spark with Unity Catalog support | Databricks Unity Catalog integration |
|---|---|---|---|---|---|
Cluster 1 | 9.1 | Unavailable | Unavailable | ||
Cluster 2 | 10.4 | Unavailable | Unavailable | ||
Cluster 3 | 11.3 | Unavailable | |||
Cluster 4 | 11.3 | ||||
Cluster 5 | 11.3 |
Legend:
✅ The feature or integration is enabled.
⛔ The feature or integration is disabled.
Databricks Metastore Magic
Databricks metastore magic allows you to migrate your data from the Databricks legacy Hive metastore to the Unity Catalog metastore while protecting data and maintaining your current processes in a single Immuta instance.
No configuration is necessary to enable this feature. For more details, see the Databricks metastore magic overview.
Caveats and Limitations
Native workspaces are not supported. Creating a native workspace on a Unity Catalog enabled host is undefined behavior and may cause data loss or crashes.
Tables must be GRANTed access to the Databricks metastore owner token configured for the integration. For the table to be accessible to the user, the full chain of catalog, schema, and table must all have the appropriate grants to this administrator user to allow them to SELECT from the table.
Direct file access to Immuta data sources is not supported.
Limited Enforcement (called available until protected by policy on the App Settings page), which makes Immuta clusters available to all Immuta users until protected by a policy, is not supported. You must set
IMMUTA_SPARK_DATABRICKS_ALLOW_NON_IMMUTA_READSandIMMUTA_SPARK_DATABRICKS_ALLOW_NON_IMMUTA_WRITEStofalsein your cluster policies manually or by selecting Protected until made available by policy in the Databricks integration section of the App Settings page.R notebooks may have path-related errors accessing tables.
Databricks on Azure will return errors when creating a database in a scratch location when Unity Catalog is enabled.
Databricks accounts deployed on Google Cloud Platform are not supported.
Next
Configure Databricks Spark integration with Unity Catalog support.
Last updated