Skip to content

Immuta Databricks Spark Integration with Unity Catalog Support Pre-Configuration Details

Prerequisites

  • Databricks Runtime 11.2.
  • Unity Catalog enabled on your Databricks cluster.
  • Unity Catalog metastore created and attached to a Databricks workspace.
  • The metastore owner you are using to manage permissions has been granted access to all catalogs, schemas, and tables that will be protected by Immuta. Data protected by Immuta should only be granted to privileged users in Unity Catalog so that the only view of that data is through an Immuta-enabled cluster.
  • You have generated a personal access token for the metastore owner that Immuta can use to read data in Unity Catalog.
  • You do not plan to use non-Unity Catalog enabled clusters with Immuta data sources. Once enabled, all access to data source tables must be on Databricks clusters with Unity Catalog enabled on runtime 11.2.

Feature Availability

Project Workspaces Databricks Tag Ingestion User Impersonation Native Query Audit Multiple Integrations
❌ ❌ ✅ ✅ ✅

For details about the supported features listed in the table above, see the pre-configuration details page for Databricks.

Caveats and Limitations

  • Native workspaces are not supported. Creating a native workspace on a Unity Catalog enabled host is undefined behavior and may cause data loss or crashes.
  • Tables must be GRANTed access to the Databricks metastore owner token configured for the integration. For the table to be accessible to the user, the full chain of catalog, schema, and table must all have the appropriate grants to this administrator user to allow them to SELECT from the table.
  • Direct file access to Immuta data sources is not supported.
  • Limited Enforcement (which makes Immuta clusters available to all Immuta users until protected by a policy) is not supported.
  • R notebooks may have path-related errors accessing tables.
  • Databricks on Azure will return errors when creating a database in a scratch location when Unity Catalog is enabled.

Next

Configure Databricks Spark integration with Unity Catalog support.