Immuta offers two integrations for Amazon Redshift:

• Amazon Redshift integration: In this integration, Immuta uses connections to configure the integration and register data objects in a single step. Once data is registered, Immuta can enforce access controls on that data.

• Amazon Redshift Spectrum integration: In this integration, Immuta generates policy-enforced views in your configured Redshift schema for tables registered as Immuta data sources. The integration is configured separately from data source registration.

The Amazon Redshift integration allows you to configure your integration and register data from Amazon Redshift in Immuta in a single step. Once data is registered, Immuta can enforce access controls on that data.

This getting started guide outlines how to connect Amazon Redshift to Immuta.

How-to guide

Reference guides

• : This guide describes the design and components of the integration.

• : This guide provides an overview of the Immuta features that provide security for your users and that allow you to prove compliance and monitor for anomalies.

• : This guide provides an overview of how to protect securables with Immuta policies.

Immuta gives everyone fast, governed access to data with the built-in controls, collaboration workflows, automated provisioning, and continuous monitoring you need to keep risk low and compliance high.

Configure Immuta

The how-to guides linked on this page illustrate how to use Amazon Redshift with Immuta. See the for information about the Amazon Redshift integration.

Immuta integrates with your data platforms so you can register your data and effectively manage access controls on that data.

This section includes guidance for connecting your data platform and keeping it synced with Immuta.

This reference guide outlines the features, policies, and audit capabilities of each data platform Immuta supports.

Once data is registered through the Amazon Redshift connection, you will access your data through Amazon Redshift as you normally would. If you are subscribed to the data source, Immuta grants you access to the data in Amazon Redshift.

When you submit a query, the SQL client submits the query to Amazon Redshift, which then processes the query and determines what data your role is allowed to see. Then, Amazon Redshift queries the database and returns the query results to the SQL client, which then returns policy-enforced data to you.

The diagram below illustrates how Immuta, Amazon Redshift, and the SQL client interact when a user queries data registered in Immuta.

Querying data

Because subscription policies are managed through roles, you must be acting under the role Immuta creates for you (immuta_<username>) to get access to the data sources you are subscribed to.

Once data is registered through the AWS Lake Formation connection, you will access your data in one of these AWS analytic engines as you normally would:

• Amazon Athena

• Amazon EMR Spark

• Amazon Redshift Spectrum

If you are subscribed to the data source, Immuta either directly grants you access to the resource through Lake Formation or generates and assigns a Lake Formation tag to that resource to grant you access. See the for details about how policies are enforced.

When you submit a query, the analytic engine requests metadata from Glue Data Catalog, which then queries Lake Formation to determine what data you are allowed to see. Then, the analytic engine requests temporary access from Lake Formation, retrieves the data from S3, and filters the data to return policy-enforced data to you.

The diagram below illustrates how the analytic engine interacts with Glue Data Catalog and Lake Formation to access data.

In this integration, Immuta generates policy-enforced views in a schema in your configured Azure Synapse Analytics Dedicated SQL pool for tables registered as Immuta data sources.

Getting started

This guide outlines how to integrate Azure Synapse Analytics with Immuta.

How-to guide

: Configure the integration in Immuta.

Reference guide

: This guide describes the design and components of the integration.

This integration allows you to manage and access data in your Databricks account across all of your workspaces. With Immuta’s Databricks Unity Catalog integration, you can write your policies in Immuta and have them enforced automatically by Databricks across data in your Unity Catalog metastore.

Getting started

This getting started guide outlines how to integrate Databricks Unity Catalog with Immuta.

How-to guides

• : Migrate from the legacy Databricks Spark integrations to the Databricks Unity Catalog integration.

Reference guide

: This guide describes the design and components of the integration.

Once data is registered through the Databricks Lakebase connection, you will access your data as you normally would. If you are subscribed to the data source, Immuta grants you access to the data through a PostgreSQL role.

When you submit a query (through PostgreSQL or through the Databricks Lakebase instance), the PostgreSQL client submits the SQL query to the PostgreSQL server, which then processes the query and determines what data your role is allowed to see. Then, the PostgreSQL server queries the database and returns the query results to the PostgreSQL client, which then returns policy-enforced data to you.

The diagram below illustrates how Immuta, the PostgreSQL server, and PostgreSQL client interact to access data.

Permissions

The user registering the connection must have the permissions below.

In the Amazon Redshift integration, Immuta administers Amazon Redshift privileges on data registered in Immuta. Then, Immuta users who have been granted access to the data sources can query them.

The sequence diagram below outlines the events that occur when an Immuta user who is subscribed to a data source queries it in Amazon Redshift.

Registering a connection

The Amazon Redshift integration is configured and data is registered through , an Immuta feature that allows administrators to register data objects in a technology through a single connection to make data registration more scalable for your organization.

In this integration, Immuta generates policy-enforced views in your configured Amazon Redshift schema for tables registered as Immuta data sources.

This guide outlines how to integrate Amazon Redshift Spectrum with Immuta.

In the Lake Formation integration, Immuta orchestrates Lake Formation access controls on data registered in the Glue Data Catalog. Then, Immuta users who have been granted access to the Glue Data Catalog table or view can query it using one of these analytic engines:

Immuta offers several features to provide security for your users and to prove compliance and monitor for anomalies.

Authentication

Registering the connection

The Databricks Lakebase integration registers data from Databricks Lakebase in Immuta and enforces subscription policies on that data.

Immuta offers several features to provide security for your users and to prove compliance and monitor for anomalies.

Authentication

Registering the connection

In the Databricks Lakebase integration, Immuta administers PostgreSQL privileges on data registered in Immuta. Then, Immuta users who have been granted access to the tables can query them with policies enforced.

The sequence diagram below outlines the events that occur when an Immuta user who is subscribed to a data source queries it in PostgreSQL.

Registering a connection

Databricks Lakebase is configured and data is registered through , an Immuta feature that allows administrators to register data objects in a technology through a single connection to make data registration more scalable for your organization.

The Databricks Spark integration is one of two integrations Immuta offers for Databricks.

In this integration, Immuta installs an Immuta-maintained Spark plugin on your Databricks cluster. When a user queries data that has been registered in Immuta as a data source, the plugin injects policy logic into the plan Spark builds so that the results returned to the user only include data that specific user should see.

The reference guides in this section are written for Databricks administrators who are responsible for setting up the integration, securing Databricks clusters, and setting up users:

In the context of the Databricks Spark integration, Immuta uses the term ephemeral to describe data sources where the associated compute resources can vary over time. This means that the compute bound to these data sources is not fixed and can change. All Databricks data sources in Immuta are ephemeral.

Ephemeral overrides are specific to each data source and user. They effectively bind cluster compute resources to a data source for a given user. Immuta uses these overrides to determine which cluster compute to use when connecting to Databricks for various maintenance operations.

The operations that use the ephemeral overrides include

• Visibility checks on the data source for a particular user. These checks assess how to apply row-level policies for specific users.

The MariaDB integration allows you to register data from MariaDB in Immuta.

The Oracle integration allows you to register data from Oracle in Immuta.

1. Click the App Settings icon in the navigation menu and scroll to the Global Integration Settings section.

2. Click the Enable Snowflake Low Row Access Policy Mode checkbox to enable the feature.

3. Confirm to allow Immuta to automatically disable impersonation for the Snowflake integration. If you do not confirm, you will not be able to enable Snowflake low row access policy mode.

The MySQL integration uses connections to register data from MySQL in Immuta.

Once data is registered through the PostgreSQL connection, you will access your data through your PostgreSQL client as you normally would. If you are subscribed to the data source, Immuta grants you access to the data in PostgreSQL.

When you submit a query, the PostgreSQL client submits the SQL query to the PostgreSQL server, which then processes the query and determines what data your role is allowed to see. Then, the PostgreSQL server queries the database and returns the query results to the PostgreSQL client, which then returns policy-enforced data to you.

The diagram below illustrates how Immuta, the PostgreSQL server, and PostgreSQL client interact to access data.

The how-to guides linked on this page illustrate how to use AWS Lake Formation with Immuta. See the reference guide for information about the AWS Lake Formation integration.

In the AWS Lake Formation integration, Immuta orchestrates Lake Formation access controls on data registered in the Glue Data Catalog. Then, Immuta users who have been granted access to the Glue Data Catalog table or view can query it using one of these analytic engines:

• Amazon Athena

• Amazon EMR Spark

• Amazon Redshift Spectrum

The sequence diagram below outlines the events that occur when an Immuta user who is subscribed to a data source submits a query in their AWS analytic engine.

See the for more details about Lake Formation access controls.

Registering a connection

AWS Lake Formation is configured and data is registered through , an Immuta feature that allows administrators to register data objects in a technology through a single connection to make data registration more scalable for your organization.

Once the Lake Formation connection is registered, you can author policies in Immuta to orchestrate Lake Formation access controls.

See the for more details about registering a connection.

Protecting data

After Glue Data Catalog views and tables are registered in Immuta, you can author subscription policies in Immuta to orchestrate Lake Formation access controls. Once a subscription policy is applied, users can be subscribed to data sources in the following ways:

• Manually subscribed: If a data owner , Immuta issues a grant directly to the data object in AWS.

• Automatically subscribed through policy logic: When a policy is applied to a data source, users who meet the conditions of the policy will be automatically subscribed to the data source. Then, Immuta generates a Lake Formation tag and applies it to the corresponding data object in AWS and grants subscribers access to that tag, which in turn grants them access to the data. See the for details about this process.

Consider the following example that illustrates how Immuta enforces a subscription policy that only allows users in the analysts group to access the yellow-table. When this policy is authored and applied to the data source, Immuta generates a Lake Formation (LF) tag that is applied to the Glue Data Catalog yellow-table and permissions on that tag are granted to all AWS users (registered in Immuta) that are part of the analysts group.

In the image above, the user in the analysts group accesses yellow-table, while the user who is a part of the research group is denied access.

See the for guidance on applying a subscription policy to a data source. See the page for details about the subscription policy types supported and permissions Immuta grants on securables registered as Immuta data sources.

This page describes the Azure Synapse Analytics integration, through which Immuta applies policies directly in Azure Synapse Analytics. For a tutorial on configuring Azure Synapse Analytics see the Azure Synapse Integration page.

Overview

The Azure Synapse Analytics is a policy push integration that allows Immuta to apply policies directly in Azure Synapse Analytics Dedicated SQL pools without the need for users to go through a proxy. Instead, users can work within their existing Synapse Studio and have per-user policies dynamically applied at query time.

Architecture

This integration works on a per-Dedicated-SQL-pool basis: all of Immuta's policy definitions and user entitlements data need to be in the same pool as the target data sources because Dedicated SQL pools do not support cross-database joins. Immuta creates schemas inside the configured Dedicated SQL pool that contain policy-enforced views that users query.

When the integration is configured, the Application Admin specifies the

• Immuta Database: This is the pre-existing database Immuta uses. Immuta will create views from the tables contained in this database, and all schemas and views created by Immuta will exist in this database, such as the schemas immuta_system, immuta_functions, and the immuta_procedures that contain the tables, views, UDFs, and stored procedures that support the integration.

• Immuta Schema: The schema that Immuta manages. All views generated by Immuta for tables registered as data sources will be created in this schema.

For a tutorial on configuring the integration see the .

Data source naming convention

Synapse data sources are represented as views and are under one schema instead of a database, so their view names are a combination of their schema and table name, separated by an underscore.

For example, with a configuration that uses IMMUTA as the schema in the database dedicated_pool, the view name for the data source dedicated_pool.tpc.case would be dedicated_pool.IMMUTA.tpc_case.

You can see the view information on the data source details page under Connection Information.

Policy enforcement

This integration uses webhooks to keep views up-to-date with the corresponding Immuta data sources. When a data source or policy is created, updated, or disabled, a webhook is called that creates, modifies, or deletes the dynamic view in the Immuta schema. Note that only standard views are available because Azure Synapse Analytics Dedicated SQL pools do not support secure views.

Integration health status

The definitions for each status and the state of configured data platform integrations is available in the .

Data flow

1. An Immuta Application Administrator , registering their initial Synapse Dedicated SQL pool with Immuta.

2. Immuta creates Immuta schemas inside the configured Synapse Dedicated SQL pool.

3. A Data Owner in Immuta as data sources. A Data Owner, Data Governor, or Administrator or in Immuta.

When you enable Unity Catalog, Immuta automatically migrates your existing Databricks data sources in Immuta to reference the legacy hive_metastore catalog to account for Unity Catalog's three-level hierarchy. New data sources will reference the Unity Catalog metastore you create and attach to your Databricks workspace.

Because the hive_metastore catalog is not managed by Unity Catalog, existing data sources in the hive_metastore cannot have Unity Catalog access controls applied to them. Data sources in the Hive Metastore must be managed by the Databricks Spark integration.

To allow Immuta to administer Unity Catalog access controls on that data, move the data to Unity Catalog and re-register those tables in Immuta by completing the steps below. If you don't move all data before configuring the integration, metastore magic will protect your existing data sources throughout the migration process.

1. Ensure that all Databricks clusters that have Immuta installed are stopped and the Immuta configuration is removed from the cluster. Immuta-specific cluster configuration is no longer needed with the Databricks Unity Catalog integration.

2. Move all data into Unity Catalog before configuring Immuta with Unity Catalog. Existing data sources will need to be re-created after they are moved to Unity Catalog and the Unity Catalog integration is configured.

3. .

This integration enforces policies on Databricks securables registered in the legacy Hive metastore. Once these securables are registered as Immuta data sources, users can query policy-enforced data on Databricks clusters.

The guides in this section outline how to integrate Databricks Spark with Immuta.

Getting started

This getting started guide outlines how to integrate Databricks with Immuta.

How-to guides

• : Configure the Databricks Spark integration.

• : Manually update your cluster to reflect changes in the Immuta init script or cluster policies.

• : Register a Databricks library with Immuta as a trusted library to avoid Immuta security manager errors when using third-party libraries.

Reference guides

• : This guide describes the design and components of the integration.

• : This guide provides an overview of the Immuta features that provide security for your users and Databricks clusters and that allow you to prove compliance and monitor for anomalies.

• : This guide provides an overview of registering Databricks securables and protecting them with Immuta policies.

If a Databricks cluster needs to be manually updated to reflect changes in the Immuta init script or cluster policies, you can remove and set up your integration again to get the updated policies and init script.

1. Log in to Immuta as an Application Admin.

2. Click the App Settings icon in the navigation menu and scroll to the Integration Settings section.

3. Your existing Databricks Spark integration should be listed here; expand it and note the configuration values. Now select Remove to remove your integration.

4. Click Add Integration and select Databricks Integration to add a new integration.

5. Enter your Databricks Spark integration settings again as configured previously.

6. Click Add Integration to add the integration, and then select Configure Cluster Policies to set up the updated cluster policies and init script.

7. Select the cluster policies you wish to use for your Immuta-enabled Databricks clusters.

8. Automatically push cluster policies and the init script (recommended) or manually update your cluster policies.

   • Automatically push cluster policies

     1. Select Automatically Push Cluster Policies and enter your privileged Databricks access token. This token must have privileges to write to cluster policies.
9. Restart any Databricks clusters using these updated policies for the changes to take effect.

1. In the Databricks Clusters UI, install your third-party library .jar or Maven artifact with Library Source Upload, DBFS, DBFS/S3, or Maven. Alternatively, use the Databricks libraries API.

2. In the Databricks Clusters UI, add the IMMUTA_SPARK_DATABRICKS_TRUSTED_LIB_URIS property as a Spark environment variable and set it to your artifact's URI. To specify more than one trusted library, comma delimit the URIs:

1. Once you've finished making your changes, restart the cluster.

2. Once the cluster is up, execute a command in a notebook. If the trusted library installation is successful, you should see driver log messages like this:

This page outlines the configuration for setting up project UDFs, which allow users to set their current project in Immuta through Spark. For details about the specific functions available and how to use them, see the Use Project UDFs (Databricks) page.

1. Lower the web service cache timeout in Immuta:

   1. Click the App Settings icon and scroll to the HDFS Cache Settings section.

   2. Lower the Cache TTL of HDFS user names (ms) to 0.

2. Raise the cache timeout on your Databricks cluster: In the Spark environment variables section, set the IMMUTA_CURRENT_PROJECT_CACHE_TIMEOUT_SECONDS and IMMUTA_PROJECT_CACHE_TIMEOUT_SECONDS to high values (like 10000).

   Note: These caches will be invalidated on cluster when a user calls immuta.set_current_project, so they can effectively be cached permanently on cluster to avoid periodically reaching out to the web service.

This page provides guidelines for troubleshooting issues with the Databricks Spark integration and resolving Py4J security and Databricks trusted library errors.

Debugging the integration

For easier debugging of the Databricks Spark integration, follow the recommendations below.

• Enable cluster init script logging:

  • In the cluster page in Databricks for the target cluster, navigate to Advanced Options -> Logging.

  • Change the Destination from NONE to DBFS and change the path to the desired output location. Note: The unique cluster ID will be added onto the end of the provided path.

• View the Spark UI on your target Databricks cluster: On the cluster page, click the Spark UI tab, which shows the Spark application UI for the cluster. If you encounter issues creating Databricks data sources in Immuta, you can also view the JDBC/ODBC Server portion of the Spark UI to see the result of queries that have been sent from Immuta to Databricks.

Using the validation and debugging notebook

The validation and debugging notebook is designed to be used by or under the guidance of an Immuta support professional. Reach out to your Immuta representative for assistance.

1. Import the notebook into a Databricks workspace by navigating to Home in your Databricks instance.

2. Click the arrow next to your name and select Import.

3. Once you have executed commands in the notebook and populated it with debugging information, export the notebook and its contents by opening the File menu, selecting Export, and then selecting DBC Archive.

Py4J security error

• Error Message: py4j.security.Py4JSecurityException: Constructor <> is not allowlisted

• Explanation: This error indicates you are being blocked by Py4J security rather than the Immuta Security Manager. Py4J security is strict and generally ends up blocking many ML libraries.

• Solution: Turn off Py4J security on the offending cluster by setting IMMUTA_SPARK_DATABRICKS_PY4J_STRICT_ENABLED=false

Databricks trusted library errors

Check the driver logs for details. Some possible causes of failure include

• One of the Immuta-configured trusted library URIs does not point to a Databricks library. Check that you have configured the correct URI for the Databricks library.

• For trusted Maven artifacts, the URI must follow this format: maven:/group.id:artifact-id:version.

• Databricks failed to install a library. Any Databricks library installation errors will appear in the Databricks UI under the Libraries tab.

Immuta is compatible with Snowflake Secure Data Sharing. Using both Immuta and Snowflake, organizations can share the policy-protected data of their Snowflake database with other Snowflake accounts with Immuta policies enforced in real time.

Prerequisites:

• Snowflake integration enabled

• Snowflake tables registered in Immuta as data sources

Create Immuta Policies to Protect the Data

Required Permission: Immuta: GOVERNANCE

to fit your organization's compliance requirements.

It's important to understand that subscription policies are not relevant to Snowflake data shares, because the act of sharing the data is the subscription policy. Data policies can be enforced on the consuming account from the producer account on a share following these instructions.

Register the Snowflake Data Consumer with Immuta

Required Permission: Immuta: USER_ADMIN

To register the Snowflake data consumer in Immuta,

1. .

2. to match the account ID for the data consumer. This value is the output on the data consumer side when SELECT CURRENT_ACCOUNT() is run in Snowflake.

3. for your organization's policies.

Create the Snowflake Data Share

Required Permission: Snowflake ACCOUNTADMIN

To share the policy-protected data source,

1. of the Snowflake table that has been registered in Immuta.

2. Grant reference usage on the Immuta database to the share you created:

   Replace the content in angle brackets above with the name of your Immuta database and Snowflake data share.

To migrate from the private preview version of table grants (available before September 2022) to the GA version, complete the steps below.

1. Navigate to the App Settings page.

2. Scroll to the Global Integrations Settings section.

3. Uncheck the Snowflake Table Grants checkbox to disable the feature.

4. Click Save. Wait for about 1 minute per 1000 users. This gives time for Immuta to drop all the previously created user roles.

5. Use the to re-enable the feature.

In the PostgreSQL integration, Immuta registers data from PostgreSQL and enforces subscription policies on that data.

This getting started guide outlines how to integrate PostgreSQL with Immuta.

How-to guide

Reference guides

• : This guide describes the design and components of the integration.

• : This guide provides an overview of the Immuta features that provide security for your users and that allow you to prove compliance and monitor for anomalies.

• : This guide provides an overview of how to protect securables with Immuta policies.

Immuta offers several features to provide security for your users and to prove compliance and monitor for anomalies.

Authentication

Registering the connection

The how-to guides linked on this page illustrate how to integrate Databricks Unity Catalog with Immuta. See the for information about the Databricks Unity Catalog integration.

Requirements:

• Unity Catalog and attached to a Databricks workspace. Immuta supports configuring a single metastore for each configured integration, and that metastore may be attached to multiple Databricks workspaces.

• Unity Catalog enabled on your Databricks cluster or SQL warehouse. All SQL warehouses have Unity Catalog enabled if your workspace is attached to a Unity Catalog metastore.

The how-to guides linked on this page illustrate how to use Databricks Lakebase with Immuta. See the reference guide for information about the Databricks Lakebase integration.

The how-to guides linked on this page illustrate how to integrate Databricks Spark with Immuta.

Requirements

• If Databricks Unity Catalog is enabled in a Databricks workspace, you must use an when you set up the Databricks Spark integration to create an Immuta-enabled cluster.

• If Databricks Unity Catalog is not enabled in your Databricks workspace, you must disable Unity Catalog in your Immuta tenant before proceeding with your configuration of Databricks Spark:

This page outlines how to enable access to DBFS in Databricks for non-sensitive data. Databricks administrators should place the desired configuration in the Spark environment variables.

DBFS FUSE mount

This Databricks feature mounts DBFS to the local cluster filesystem at /dbfs. Although disabled when using process isolation, this feature can safely be enabled if raw, unfiltered data is not stored in DBFS and all users on the cluster are authorized to see each other’s files. When enabled, the entirety of DBFS essentially becomes a scratch path where users can read and write files in /dfbs/path/to/my/file as though they were local files.

When using Delta Lake, the API does not go through the normal Spark execution path. This means that Immuta's Spark extensions do not provide protection for the API. To solve this issue and ensure that Immuta has control over what a user can access, the Delta Lake API is blocked.

Spark SQL can be used instead to give the same functionality with all of Immuta's data protections.

Requests

Below is a table of the Delta Lake API with the Spark SQL that may be used instead.

In the PostgreSQL integration, Immuta administers PostgreSQL privileges on data registered in Immuta. Then, Immuta users who have been granted access to the tables can query them with policies enforced.

The sequence diagram below outlines the events that occur when an Immuta user who is subscribed to a data source queries it in PostgreSQL.

Registering a connection

PostgreSQL is configured and data is registered through , an Immuta feature that allows administrators to register data objects in a technology through a single connection to make data registration more scalable for your organization.

Immuta offers several features to provide security for your users and to prove compliance and monitor for anomalies.

Authentication

Registering the connection

The how-to guides linked on this page illustrate how to integrate Amazon Redshift Spectrum with Immuta. See the reference guide for information about the Amazon Redshift Spectrum integration.

The how-to guides linked on this page illustrate how to integrate Azure Synapse Analytics with Immuta. See the reference guide for information about the Azure Synapse Analytics integration.

Requirement: A running Dedicated SQL pool

Immuta offers three integrations for Databricks:

• Databricks Unity Catalog integration: This integration supports working with database objects registered in Unity Catalog.

• Databricks Lakebase integration: This connection supports working with Lakebase Postgres database objects within Databricks Lakebase.

• : This integration supports working with .

Which integration should you use?

To determine which integration you should use, consider which metastore you use:

• Legacy Hive metastore: Databricks recommends that you migrate all data from the legacy Hive metastore to Unity Catalog. However, when this migration is not possible, use the to protect securables registered in the Hive metastore.

• Unity Catalog: To protect securables registered in the Unity Catalog metastore, use the .

  • : To register and protect fully managed PostgreSQL-compatible data objects, use the Databricks Lakebase integration.

Metastore magic

Databricks metastore magic allows you to migrate your data from the Databricks legacy Hive metastore to the Unity Catalog metastore while protecting data and maintaining your current processes in a single Immuta instance.

Databricks metastore magic is for organizations who intend to use the , but must still protect tables in the Hive metastore until they can migrate all of their data to Unity Catalog.

Requirement

Unity Catalog support is enabled in Immuta.

Databricks metastores and Immuta policy enforcement

Databricks has two built-in metastores that contain metadata about your tables, views, and storage credentials:

• Legacy Hive metastore: Created at the workspace level. This metastore contains metadata of the registered securables in that workspace available to query.

• Unity Catalog metastore: Created at the account level and is attached to one or more Databricks workspaces. This metastore contains metadata of the registered securables available to query. All clusters on that workspace use the configured metastore and all workspaces that are configured to use a single metastore share those securables.

Databricks allows you to use the legacy Hive metastore and the Unity Catalog metastore simultaneously. However, Unity Catalog does not support controls on the Hive metastore, so you must attach a Unity Catalog metastore to your workspace and move existing databases and tables to the attached Unity Catalog metastore to use the governance capabilities of Unity Catalog.

Immuta's Databricks Spark integration and Unity Catalog integration enforce access controls on the Hive and Unity Catalog metastores, respectively. However, because these metastores have two distinct security models, users were discouraged from using both in a single Immuta instance before metastore magic; the Databricks Spark integration and Unity Catalog integration were unaware of each other, so using both concurrently caused undefined behavior.

Databricks metastore magic solution

Metastore magic reconciles the distinct security models of the legacy Hive metastore and the Unity Catalog metastore, allowing you to use multiple metastores (specifically, the Hive metastore or alongside Unity Catalog metastores) within a Databricks workspace and single Immuta instance and keep policies enforced on all your tables as you migrate them. The diagram below shows Immuta enforcing policies on registered tables across workspaces.

In clusters A and D, Immuta enforces policies on data sources in each workspace's Hive metastore and in the Unity Catalog metastore shared by those workspaces. In clusters B, C, and E (which don't have Unity Catalog enabled in Databricks), Immuta enforces policies on data sources in the Hive metastores for each workspace.

Enforce policies as you migrate

With metastore magic, the Databricks Spark integration enforces policies only on data in the Hive metastore, while the Unity Catalog integration enforces policies on tables in the Unity Catalog metastore.

To enforce plugin-based policies on Hive metastore tables and Unity Catalog native controls on Unity Catalog metastore tables, enable the Databricks Spark integration and the Databricks Unity Catalog integration. Note that some Immuta policies are not supported in the Databricks Unity Catalog integration. See the for details.

Enforcing policies on Databricks SQL

Databricks SQL cannot run the Databricks Spark plugin to protect tables, so Hive metastore data sources will not be policy enforced in Databricks SQL.

To enforce policies on data sources in Databricks SQL, use to manually lock down Hive metastore data sources and the Databricks Unity Catalog integration to protect tables in the Unity Catalog metastore. Table access control is enabled by default on SQL warehouses, and any Databricks cluster without the Immuta plugin must have table access control enabled.

This page provides a tutorial for enabling the Azure Synapse Analytics integration on the Immuta app settings page. To configure this integration via the Immuta API, see the Configure an Azure Synapse Analytics integration API guide.

For an overview of the integration, see the Azure Synapse Analytics overview documentation.

Requirement

• A running dedicated SQL pool is required.

Prerequisites

If you are using the OAuth authentication method,

• Ensure that Microsoft Entra ID is on the same account as the Azure Synapse Analytics workspace and dedicated SQL pool.

• .

• Select Accounts in this organizational directory only as the account type.

Add an Azure Synapse Analytics integration

1. Click the App Settings icon in the navigation menu.

2. Click the Integrations tab.

3. Click the +Add Integration button and select Azure Synapse Analytics from the dropdown menu.

Select your configuration method

You have two options for configuring your Azure Synapse Analytic environment:

• : Grant Immuta one-time use of credentials to automatically configure your environment and the integration.

• : Run the Immuta script in your Azure Synapse Analytics environment yourself to configure the integration.

Automatic setup

Enter the username and password in the Privileged User Credentials section.

Manual setup

1. Select Manual.

2. Download, fill out the appropriate fields, and run the bootstrap master script and bootstrap script linked in the Setup section. Note: The master script is not required if you're using the OAuth authentication method.

3. Select the authentication method:

Save the configuration

Click Save.

Register data

.

Edit an Azure Synapse Analytics integration

1. Click the App Settings icon in the navigation menu.

2. Navigate to the Integrations tab and click the down arrow next to the Azure Synapse Analytics Integration.

3. Edit the field you want to change. Note any field shadowed is not editable, and the integration must be disabled and re-installed to change it.

Remove an Azure Synapse Analytics integration

1. Click the App Settings icon in the navigation menu.

2. Navigate to the Integrations tab and click the down arrow next to the Azure Synapse Analytics Integration.

3. Click the checkbox to disable the integration.

When the Databricks Spark plugin is running on a Databricks cluster, all Databricks users running jobs or queries are either a privileged user or a non-privileged user:

• Privileged users: Privileged users can effectively read from and write to any table or view in the cluster Metastore, or any file path accessible by the cluster, without restriction. Privileged users are either Databricks workspace admins or users specified in IMMUTA_SPARK_ACL_ALLOWLIST. Any user writing queries or jobs impersonating another user is a non-privileged user, even if they are impersonating a privileged user.\

  Privileged users have effective authority to read from and write to any securable in the cluster metastore or file path, because in almost all cases Databricks clusters running with the Immuta Spark plug-in installed have disabled Hive metastore table access control. However, if Hive metastore table access control is enabled on the cluster, privileged users will have the authority granted to them that is specified by table access control.

• Non-privileged users: Non-privileged users are any users who are not privileged users, and all authorization for non-privileged users is determined by Immuta policies.

Whether a user is a privileged user or a non-privileged user, for a given query or job, is cached once first determined, based on . This caching can be disabled entirely by setting the value of that environment variable to 0.

Mapping Databricks users to Immuta

Usernames in Databricks must match the usernames in the connected Immuta tenant. By default, the Immuta Spark plugin checks the Databricks username against the username within Immuta's internal IAM to determine access. However, you can integrate your existing IAM with Immuta and use that instead of the default internal IAM. Ideally, you should use the same identity manager for Immuta that you use for Databricks. See the for a list of supported identity providers and protocols.

It is possible within Immuta to have multiple users share the same username if they exist within different IAMs. In this case, the cluster can be configured to look up users from a specified IAM. To do this, the value of the must be updated to be the targeted IAM ID configured within the Immuta tenant. The targeted IAM ID can be found on the . Each Databricks cluster can only be mapped to one IAM.

User impersonation

Databricks user impersonation allows a Databricks user to impersonate an Immuta user. With this feature,

• the Immuta user who is being impersonated does not have to have a Databricks account, but they must have an Immuta account.

• the Databricks user who is impersonating an Immuta user does not have to be associated with Immuta. For example, this could be a service account.

When acting under impersonation, the Databricks user loses their privileged access, so they can only access the tables the Immuta user has access to and only perform DDL commands when that user is acting under an allowed circumstance (such as workspaces, scratch paths, or non-Immuta reads/writes).

Use the to enable user impersonation.

Audited queries

Audited queries include an impersonationUser field, which identifies the Databricks user impersonating the Immuta user:

The how-to guides linked on this page illustrate how to use PostgreSQL with Immuta. See the reference guide for information about the PostgreSQL integration.

This page provides an overview of the Amazon Redshift Spectrum integration in Immuta. For a tutorial detailing how to enable this integration, see the .

How the integration works

The Amazon Redshift Spectrum integration is a policy push integration that allows Immuta to apply policies directly on Immuta-created views in Redshift. This allows data analysts to query Redshift views directly instead of going through a proxy and have per-user policies dynamically applied at query time.

The Amazon Redshift Spectrum integration creates views from the tables within the database specified when configured. Then, the user can choose the name for the schema where all the Immuta-generated views will reside. Immuta will also create the schemas

This page describes the Azure Synapse integration, configuration options, and features. See the for a tutorial on enabling the integration and these features through the app settings page.

Feature availability

Requirements

Immuta offers several features to provide security for your users and Databricks clusters and to prove compliance and monitor for anomalies.

Authentication

Once a Databricks securable is registered in Immuta as a data source and you are subscribed to that data source, you must access that data through SQL:

See the sections below for more guidance on accessing data using , , and .

The how-to guides linked on this page illustrate how to integrate Snowflake with Immuta. See the for information about the Snowflake integration.

Requirements

• Snowflake enterprise edition

• Access to a Snowflake account that can create a Snowflake user

1. Navigate to the App Settings page.

2. Scroll to the Global Integrations Settings section.

3. Ensure the Snowflake Table Grants checkbox is checked. It is enabled by default.

Contact your Immuta representative to enable this feature in your Immuta tenant.

Immuta manages access to Snowflake tables by administering Snowflake and on those tables, allowing users to query tables directly in Snowflake while dynamic policies are enforced.

This getting started guide outlines how to integrate your Snowflake account with Immuta.