To delete the host,

1. Click Data in the navigation menu and select Connections.

2. Click the overflow menu in the Actions column and select Delete Connection.

3. Click Delete Host again to confirm.

Immuta offers two integrations for Amazon Redshift:

• : In this integration, Immuta uses to configure the integration and register data objects in a single step. Once data is registered, Immuta can enforce access controls on that data.

• : In this integration, Immuta generates policy-enforced views in your configured Redshift schema for tables registered as Immuta data sources. The integration is configured separately from data source registration.

Apply policy-driven controls and automated workflows to deliver secure, consistent access for users and AI agents.

In this integration, Immuta generates policy-enforced views in your configured Amazon Redshift schema for tables registered as Immuta data sources.

This guide outlines how to integrate Amazon Redshift Spectrum with Immuta.

: This guide describes the design of the integration and policy enforcement.

Once data is registered through the Databricks Unity Catalog connection, you will access your data through your Databricks queries as you normally would. If you are subscribed to the data source, Immuta grants you access to the data in Databricks.

When you submit a query, the Databricks Unity Catalog client submits the SQL query to Databricks, which then processes the query and determines what data your group is allowed to see. Then, Databricks queries the database and returns the query results to the Databricks Unity Catalog client, which then returns policy-enforced data to you.

The diagram below illustrates how Immuta, Databricks, and Databricks Untiy Catalog client interact to access data.

Query Databricks Unity Catalog securables exactly how you normally would in Databricks. Because grants are managed through groups and data policies are managed through UDFs, there is no additional setup required from the data consumers.

Run object sync

1. Navigate to the connection or connection-backed object.

2. Click Run Object Sync.

Edit the connection details

1. Click the Edit button in the Connection details box.

2. Edit the field you want:

   1. SSL Mode

   2. Authentication Method: If you are editing the authentication method to be a different system account user, .

3. Select Validate, and once the validation is successful, select Save.

1. Click the Edit button in the Object Sync settings box.

2. Opt to enable or disable object sync.

3. Opt to update the Start Time and Run Frequency from the dropdown menus.

4. Opt for all data objects to be scanned despite the policy sync setting using the checkbox.

1. Click the Edit button in the Object settings box.

2. Opt to enable or disable Policy Synchronization: When policy sync is enabled, all child objects will be represented by Immuta data sources and policies will be applied to the data sources.

3. Opt to Add Users and Groups as data owners:

Immuta offers several features to provide security for your users and to prove compliance and monitor for anomalies.

Authentication

Registering the connection

The Amazon Redshift connection supports username and password authentication to register a connection. The credentials provided must be for an account with the permissions listed in the Register an Amazon Redshift connection guide. Passwords hashed using MD5 are unsupported for authentication.

Identity providers for user authentication

The built-in Immuta IAM can be used as a complete solution for authentication and user entitlement. However, you can connect your existing identity management provider to Immuta to use that system for authentication and user entitlement instead.

Each of the supported identity providers includes a specific set of configuration options that enable Immuta to communicate with the IAM system and map the users, permissions, groups, and attributes into Immuta.

See the Identity managers guide for a list of supported providers and details.

See the Amazon Redshift integration reference guide for details about mapping user accounts to Immuta.

Auditing and compliance

Immuta provides governance reports so that data owners and governors can monitor users' access to data and detect anomalies in behavior.

Immuta governance reports allow users with the GOVERNANCE Immuta permission to use a natural language builder to instantly create reports that delineate user activity across Immuta. These reports can be based on various entity types, including users, groups, projects, data sources, purposes, policy types, or connection types.

See the page for a list of report types and guidance.

In the Amazon Redshift integration, Immuta administers Amazon Redshift privileges on data registered in Immuta. Then, Immuta users who have been granted access to the data sources can query them.

The sequence diagram below outlines the events that occur when an Immuta user who is subscribed to a data source queries it in Amazon Redshift.

Registering a connection

The Amazon Redshift integration is configured and data is registered through connections, an Immuta feature that allows administrators to register data objects in a technology through a single connection to make data registration more scalable for your organization.

Once the Amazon Redshift connection is registered, you can author subscription and data policies in Immuta to enforce access controls.

See the Amazon Redshift integration reference guide for more details about registering a connection.

Protecting data

After data is registered in Immuta, you can author subscription and data policies in Immuta to enforce access controls.

When a subscription policy is applied to a data source, users who meet the conditions of the policy will be automatically subscribed to the data source. Immuta creates roles for those users (if an Immuta-generated role for them does not already exist) and grants Amazon Redshift privileges to that role. Once a data policy is applied to a data source, Immuta generates a masking or row-level policy in Amazon Redshift and attaches that policy to the data object it applies to.

Consider the following example that illustrates how Immuta enforces a subscription policy that only allows users in the analysts group to access the yellow-table. When this policy is authored and applied to the data source, Immuta issues a SQL statement in Amazon Redshift that grants the SELECT privilege on yellow-table to users registered in Immuta that are part of the analysts group.

In the image above, the user in the analysts group accesses yellow-table, while the user who is a part of the research group is denied access.

See the or the for guidance on applying policies to a data source. See the Amazon Redshift integration page for details about the .

The how-to guides linked on this page illustrate how to integrate Amazon Redshift Spectrum with Immuta. See the for information about the Amazon Redshift Spectrum integration.

In the AWS Lake Formation integration, Immuta orchestrates Lake Formation access controls on data registered in the Glue Data Catalog. Then, Immuta users who have been granted access to the Glue Data Catalog table or view can query it using one of these analytic engines:

• Amazon Athena

• Amazon EMR Spark

To delete the host,

1. Click Data in the navigation menu and select Connections.

2. Click the overflow menu in the Actions column and select Delete Connection.

3. Download the cleanup script from the modal.

Once data is registered through the Amazon Redshift connection, you will access your data through Amazon Redshift as you normally would. If you are subscribed to the data source, Immuta grants you access to the data in Amazon Redshift.

When you submit a query, the SQL client submits the query to Amazon Redshift, which then processes the query and determines what data your role is allowed to see. Then, Amazon Redshift queries the database and returns the query results to the SQL client, which then returns policy-enforced data to you.

The diagram below illustrates how Immuta, Amazon Redshift, and the SQL client interact when a user queries data registered in Immuta.

Because subscription policies are managed through roles, you must be acting under the role Immuta creates for you (immuta_<username>) to get access to the data sources you are subscribed to.

Once data is registered through the Google BigQuery viewless connection, you will access your data through your Google BigQuery client as you normally would. If you are subscribed to the data source, Immuta grants you access to the data in Google BigQuery.

When you submit a query, the Google BigQuery client submits the SQL query to Google BigQuery, which then processes the query and determines what data you are allowed to see. Then, Google BigQuery returns the query results to the client, which then returns policy-enforced data to you.

The diagram below illustrates how Immuta, the Google BigQuery, and the Google BigQuery client interact to access data.

The Oracle integration allows you to register data from Oracle in Immuta.

: This guide describes the design and components of the integration.

The MariaDB integration allows you to register data from MariaDB in Immuta.

: This guide describes the design and components of the integration.

The MySQL integration uses connections to register data from MySQL in Immuta.

: This guide describes the design and components of the integration.

Immuta offers two integrations for Google BigQuery:

• Google BigQuery viewless integration: In this integration, Immuta uses connections to configure the integration and manage access controls on base tables in Google BigQuery.

• Google BigQuery view-based integration: When Google BigQuery data sources are registered after the integration is configured, Immuta creates and manages views for each data source registered. Users can then query policy-protected data in Google BigQuery as secure views within the Immuta-created dataset.

This integration enforces policies on Databricks securables registered in the legacy Hive metastore. Once these securables are registered as Immuta data sources, users can query policy-enforced data on Databricks clusters.

The guides in this section outline how to integrate Databricks Spark with Immuta.

This getting started guide outlines how to integrate Databricks with Immuta.

How-to guides

• Configure a Databricks Spark integration: Configure the Databricks Spark integration.

• Manually update your Databricks cluster: Manually update your cluster to reflect changes in the Immuta init script or cluster policies.

• : Register a Databricks library with Immuta as a trusted library to avoid Immuta security manager errors when using third-party libraries.

• : Raise the caching on-cluster and lower the cache timeouts for the Immuta web service to allow use of project UDFs in Spark jobs.

• : Run R and Scala spark-submit jobs on your Databricks cluster.

• : Access DBFS in Databricks for non-sensitive data.

• : Resolve errors in the Databricks Spark configuration.

• : This guide describes the design and components of the integration.

• : This guide provides an overview of the Immuta features that provide security for your users and Databricks clusters and that allow you to prove compliance and monitor for anomalies.

• : This guide provides an overview of registering Databricks securables and protecting them with Immuta policies.

• : This guide provides an overview of how Databricks users access data registered in Immuta.

In the Databricks Lakebase integration, Immuta administers PostgreSQL privileges on data registered in Immuta. Then, Immuta users who have been granted access to the tables can query them with policies enforced.

The sequence diagram below outlines the events that occur when an Immuta user who is subscribed to a data source queries it in PostgreSQL.

Registering a connection

Databricks Lakebase is configured and data is registered through connections, an Immuta feature that allows administrators to register data objects in a technology through a single connection to make data registration more scalable for your organization.

Once the Databricks Lakebase connection is registered, you can author subscription policies in Immuta to enforce access controls.

See the Databricks Lakebase connection reference guide for more details about registering a connection.

Protecting data

After tables are registered in Immuta, you can author subscription policies in Immuta to enforce access controls.

When a policy is applied to a data source, users who meet the conditions of the policy will be automatically subscribed to the data source. Then, Immuta issues a SQL statement in PostgreSQL that grants the SELECT privilege to users on those tables.

Consider the following example that illustrates how Immuta enforces a subscription policy that only allows users in the analysts group to access to yellow-table. When this policy is authored and applied to the data source, Immuta issues a SQL statement in PostgreSQL that grants the SELECT privilege on yellow-table to users (registered in Immuta) that are part of the analysts group.

In the image above, the user in the analysts group accesses yellow-table, while the user who is a part of the research group is denied access.

See the for guidance on applying a subscription policy to a data source. See the page details about the subscription policy types supported and PostgreSQL privileges Immuta grants on tables registered as Immuta data sources.

If a Databricks cluster needs to be manually updated to reflect changes in the Immuta init script or cluster policies, you can remove and set up your integration again to get the updated policies and init script.

1. Log in to Immuta as an Application Admin.

2. Click the App Settings icon in the navigation menu and scroll to the Integration Settings section.

3. Your existing Databricks Spark integration should be listed here; expand it and note the configuration values. Now select Remove to remove your integration.

4. Click Add Integration and select Databricks Integration to add a new integration.

5. Enter your Databricks Spark integration settings again as configured previously.

6. Click Add Integration to add the integration, and then select Configure Cluster Policies to set up the updated cluster policies and init script.

7. Select the cluster policies you wish to use for your Immuta-enabled Databricks clusters.

8. Automatically push cluster policies and the init script (recommended) or manually update your cluster policies.

   • Automatically push cluster policies

     1. Select Automatically Push Cluster Policies and enter your privileged Databricks access token. This token must have privileges to write to cluster policies.

9. Restart any Databricks clusters using these updated policies for the changes to take effect.

In the Lake Formation integration, Immuta orchestrates Lake Formation access controls on data registered in the Glue Data Catalog. Then, Immuta users who have been granted access to the Glue Data Catalog table or view can query it using one of these analytic engines:

• Amazon Athena

• Amazon EMR Spark

• Amazon Redshift Spectrum

This getting started guide outlines how to integrate AWS Lake Formation with Immuta.

• : This guide describes the design and components of the integration.

• : This guide provides an overview of the Immuta features that provide security for your users and that allow you to prove compliance and monitor for anomalies.

• : This guide provides an overview of how to protect AWS securables with Immuta policies.

• : This guide provides an overview of how AWS users access data registered in Immuta.

When you enable Unity Catalog, Immuta automatically migrates your existing Databricks data sources in Immuta to reference the legacy hive_metastore catalog to account for Unity Catalog's three-level hierarchy. New data sources will reference the Unity Catalog metastore you create and attach to your Databricks workspace.

Because the hive_metastore catalog is not managed by Unity Catalog, existing data sources in the hive_metastore cannot have Unity Catalog access controls applied to them. Data sources in the Hive Metastore must be managed by the Databricks Spark integration.

To allow Immuta to administer Unity Catalog access controls on that data, move the data to Unity Catalog and re-register those tables in Immuta by completing the steps below. If you don't move all data before configuring the integration, metastore magic will protect your existing data sources throughout the migration process.

1. Ensure that all Databricks clusters that have Immuta installed are stopped and the Immuta configuration is removed from the cluster. Immuta-specific cluster configuration is no longer needed with the Databricks Unity Catalog integration.

2. Move all data into Unity Catalog before configuring Immuta with Unity Catalog. Existing data sources will need to be re-created after they are moved to Unity Catalog and the Unity Catalog integration is configured.

3. .

1. In the Databricks Clusters UI, install your third-party library .jar or Maven artifact with Library Source Upload, DBFS, DBFS/S3, or Maven. Alternatively, use the Databricks libraries API.

2. In the Databricks Clusters UI, add the IMMUTA_SPARK_DATABRICKS_TRUSTED_LIB_URIS property as a Spark environment variable and set it to your artifact's URI. To specify more than one trusted library, comma delimit the URIs:

1. Once you've finished making your changes, restart the cluster.

2. Once the cluster is up, execute a command in a notebook. If the trusted library installation is successful, you should see driver log messages like this:

Once data is registered through the AWS Lake Formation connection, you will access your data in one of these AWS analytic engines as you normally would:

• Amazon Athena

• Amazon EMR Spark

• Amazon Redshift Spectrum

If you are subscribed to the data source, Immuta either directly grants you access to the resource through Lake Formation or generates and assigns a Lake Formation tag to that resource to grant you access. See the for details about how policies are enforced.

When you submit a query, the analytic engine requests metadata from Glue Data Catalog, which then queries Lake Formation to determine what data you are allowed to see. Then, the analytic engine requests temporary access from Lake Formation, retrieves the data from S3, and filters the data to return policy-enforced data to you.

The diagram below illustrates how the analytic engine interacts with Glue Data Catalog and Lake Formation to access data.

The Databricks Spark integration is one of two integrations Immuta offers for Databricks.

In this integration, Immuta installs an Immuta-maintained Spark plugin on your Databricks cluster. When a user queries data that has been registered in Immuta as a data source, the plugin injects policy logic into the plan Spark builds so that the results returned to the user only include data that specific user should see.

The reference guides in this section are written for Databricks administrators who are responsible for setting up the integration, securing Databricks clusters, and setting up users:

• Installation and compliance: This guide includes information about what Immuta creates in your Databricks environment and securing your Databricks clusters.

• : Consult this guide for information about customizing the Databricks Spark integration settings.

• : Consult this guide for information about connecting data users and setting up user impersonation.

• : This guide provides a list of Spark environment variables used to configure the integration.

• : This guide describes ephemeral overrides and how to configure them to reduce the risk that a user has overrides set to a cluster (or multiple clusters) that aren't currently up.

Immuta offers several features to provide security for your users and to prove compliance and monitor for anomalies.

Authentication

Registering the connection

The Lake Formation integration supports the following authentication methods to register a connection:

• Access using AWS IAM role (recommended): Immuta will assume this role when interacting with the AWS API. This option allows you to provide Immuta with an IAM role from your AWS account that is granted a trust relationship with Immuta's IAM role. Immuta will assume this IAM role from Immuta's AWS account in order to perform any operations in your AWS account.

• Access using access key and secret access key: These credentials are used temporarily by Immuta to register the connection.

Identity providers for user authentication

The built-in Immuta IAM can be used as a complete solution for authentication and user entitlement. However, you can connect your existing identity management provider to Immuta to use that system for authentication and user entitlement instead.

Each of the supported identity providers includes a specific set of configuration options that enable Immuta to communicate with the IAM system and map the users, permissions, groups, and attributes into Immuta.

See the for a list of supported providers and details.

See the for details about user provisioning and mapping AWS user accounts to Immuta.

Immuta provides governance reports so that data owners and governors can monitor users' access to data and detect anomalies in behavior.

Immuta governance reports allow users with the GOVERNANCE Immuta permission to use a natural language builder to instantly create reports that delineate user activity across Immuta. These reports can be based on various entity types, including users, groups, projects, data sources, purposes, policy types, or connection types.

See the page for a list of report types and guidance.

In this integration, Immuta generates policy-enforced views in a schema in your configured Azure Synapse Analytics Dedicated SQL pool for tables registered as Immuta data sources.

This guide outlines how to integrate Azure Synapse Analytics with Immuta.

How-to guide

Azure Synapse Analytics configuration: Configure the integration in Immuta.

Reference guide

Azure Synapse Analytics integration reference guide: This guide describes the design and components of the integration.

Once data is registered through the Databricks Lakebase connection, you will access your data as you normally would through the PostgreSQL connection. If you are subscribed to the data source, Immuta grants you access to the data through a PostgreSQL role.

When you submit a query through the PostgreSQL connection, the PostgreSQL client submits the SQL query to the PostgreSQL server, which then processes the query and determines what data your role is allowed to see. Then, the PostgreSQL server queries the database and returns the query results to the PostgreSQL client, which then returns policy-enforced data to you.

The diagram below illustrates how Immuta, the PostgreSQL server, and PostgreSQL client interact to access data.

Once data is registered through the PostgreSQL connection, you will access your data through your PostgreSQL client as you normally would. If you are subscribed to the data source, Immuta grants you access to the data in PostgreSQL.

When you submit a query, the PostgreSQL client submits the SQL query to the PostgreSQL server, which then processes the query and determines what data your role is allowed to see. Then, the PostgreSQL server queries the database and returns the query results to the PostgreSQL client, which then returns policy-enforced data to you.

The diagram below illustrates how Immuta, the PostgreSQL server, and PostgreSQL client interact to access data.

This page describes the recommended workflow for setting up Immuta with . Following these steps connects your Amazon Redshift environment, registers users, ingests metadata, and enables automated governance through global policies and access requests.

To complete this workflow, ensure you have:

• Access to an Amazon Redshift account and the permissions required to configure the connection

• An Immuta application administrator who can register the connection

For the recommended implementation described on this page, you should also have:

The user registering the connection must have the permissions below.

• APPLICATION_ADMIN Immuta permission

• The Amazon Redshift user registering the connection must be a superuser or have the following Amazon Redshift privileges:

The how-to guides linked on this page illustrate how to use AWS Lake Formation with Immuta. See the for information about the AWS Lake Formation integration.

The how-to guides linked on this page illustrate how to use Databricks Lakebase with Immuta. See the reference guide for information about the Databricks Lakebase integration.

The how-to guides linked on this page illustrate how to integrate Databricks Unity Catalog with Immuta. See the for information about the Databricks Unity Catalog integration.

Requirements:

• Unity Catalog and attached to a Databricks workspace. Immuta supports configuring a single metastore for each configured integration, and that metastore may be attached to multiple Databricks workspaces.

• Unity Catalog enabled on your Databricks cluster or SQL warehouse. All SQL warehouses have Unity Catalog enabled if your workspace is attached to a Unity Catalog metastore.

The how-to guides linked on this page illustrate how to integrate Azure Synapse Analytics with Immuta. See the for information about the Azure Synapse Analytics integration.

Requirement: A running Dedicated SQL pool

The how-to guides linked on this page illustrate how to integrate Databricks Spark with Immuta.

Requirements

• If Databricks Unity Catalog is enabled in a Databricks workspace, you must use an when you set up the Databricks Spark integration to create an Immuta-enabled cluster.

• If Databricks Unity Catalog is not enabled in your Databricks workspace, you must disable Unity Catalog in your Immuta tenant before proceeding with your configuration of Databricks Spark:

The Databricks Lakebase integration registers data from Databricks Lakebase in Immuta and enforces subscription policies on that data.

This getting started guide outlines how to connect Databricks Lakebase to Immuta.

• : This guide describes the design and components of the integration.

• : This guide provides an overview of the Immuta features that provide security for your users and that allow you to prove compliance and monitor for anomalies.

This page outlines the configuration for setting up project UDFs, which allow users to set their current project in Immuta through Spark. For details about the specific functions available and how to use them, see the .

1. Lower the web service cache timeout in Immuta:

   1. Click the App Settings icon and scroll to the HDFS Cache Settings section.

Immuta offers several features to provide security for your users and to prove compliance and monitor for anomalies.

The Databricks Lakebase connection supports OAuth machine-to-machine (M2M) authentication to register a connection.

The Databricks Lakebase connection authenticates as a Databricks identity and generates an OAuth token. Immuta then uses that token as a password when connecting to PostgreSQL. To enable secure, automated machine-to machine access to the database instance, the connection must obtain an OAuth token using a Databricks service principal. See the for more details.

The built-in Immuta IAM can be used as a complete solution for authentication and user entitlement. However, you can connect your existing identity management provider to Immuta to use that system for authentication and user entitlement instead.

Each of the supported identity providers includes a specific set of configuration options that enable Immuta to communicate with the IAM system and map the users, permissions, groups, and attributes into Immuta.

See the for a list of supported providers and details.

1. Navigate to the connection or connection-backed object.

2. Click Run Object Sync.

1. Click the Edit button in the Connection details box.

Immuta offers several features to provide security for your users and to prove compliance and monitor for anomalies.

The Google BigQuery viewless integration supports the following authentication method to register a connection:

• GCP service account key file: This is the private key file generated when you to use to connect to Google BigQuery. These credentials are used by Immuta to register the connection and maintain state between Immuta and Google BigQuery.

The built-in Immuta IAM can be used as a complete solution for authentication and user entitlement. However, you can connect your existing identity management provider to Immuta to use that system for authentication and user entitlement instead.

Each of the supported identity providers includes a specific set of configuration options that enable Immuta to communicate with the IAM system and map the users, permissions, groups, and attributes into Immuta.

In the PostgreSQL integration, Immuta registers data from PostgreSQL and enforces subscription policies on that data.

This getting started guide outlines how to integrate PostgreSQL with Immuta.

• : This guide describes the design and components of the integration.

• : This guide provides an overview of the Immuta features that provide security for your users and that allow you to prove compliance and monitor for anomalies.

In the PostgreSQL integration, Immuta administers PostgreSQL privileges on data registered in Immuta. Then, Immuta users who have been granted access to the tables can query them with policies enforced.

The sequence diagram below outlines the events that occur when an Immuta user who is subscribed to a data source queries it in PostgreSQL.

PostgreSQL is configured and data is registered through , an Immuta feature that allows administrators to register data objects in a technology through a single connection to make data registration more scalable for your organization.

Once the PostgreSQL connection is registered, you can author subscription policies in Immuta to enforce access controls.

See the for more details about registering a connection.

After tables are registered in Immuta, you can author subscription policies in Immuta to enforce access controls.

When a policy is applied to a data source, users who meet the conditions of the policy will be

This page provides guidelines for troubleshooting issues with the Databricks Spark integration and resolving Py4J security and Databricks trusted library errors.

Debugging the integration

For easier debugging of the Databricks Spark integration, follow the recommendations below.

• Enable cluster init script logging:

  • In the cluster page in Databricks for the target cluster, navigate to Advanced Options -> Logging.

  • Change the Destination from NONE to DBFS and change the path to the desired output location. Note: The unique cluster ID will be added onto the end of the provided path.

• View the Spark UI on your target Databricks cluster: On the cluster page, click the Spark UI tab, which shows the Spark application UI for the cluster. If you encounter issues creating Databricks data sources in Immuta, you can also view the JDBC/ODBC Server portion of the Spark UI to see the result of queries that have been sent from Immuta to Databricks.

The validation and debugging notebook is designed to be used by or under the guidance of an Immuta support professional. Reach out to your Immuta representative for assistance.

1. Import the notebook into a Databricks workspace by navigating to Home in your Databricks instance.

2. Click the arrow next to your name and select Import.

3. Once you have executed commands in the notebook and populated it with debugging information, export the notebook and its contents by opening the File menu, selecting Export, and then selecting DBC Archive.

• Error Message: py4j.security.Py4JSecurityException: Constructor <> is not allowlisted

• Explanation: This error indicates you are being blocked by Py4J security rather than the Immuta Security Manager. Py4J security is strict and generally ends up blocking many ML libraries.

• Solution: Turn off Py4J security on the offending cluster by setting IMMUTA_SPARK_DATABRICKS_PY4J_STRICT_ENABLED=false in the environment variables section. Additionally, because there are limitations to the security mechanisms Immuta employs on-cluster when Py4J security is disabled, ensure that all users on the cluster have the same level of access to data, as users could theoretically see (policy-enforced) data that other users have queried.

Check the driver logs for details. Some possible causes of failure include

• One of the Immuta-configured trusted library URIs does not point to a Databricks library. Check that you have configured the correct URI for the Databricks library.

• For trusted Maven artifacts, the URI must follow this format: maven:/group.id:artifact-id:version.

• Databricks failed to install a library. Any Databricks library installation errors will appear in the Databricks UI under the Libraries tab.

Authentication methods

Registering the connection

The Databricks Unity Catalog integration supports the following authentication methods to register a connection. The credentials provided must be for an account with the permissions listed in the Register a Databricks Unity Catalog connection guide.

• Personal access token (PAT)

• OAuth machine-to-machine (M2M)

Identity providers for user authentication

The built-in Immuta IAM can be used as a complete solution for authentication and user entitlement. However, you can connect your existing identity management provider to Immuta to use that system for authentication and user entitlement instead.

Each of the supported identity providers includes a specific set of configuration options that enable Immuta to communicate with the IAM system and map the users, permissions, groups, and attributes into Immuta.

See the for a list of supported providers and details.

See the for details about user provisioning and mapping user accounts to Immuta.

Immuta provides auditing features and governance reports so that data owners and governors can monitor users' access to data and detect anomalies in behavior.

You can view the information in these audit logs on or export the full audit logs to S3 and ADLS for long-term backup and processing with log data processors and tools. This capability fosters convenient integrations with log monitoring services and data pipelines.

See the for details about these capabilities and how they work with the Databricks Unity Catalog integration.

Immuta uses Databricks tables from the system catalog to understand the queries users make and present them in the query audit logs.

The audit ingest is set when and can be scoped to only ingest specific workspaces if needed. The default ingest frequency is every 24 hours, but this can be configured to a different frequency on the . Additionally, audit ingestion can be manually requested at any time from the Immuta audit page. When manually requested, it will only search for new queries that were created since the last query that had been audited. The job is run in the background, so the new queries will not be immediately available.

See the for details about the contents of the logs and an example of the resulting audit record.

Immuta governance reports allow users with the GOVERNANCE Immuta permission to use a natural language builder to instantly create reports that delineate user activity across Immuta. These reports can be based on various entity types, including users, groups, projects, data sources, purposes, policy types, or connection types.

See the page for a list of report types and guidance.

The how-to guides linked on this page illustrate how to use PostgreSQL with Immuta. See the reference guide for information about the PostgreSQL integration.

When using Delta Lake, the API does not go through the normal Spark execution path. This means that Immuta's Spark extensions do not provide protection for the API. To solve this issue and ensure that Immuta has control over what a user can access, the Delta Lake API is blocked.

Spark SQL can be used instead to give the same functionality with all of Immuta's data protections.

Requests

Below is a table of the Delta Lake API with the Spark SQL that may be used instead.

See here for a complete list of the .

When a table is created in a project workspace, you can merge a different Immuta data source from that workspace into that table you created.

1. .

2. Create a temporary view of the Immuta data source you want to merge into that table.

3. Use that temporary view as the data source you add to the project workspace.

4. Run the following command:

This page outlines how to enable access to DBFS in Databricks for non-sensitive data. Databricks administrators should place the desired configuration in the Spark environment variables.

DBFS FUSE mount

This Databricks feature mounts DBFS to the local cluster filesystem at /dbfs. Although disabled when using process isolation, this feature can safely be enabled if raw, unfiltered data is not stored in DBFS and all users on the cluster are authorized to see each other’s files. When enabled, the entirety of DBFS essentially becomes a scratch path where users can read and write files in /dfbs/path/to/my/file as though they were local files.

For example,

%sh echo "I'm creating a new file in DBFS" > /dbfs/my/newfile.txt

In Python,

%python
with open("/dbfs/my/newfile.txt", "w") as f:
  f.write("I'm creating a new file in DBFS")

Note: This solution also works in R and Scala.

Enable DBFS FUSE mount

To enable the DBFS FUSE mount, set this configuration in the Spark environment variables: IMMUTA_SPARK_DATABRICKS_DBFS_MOUNT_ENABLED=true.

Scratch paths will work when performing arbitrary remote filesystem operations with fs magic or Scala dbutils.fs functions. For example,

To support %fs magic and Scala DBUtils with scratch paths, configure

To use dbutils in Python, set this configuration: immuta.spark.databricks.py4j.strict.enabled=false.

This section illustrates the workflow for getting a file from a remote scratch path, editing it locally with Python, and writing it back to a remote scratch path.

1. Get the file from remote storage:

2. Make a copy if you want to explicitly edit localScratchFile, as it will be read-only and owned by root:

3. Write the new file back to remote storage:

In the context of the Databricks Spark integration, Immuta uses the term ephemeral to describe data sources where the associated compute resources can vary over time. This means that the compute bound to these data sources is not fixed and can change. All Databricks data sources in Immuta are ephemeral.

Ephemeral overrides are specific to each data source and user. They effectively bind cluster compute resources to a data source for a given user. Immuta uses these overrides to determine which cluster compute to use when connecting to Databricks for various maintenance operations.

The operations that use the ephemeral overrides include

• Visibility checks on the data source for a particular user. These checks assess how to apply row-level policies for specific users.

• Stats collection triggered by a specific user.

• Validating a custom WHERE clause policy against a data source. When owners or governors create custom WHERE clause policies, Immuta uses compute resources to validate the SQL in the policy. In this case, the ephemeral overrides for the user writing the policy are used to contact a cluster for SQL validation.

• High cardinality column detection. Certain advanced policy types (e.g., minimization) in Immuta require a high cardinality column, and that column is computed on data source creation. It can be recomputed on demand and, if so, will use the ephemeral overrides for the user requesting computation.

An ephemeral override request can be triggered when a user queries the securable corresponding to a data source in a Databricks cluster with the Spark plug-in configured. The actual triggering of this request depends on the .

Ephemeral overrides can also be set for a data source in the Immuta UI by navigating to a data source page, clicking on the data source actions button, and selecting Ephemeral overrides from the dropdown menu.

Ephemeral override requests made from a cluster for data sources and users where ephemeral overrides were set in the UI will not be successful.

If ephemeral overrides are never set (either through the user interface or the cluster configuration), the system will continue to use the connection details directly associated with the data source, which are set during .

Ephemeral overrides can be problematic in environments that have a dedicated cluster to handle maintenance activities, since ephemeral overrides can cause these operations to execute on a different cluster than the dedicated one.

To reduce the risk that a user has overrides set to a cluster (or multiple clusters) that aren't currently up, complete one of the following actions:

• Direct all clusters' HTTP paths for overrides to a cluster dedicated for metadata queries using the .

• Disable ephemeral overrides completely by setting the to false.

In the Google BigQuery integration, Immuta administers Google BigQuery privileges on data registered in Immuta. Then, Immuta users who have been granted access to the tables can query them with policies enforced.

The sequence diagram below outlines the events that occur when an Immuta user who is subscribed to a data source queries it in Google BigQuery.

Registering a connection

The Google BigQuery viewless integration is configured and data is registered through connections, an Immuta feature that allows administrators to register data objects in a technology through a single connection to make data registration more scalable for your organization.

Once the Google BigQuery viewless connection is registered, you can author subscription policies in Immuta to enforce access controls.

See the Google BigQuery viewless integration reference guide for more details about registering a connection.

Protecting data

After objects are registered in Immuta, you can author subscription policies or row-level data policies in Immuta to enforce access controls.

Subscription policies

When a subscription policy is applied to a data source, users who meet the conditions of the policy will be automatically subscribed to the data source. Then, Immuta updates the allow policy on that resource in Google BigQuery to grant users access to that object. For details about how this access is enforced, see the .

Consider the following example that illustrates how Immuta enforces a subscription policy that only allows users in the analysts group to access the yellow-table. When this policy is authored and applied to the data source, Immuta updates the allow policy on yellow-table to grant access to users (registered in Immuta) that are part of the analysts group.

In the image above, the user in the analysts group accesses yellow-table, while the user who is a part of the research group is denied access.

See the for guidance on applying a subscription policy to a data source. See the for details about the subscription policy types supported and Google BigQuery privileges Immuta grants on objects registered as Immuta data sources.

When a row-level data policy is applied to a table, Immuta creates a row-level access policy in Google BigQuery that filters rows that subscribed users are able to see. For details about how row-access policies are enforced, see the .

In the example below, Immuta creates a Google BigQuery row-level access policy that filters rows based on whether users' groups match values in a specified column:

Only show rows where user is a member of a group that matches the value in columns tagged location.

When the user in the OH group queries yellow-table, which has the location tag applied to the state column, Google BigQuery returns the result with only rows that contain OH in the state column.

See the for guidance on authoring data policies in Immuta and the for the Google BigQuery viewless integration.

Immuta offers several features to provide security for your users and to prove compliance and monitor for anomalies.

Authentication

Registering the connection

The PostgreSQL integration supports the following authentication methods to register a connection:

• Amazon Aurora and Amazon RDS deployments

  • Access using AWS IAM role (recommended): Immuta will assume this IAM role from Immuta's AWS account when interacting with the AWS API to perform any operations in your AWS account. This option allows you to provide Immuta with an IAM role from your AWS account that is granted a trust relationship with Immuta's IAM role.

  • Access using access key and secret access key: These credentials are used temporarily by Immuta to register the connection. The access key ID and secret access key provided must be for an AWS account with the permissions listed in the .

  • Username and password: These credentials are used temporarily by Immuta to register the connection. The credentials provided must be for an account with the permissions listed in the . Passwords hashed using MD5 are unsupported for authentication.

• Neon and PostgreSQL deployments

  • Username and password: These credentials are used temporarily by Immuta to register the connection. The credentials provided must be for an account with the permissions listed in the . Passwords hashed using MD5 are unsupported for authentication.

The built-in Immuta IAM can be used as a complete solution for authentication and user entitlement. However, you can connect your existing identity management provider to Immuta to use that system for authentication and user entitlement instead.

Each of the supported identity providers includes a specific set of configuration options that enable Immuta to communicate with the IAM system and map the users, permissions, groups, and attributes into Immuta.

See the for a list of supported providers and details.

See the for details about user provisioning and mapping user accounts to Immuta.

Immuta provides governance reports so that data owners and governors can monitor users' access to data and detect anomalies in behavior.

Immuta governance reports allow users with the GOVERNANCE Immuta permission to use a natural language builder to instantly create reports that delineate user activity across Immuta. These reports can be based on various entity types, including users, groups, projects, data sources, purposes, policy types, or connection types.

See the page for a list of report types and guidance.

Immuta offers three integrations for Databricks:

• : This integration supports working with database objects registered in Unity Catalog.

• : This connection supports working with Lakebase Postgres

• : This integration supports working with .

The user registering the connection must have the permissions below.

• APPLICATION_ADMIN Immuta permission

• The account credentials you provide to register the connection should be a

This page describes the recommended workflow for setting up Immuta with . Following these steps connects your Google BigQuery environment, registers users, ingests metadata, and enables automated governance through global policies and access requests.

To complete this workflow, ensure you have:

• Access to BigQuery Enterprise or Enterprise Plus edition and the permissions required to configure the connection

• An Immuta application administrator who can register the connection

For the recommended implementation described on this page, you should also have:

When the Databricks Spark plugin is running on a Databricks cluster, all Databricks users running jobs or queries are either a privileged user or a non-privileged user:

• Privileged users: Privileged users can effectively read from and write to any table or view in the cluster Metastore, or any file path accessible by the cluster, without restriction. Privileged users are either or users specified in . Any user writing queries or jobs impersonating another user is a non-privileged user, even if they are impersonating a privileged user.

  Privileged users have effective authority to read from and write to any securable in the cluster metastore or file path, because in almost all cases Databricks clusters running with the Immuta Spark plug-in installed have disabled . However, if Hive metastore table access control is enabled on the cluster, privileged users will have the authority granted to them that is specified by table access control.

The user registering the connection must have the permissions below.

• APPLICATION_ADMIN Immuta permission

• Either of the following Oracle system privileges:

This page provides a tutorial for enabling the Azure Synapse Analytics integration on the Immuta app settings page. To configure this integration via the Immuta API, see the Configure an Azure Synapse Analytics integration API guide.

For an overview of the integration, see the Azure Synapse Analytics overview documentation.

Requirement

• A running dedicated SQL pool is required.

Permissions

The user configuring the integration must have the permissions below.

• APPLICATION_ADMIN Immuta permission

• The account credentials you use to set up the integration must have these Azure Synapse Analytics privileges:

  • Server-level principal login or be a member of the loginmanager database role in the master database

  • CONTROL privilege on an existing database that will store all Immuta-generated views and schemas.

For descriptions and explanations of privileges Immuta needs to enforce policies in Azure Synapse Analytics, see the .

If you are using the OAuth authentication method,

• Ensure that Microsoft Entra ID is on the same account as the Azure Synapse Analytics workspace and dedicated SQL pool.

• .

• Select Accounts in this organizational directory only as the account type.

1. Click the App Settings icon in the navigation menu.

2. Click the Integrations tab.

3. Click the +Add Integration button and select Azure Synapse Analytics from the dropdown menu.

4. Complete the Host,

You have two options for configuring your Azure Synapse Analytic environment:

• : Grant Immuta one-time use of credentials to automatically configure your environment and the integration.

• : Run the Immuta script in your Azure Synapse Analytics environment yourself to configure the integration.

Enter the username and password in the Privileged User Credentials section.

1. Select Manual.

2. Download, fill out the appropriate fields, and run the bootstrap master script and bootstrap script linked in the Setup section as a user with the . Note: The master script is not required if you're using the OAuth authentication method.

3. Select the authentication method:

Click Save.

.

1. Click the App Settings icon in the navigation menu.

2. Navigate to the Integrations tab and click the down arrow next to the Azure Synapse Analytics Integration.

3. Edit the field you want to change. Note any field shadowed is not editable, and the integration must be disabled and re-installed to change it.

4. Use the authentication method and credentials you provided when initially configuring the integration.

1. Click the App Settings icon in the navigation menu.

2. Navigate to the Integrations tab and click the down arrow next to the Azure Synapse Analytics Integration.

3. Click the checkbox to disable the integration.

4. Enter the credentials that were used to initially configure the integration.

This page provides an overview of the Amazon Redshift Spectrum integration in Immuta. For a tutorial detailing how to enable this integration, see the installation guide.

How the integration works

The Amazon Redshift Spectrum integration is a policy push integration that allows Immuta to apply policies directly on Immuta-created views in Redshift. This allows data analysts to query Redshift views directly instead of going through a proxy and have per-user policies dynamically applied at query time.

The Amazon Redshift Spectrum integration creates views from the tables within the database specified when configured. Then, the user can choose the name for the schema where all the Immuta-generated views will reside. Immuta will also create the schemas immuta_system, immuta_functions, and immuta_procedures to contain the tables, views, UDFs, and stored procedures that support the integration. Immuta then creates a system role and gives that system account the following privileges:

• ALL PRIVILEGES ON DATABASE IMMUTA_DB

• ALL PRIVILEGES ON ALL SCHEMAS IN DATABASE IMMUTA_DB

• USAGE ON FUTURE PROCEDURES IN SCHEMA IMMUTA_DB.IMMUTA_PROCEDURES

• USAGE ON LANGUAGE PLPYTHONU

Additionally the PUBLIC role will be granted the following privileges:

• USAGE ON DATABASE IMMUTA_DB

• TEMP ON DATABASE IMMUTA_DB

• USAGE ON SCHEMA IMMUTA_DB.IMMUTA_PROCEDURES

Once the integration is configured, data owners must .

1. An Immuta application administrator, creates an immuta database in Amazon Redshift (that will contain Immuta policy definitions and user entitlements), , and registers Redshift warehouse and databases with Immuta.

2. A data owner registers Redshift tables in Immuta as .

3. A data owner, data governor, or administrator or user in Immuta.

SQL statements are used to create all views, including a join to the secure view: immuta_system.user_profile. This secure view is a select from the immuta_system.profile table (which contains all Immuta users and their current groups, attributes, projects, and a list of valid tables they have access to) with a constraint immuta__userid = current_user() to ensure it only contains the profile row for the current user. The immuta_system.user_profile view is readable by all users, but will only display the data that corresponds to the user executing the query.

The Amazon Redshift Spectrum integration uses webhooks to keep views up-to-date with Immuta data sources. When a data source or policy is created, updated, or disabled, a webhook will be called that will create, modify, or delete the dynamic view. The immuta_system.profile table is updated through webhooks when a user's groups or attributes change, they switch projects, they acknowledge a purpose, or when their data source access is approved or revoked. The profile table can only be read and updated by the Immuta system account.

The definitions for each status and the state of configured data platform integrations is available in the .

The definitions for each status and the state of configured data platform integrations is available in the .

All Redshift cluster types are supported for the Amazon Redshift Spectrum integration, and Immuta's views must exist in the same database as the raw tables. See the for details about setting up this database for Immuta-managed resources.

Immuta supports a single integration with secure views in a single database per cluster.

The Amazon Redshift Spectrum integration supports username and password authentication to configure the integration and create data sources.

Immuta cannot ingest tags from Amazon Redshift Spectrum, but you can connect any of these to work with your integration.

Impersonation allows users to query data as another Immuta user in Amazon Redshift Spectrum. To enable user impersonation, see the .

Users can enable multiple with a single Immuta tenant.

• The host of the data source must match the host of the integration for the view to be created.

• When using multiple Amazon Redshift Spectrum integrations, a user has to have the same user account across all hosts.

• Case sensitivity of database, table, and column identifiers is not supported. The must be set to false (default setting) for your Redshift cluster to configure the integration and register data sources.

Requirements

• Amazon RDS or Amazon Aurora for MySQL

Permissions

The user registering the connection must have the permissions below.

• APPLICATION_ADMIN Immuta permission

• The MySQL user registering the connection must be the root user or have the GRANT OPTION MySQL privilege.

1. Create a new database user in MySQL to serve as the Immuta system account. Immuta will use this system account continuously to crawl the database you register. How you create this user depends on your . Follow the instructions linked below to create this user:

   1. : Follow the to create the database user and assign that user a password.

   2. :

1. In Immuta, click Data and select Connections in the navigation menu.

2. Click the + Add Connection button.

3. Select the MySQL tile.

4. Select your deployment type:

Requirement

• Amazon RDS for MariaDB

Permissions

The user registering the connection must have the permissions below.

• APPLICATION_ADMIN Immuta permission

• The MariaDB user setting up the connection must be the root user or have the GRANT OPTION MariaDB privilege.

1. Create a new database user in MariaDB to serve as the Immuta system account. Immuta will use this system account continuously to crawl the database you register. How you create this user depends on your . Follow the instructions linked below to create this user:

   1. : Follow the to create the database user and assign that user a password.

   2. :

1. In Immuta, click Data and select Connections in the navigation menu.

2. Click the + Add Connection button.

3. Select the MariaDB tile.

4. Select RDS as the deployment method.

Once a Databricks securable is registered in Immuta as a data source and you are subscribed to that data source, you must access that data through SQL:

df = spark.sql("select * from immuta.table")

import org.apache.spark.sql.SparkSession

%sql
select * from immuta

library(SparkR)
df <- SparkR

See the sections below for more guidance on accessing data using Delta Lake, direct file reads in Spark for file paths, and user impersonation.

Delta Lake

When using Delta Lake, the API does not go through the normal Spark execution path. This means that Immuta's Spark extensions do not provide protection for the API. To solve this issue and ensure that Immuta has control over what a user can access, the Delta Lake API is blocked.

Spark SQL can be used instead to give the same functionality with all of Immuta's data protections. See the for a list of corresponding Spark SQL calls to use.

In addition to supporting direct file reads through workspace and scratch paths, Immuta allows direct file reads in Spark for file paths. As a result, users who prefer to interact with their data using file paths or who have existing workflows revolving around file paths can continue to use these workflows without rewriting those queries for Immuta.

When reading from a path in Spark, the Immuta Databricks Spark plugin queries the Immuta Web Service to find Databricks data sources for the current user that are backed by data from the specified path. If found, the query plan maps to the Immuta data source and follows existing code paths for policy enforcement.

Users can read data from individual parquet files in a sub-directory and partitioned data from a sub-directory (or by using a where predicate). Expand the blocks below to view examples of reading data using these methods.

• Direct file reads for Immuta data sources only apply to data sources created from tables, not data sources created from views or queries.

• If more than one data source has been created for a path, Immuta will use the first valid data source it finds. It is therefore not recommended to use this integration when more than one data source has been created for a path.

• In Databricks, multiple input paths are supported as long as they belong to the same data source.

• CSV-backed tables are not currently supported.

User impersonation allows Databricks users to query data as another Immuta user. To impersonate another user, see the .

Immuta offers several features to provide security for your users and Databricks clusters and to prove compliance and monitor for anomalies.

Immuta supports the following authentication methods to configure the Databricks Spark integration and register data sources:

• OAuth machine-to-machine (M2M): Immuta uses the to integrate with , which allows Immuta to authenticate with Databricks using a client secret. Once Databricks verifies the Immuta service principal’s identity using the client secret, Immuta is granted a temporary OAuth token to perform token-based authentication in subsequent requests. When that token expires (after one hour), Immuta requests a new temporary token.

  See the for more details.

The Oracle integration allows you to register data from Oracle in Immuta. Immuta supports Oracle on Amazon RDS.

Oracle is configured and data is registered through , an Immuta feature that allows you to register your data objects through a single connection to make data registration more scalable for your organization. Instead of registering schema and databases individually, you can register them all at once and allow Immuta to monitor your data platform for changes so that data sources are added and removed automatically to reflect the state of data in your data platform.

When the , Immuta ingests and stores connection metadata in the Immuta metadata database. In the example below, the Immuta application administrator connects the database that contains marketing-data, research-data, and cs-data tables. Immuta

The MySQL integration uses connections to register data from MySQL in Immuta. Immuta supports the following deployment methods:

• Amazon Aurora with MySQL

• Amazon RDS with MySQL

MySQL is configured and data is registered through , an Immuta feature that allows you to register your data objects through a single connection to make data registration more scalable for your organization. Instead of registering schema and databases individually, you can register them all at once and allow Immuta to monitor your data platform for changes so that data sources are added and removed automatically to reflect the state of data in your data platform.

The MariaDB integration allows you to register data from MariaDB in Immuta. Immuta supports MariaDB on Amazon RDS.

MariaDB is configured and data is registered through , an Immuta feature that allows you to register your data objects through a single connection to make data registration more scalable for your organization. Instead of registering schema and databases individually, you can register them all at once and allow Immuta to monitor your data platform for changes so that data sources are added and removed automatically to reflect the state of data in your data platform.

When the , Immuta ingests and stores connection metadata in the Immuta metadata database. In the example below, the Immuta application administrator connects the database that contains marketing-data, research-data, and cs-data tables. Immuta

This guide illustrates how to run R and Scala spark-submit jobs on Databricks, including prerequisites and caveats.

Before you can run spark-submit jobs on Databricks, complete the following steps.

1. Initialize the Spark session:

   1. Enter these settings into the R submit script to allow the R script to access Immuta data sources, scratch paths, and workspace tables:

This page describes the recommended workflow for setting up Immuta with . Following these steps connects your Snowflake environment, registers users, ingests metadata, and enables automated governance through global policies and access requests.

To complete this workflow, ensure you have:

• Access to a Snowflake account and the permissions required to configure the connection

• An Immuta application administrator who can register the connection

For the recommended implementation described on this page, you should also have: