The how-to guides linked on this page illustrate how to integrate Redshift with Immuta. See the reference guide for information about the Redshift integration.

Requirement: Redshift cluster with an RA3 node is required for the multi-database integration. For other instance types, you may configure a single-database integration using one of the Redshift Spectrum options.

In this integration, Immuta generates policy-enforced views in your configured Redshift schema for tables registered as Immuta data sources.

Getting started

This guide outlines how to integrate Redshift with Immuta.

How-to guides

• Redshift integration configuration: Configure the integration in Immuta.

• Redshift Spectrum configuration: Configure Redshift Spectrum in Immuta.

Reference guide

Redshift integration reference guide: This guide describes the design and components of the integration.

The how-to guides linked on this page illustrate how to use AWS Lake Formation with Immuta. See the reference guide for information about the AWS Lake Formation connection.

Immuta integrates with your data platforms so you can register your data and effectively manage access controls on that data.

This section includes guidance for connecting your data platform and keeping it synced with Immuta.

Data platforms overview

This reference guide outlines the features, policies, and audit capabilities of each data platform Immuta supports.

Integrations and connections

The guides in these sections include information about how to connect your data platform to Immuta:

• Amazon Redshift

• Amazon S3

• AWS Lake Formation

• Azure Synapse Analytics

• Databricks: This section includes guides for Databricks Spark and Databricks Unity Catalog integrations.

• Google BigQuery

• MariaDB

• Oracle

• PostgreSQL

• Snowflake

• SQL Server

• Starburst (Trino)

• Teradata

Queries Immuta runs in remote platforms

This reference guide outlines the actions and features that trigger Immuta queries in your remote platform that may incur cost.

Connect your data

Immuta integrates with your data platforms so you can register your data and effectively manage access controls on that data. This section includes concept, reference, and how-to guides for registering and managing data sources.

Immuta gives everyone fast, governed access to data with the built-in controls, collaboration workflows, automated provisioning, and continuous monitoring you need to keep risk low and compliance high.

Configure Immuta

Explore Immuta

In the Lake Formation connection, Immuta orchestrates Lake Formation access controls on data registered in the Glue Data Catalog. Then, Immuta users who have been granted access to the Glue Data Catalog table or view can query it using one of these analytic engines:

• Amazon Athena

• Amazon EMR Spark

• Amazon Redshift Spectrum

This getting started guide outlines how to connect AWS Lake Formation with Immuta.

How-to guide

Reference guides

• : This guide describes the design and components of the connection.

• : This guide provides an overview of the Immuta features that provide security for your users and that allow you to prove compliance and monitor for anomalies.

• : This guide provides an overview of how to protect AWS securables with Immuta policies.

• : This guide provides an overview of how AWS users access data registered in Immuta.

This page illustrates how to configure the Redshift integration on the Immuta app settings page. To configure this integration via the Immuta API, see the Integrations API getting started guide.

For instructions on configuring Redshift Spectrum, see the Redshift Spectrum guide.

Requirements

• A Redshift cluster with an RA3 node is required for the multi-database integration. You must use a Redshift RA3 instance type because Immuta requires cross-database views, which are only supported in Redshift RA3 instance types. For other instance types, you may configure a single-database integration using one of the Redshift Spectrum options.

• For automated installations, the credentials provided must be a Superuser or have the ability to create databases and users and modify grants.

• The enable_case_sensitive_identifier parameter must be set to false (default setting) for your Redshift cluster.

Add a Redshift integration

1. Click the App Settings icon in the navigation menu.

2. Click the Integrations tab.

3. Click the +Add Integration button and select Redshift from the dropdown menu.

4. Complete the Host and Port fields.

5. Enter an Immuta Database. This is a new database where all secure schemas and Immuta created views will be stored.

6. Opt to check the Enable Impersonation box and customize the Impersonation Role name as needed. This will allow users to natively impersonate another user.

Select your configuration method

You have two options for configuring your Redshift environment:

• Automatic setup: Grant Immuta one-time use of credentials to automatically configure your Redshift environment and the integration.

• Manual setup: Run the Immuta script in your Redshift environment yourself to configure your environment and the integration.

Automatic setup

1. Select Automatic.

2. Enter an Initial Database from your Redshift integration for Immuta to use to connect.

3. Use the dropdown menu to select your Authentication Method.

   1. Username and Password: Enter the Username and Password of the privileged user.

   2. AWS Access Key: Enter the Database User, Access Key ID, and Secret Key. Opt to enter in the Session Token.

Manual setup

1. Select Manual and download both of the bootstrap scripts.

2. Run the bootstrap script (initial database) in the Redshift initial database.

3. Run the bootstrap script (Immuta database) in the new Immuta Database in Redshift.

4. Choose your authentication method, and enter the information of the newly created account.

Save the configuration

Click Save.

Register data

Register Redshift data in Immuta.

Edit a Redshift integration

1. Click the App Settings icon in the navigation menu.

2. Navigate to the Integrations tab and click the down arrow next to the Redshift Integration.

3. Edit the field you want to change. Note any field shadowed is not editable, and the integration must be disabled and re-installed to change it.

4. Enter Username and Password.

5. Click Save.

Remove a Redshift integration

1. Click the App Settings icon in the navigation menu.

2. Navigate to the Integrations tab and click the down arrow next to the Redshift Integration.

3. Click the checkbox to disable the integration.

4. Enter the username and password that were used to initially configure the integration.

5. Click Save.

Allow Immuta to create secure views of your external tables through one of these methods:

• Configure the integration with an existing database that contains the external tables: Instead of creating an immuta database that manages all schemas and views created when Redshift data is registered in Immuta, the integration adds the Immuta-managed schemas and views to an existing database in Redshift

• Configure the integration by creating a new immuta database and re-create all of your external tables in that database.

For an overview of the integration, see the Redshift overview documentation.

Requirements

• A Redshift cluster with an AWS row-level security patch applied. Contact Immuta for guidance.

• An AWS IAM role for Redshift that is associated with your Redshift cluster.

• The enable_case_sensitive_identifier parameter must be set to false (default setting) for your Redshift cluster.

• The Redshift role used to run the Immuta bootstrap script must have the following privileges when configuring the integration to

  • Use an existing database:

    • ALL PRIVILEGES ON DATABASE for the database you configure the integration with, as you must manage grants on that database.

    • CREATE USER

    • GRANT TEMP ON DATABASE

  • Create a new database:

    • CREATE DATABASE

    • CREATE USER

    • GRANT TEMP ON DATABASE

    • REVOKE ALL PRIVILEGES ON DATABASE

• A Redshift database that contains an external schema and external tables.

Use an existing database

1. Click the App Settings icon in the navigation menu.

2. Click the Integrations tab.

3. Click the +Add Integration button and select Redshift from the dropdown menu.

4. Complete the Host and Port fields.

5. Enter the name of the database you created the external schema in as the Immuta Database. This database will store all secure schemas and Immuta-created views.

6. Opt to check the Enable Impersonation box and customize the Impersonation Role name as needed. This will allow users to natively impersonate another user.

7. Select Manual and download both of the bootstrap scripts from the Setup section. The specified role used to run the bootstrap needs to have the following privileges:

   • ALL PRIVILEGES ON DATABASE for the database you configure the integration with, as you must manage grants on that database.

   • CREATE USER

   • GRANT TEMP ON DATABASE

8. Run the bootstrap script (Immuta database) in the Redshift database that contains the external schema.

9. Choose your authentication method, and enter the credentials from the bootstrap script for the Immuta_System_Account.

10. Click Save.

Register data

Register Redshift data in Immuta.

Create a new Immuta database

1. Click the App Settings icon in the navigation menu.

2. Click the Integrations tab.

3. Click the +Add Integration button and select Redshift from the dropdown menu.

4. Complete the Host and Port fields.

5. Enter an Immuta Database. This is a new database where all secure schemas and Immuta created views will be stored.

6. Opt to check the Enable Impersonation box and customize the Impersonation Role name as needed. This will allow users to natively impersonate another user.

7. Select Manual and download both of the bootstrap scripts from the Setup section. The specified role used to run the bootstrap needs to have the following privileges:

   • ALL PRIVILEGES ON DATABASE for the database you configure the integration with, as you must manage grants on that database.

   • CREATE DATABASE

   • CREATE USER

   • GRANT TEMP ON DATABASE

8. Run the bootstrap script (initial database) in the Redshift initial database.

9. Run the bootstrap script (Immuta database) in the new Immuta Database in Redshift.

10. Choose your authentication method, and enter the credentials from the bootstrap script for the Immuta_System_Account.

11. Click Save.

Then, add your external tables to the Immuta database.

Register data

Register Redshift data in Immuta.

In this integration, Immuta generates policy-enforced views in a schema in your configured Azure Synapse Analytics Dedicated SQL pool for tables registered as Immuta data sources.

Getting started

This guide outlines how to integrate Azure Synapse Analytics with Immuta.

How-to guide

Azure Synapse Analytics configuration: Configure the integration in Immuta.

Reference guide

Azure Synapse Analytics integration reference guide: This guide describes the design and components of the integration.

This integration allows you to manage and access data in your Databricks account across all of your workspaces. With Immuta’s Databricks Unity Catalog integration, you can write your policies in Immuta and have them enforced automatically by Databricks across data in your Unity Catalog metastore.

Getting started

This getting started guide outlines how to integrate Databricks Unity Catalog with Immuta.

How-to guides

• Databricks Unity Catalog configuration: Configure the Databricks Unity Catalog integration.

• Migrate to Databricks Unity Catalog: Migrate from the legacy Databricks Spark integrations to the Databricks Unity Catalog integration.

Reference guide

Databricks Unity Catalog integration reference guide: This guide describes the design and components of the integration.

To migrate from the private preview version of table grants (available before September 2022) to the GA version, complete the steps below.

1. Navigate to the App Settings page.

2. Scroll to the Global Integrations Settings section.

3. Uncheck the Snowflake Table Grants checkbox to disable the feature.

4. Click Save. Wait for about 1 minute per 1000 users. This gives time for Immuta to drop all the previously created user roles.

5. Use the Enable Snowflake table grants tutorial to re-enable the feature.

This page provides guidelines for troubleshooting issues with the Databricks Spark integration and resolving Py4J security and Databricks trusted library errors.

Debugging the integration

For easier debugging of the Databricks Spark integration, follow the recommendations below.

• Enable cluster init script logging:

  • In the cluster page in Databricks for the target cluster, navigate to Advanced Options -> Logging.

  • Change the Destination from NONE to DBFS and change the path to the desired output location. Note: The unique cluster ID will be added onto the end of the provided path.

• View the Spark UI on your target Databricks cluster: On the cluster page, click the Spark UI tab, which shows the Spark application UI for the cluster. If you encounter issues creating Databricks data sources in Immuta, you can also view the JDBC/ODBC Server portion of the Spark UI to see the result of queries that have been sent from Immuta to Databricks.

Using the validation and debugging notebook

The validation and debugging notebook is designed to be used by or under the guidance of an Immuta support professional. Reach out to your Immuta representative for assistance.

1. Import the notebook into a Databricks workspace by navigating to Home in your Databricks instance.

2. Click the arrow next to your name and select Import.

3. Once you have executed commands in the notebook and populated it with debugging information, export the notebook and its contents by opening the File menu, selecting Export, and then selecting DBC Archive.

Py4J security error

• Error Message: py4j.security.Py4JSecurityException: Constructor <> is not allowlisted

• Explanation: This error indicates you are being blocked by Py4J security rather than the Immuta Security Manager. Py4J security is strict and generally ends up blocking many ML libraries.

• Solution: Turn off Py4J security on the offending cluster by setting IMMUTA_SPARK_DATABRICKS_PY4J_STRICT_ENABLED=false in the environment variables section. Additionally, because there are limitations to the security mechanisms Immuta employs on-cluster when Py4J security is disabled, ensure that all users on the cluster have the same level of access to data, as users could theoretically see (policy-enforced) data that other users have queried.

Databricks trusted library errors

Check the driver logs for details. Some possible causes of failure include

• One of the Immuta-configured trusted library URIs does not point to a Databricks library. Check that you have configured the correct URI for the Databricks library.

• For trusted Maven artifacts, the URI must follow this format: maven:/group.id:artifact-id:version.

• Databricks failed to install a library. Any Databricks library installation errors will appear in the Databricks UI under the Libraries tab.

This page describes the Redshift integration, configuration options, and features. For a tutorial to enable this integration, see the .

Feature Availability

Prerequisites

• For automated installations, the credentials provided must be a Superuser or have the ability to create databases and users and modify grants.

Supported Features

• Redshift datashares

• Redshift Serverless

• For configuration and data source registration instructions, see the .

Authentication Methods

The Redshift integration supports the following authentication methods to configure the integration and create data sources:

• Username and Password: Users can authenticate with their Redshift username and password.

• AWS Access Key: Users can authenticate with an .

Tag Ingestion

Immuta cannot ingest tags from Redshift, but you can connect any of these to work with your integration.

User Impersonation

Impersonation allows users to query data as another Immuta user in Redshift. To enable user impersonation, see the page.

Multiple Integrations

Users can enable multiple with a single Immuta tenant.

Redshift Limitations

• The host of the data source must match the host of the connection for the view to be created.

• When using multiple Redshift integrations, a user has to have the same user account across all hosts.

• Case sensitivity of database, table, and column identifiers is not supported. The must be set to false (default setting) for your Redshift cluster to configure the integration and register data sources.

Python UDF Specific Limitations

For most policy types in Redshift, Immuta uses SQL clauses to implement enforcement logic; however Immuta uses Python UDFs in the Redshift integration to implement the following masking policies:

• Masking using a regular expression

• Reversible masking

• Format-preserving masking

• Randomized response

The number of Python UDFs that can run concurrently per Redshift cluster is limited to one-fourth of the total concurrency level for the cluster. For example, if the Redshift cluster is configured with a concurrency of 15, a maximum of three Python UDFs can run concurrently. After the limit is reached, Python UDFs are queued for execution within workload management queues.

The SVL_QUERY_QUEUE_INFO view in Redshift, which is visible to a Redshift superuser, summarizes details for queries that spent time in a workload management (WLM) query queue. Queries must be completed in order to appear as results in the SVL_QUERY_QUEUE_INFO view.

If you find that queries on Immuta-built views are spending time in the workload management (WLM) query queue, you should either edit your Redshift cluster configuration to increase concurrency, or use fewer of the masking policies which leverage Python UDFs. For more information on increasing concurrency, see the Redshift docs on implementing .

Immuta offers several features to provide security for your users and to prove compliance and monitor for anomalies.

Security

Data processing and encryption

See the and the guides for more information about transmission of policy decision data, encryption of data in transit and at rest, and encryption key management.

Authentication

Registering the connection

The Lake Formation connection supports the following authentication methods to register a connection:

• Access using AWS IAM role (recommended): Immuta will assume this role when interacting with the AWS API. This option allows you to provide Immuta with an IAM role from your AWS account that is granted a trust relationship with Immuta's IAM role. Immuta will assume this IAM role from Immuta's AWS account in order to perform any operations in your AWS account.

• Access using access key and secret access key: These credentials are used temporarily by Immuta to register the connection.

Identity providers for user authentication

The built-in Immuta IAM can be used as a complete solution for authentication and user entitlement. However, you can connect your existing identity management provider to Immuta to use that system for authentication and user entitlement instead.

Each of the supported identity providers includes a specific set of configuration options that enable Immuta to communicate with the IAM system and map the users, permissions, groups, and attributes into Immuta.

See the for a list of supported providers and details.

See the for details about user provisioning and mapping AWS user accounts to Immuta.

Auditing and compliance

Immuta provides governance reports so that data owners and governors can monitor users' access to data and detect anomalies in behavior.

Immuta governance reports allow users with the GOVERNANCE Immuta permission to use a natural language builder to instantly create reports that delineate user activity across Immuta. These reports can be based on various entity types, including users, groups, projects, data sources, purposes, policy types, or connection types.

See the page for a list of report types and guidance.

Once data is registered through the AWS Lake Formation connection, you will access your data in one of these AWS analytic engines as you normally would:

• Amazon Athena

• Amazon EMR Spark

• Amazon Redshift Spectrum

If you are subscribed to the data source, Immuta either directly grants you access to the resource through Lake Formation or generates and assigns a Lake Formation tag to that resource to grant you access. See the for details about how policies are enforced.

When you submit a query, the analytic engine requests metadata from Glue Data Catalog, which then queries Lake Formation to determine what data you are allowed to see. Then, the analytic engine requests temporary access from Lake Formation, retrieves the data from S3, and filters the data to returns policy-enforced data to you.

The diagram below illustrates how the analytic engine interacts with Glue Data Catalog and Lake Formation to access data.

The Databricks Spark integration is one of two integrations Immuta offers for Databricks.

In this integration, Immuta installs an Immuta-maintained Spark plugin on your Databricks cluster. When a user queries data that has been registered in Immuta as a data source, the plugin injects policy logic into the plan Spark builds so that the results returned to the user only include data that specific user should see.

The reference guides in this section are written for Databricks administrators who are responsible for setting up the integration, securing Databricks clusters, and setting up users:

• : This guide includes information about what Immuta creates in your Databricks environment and securing your Databricks clusters.

• : Consult this guide for information about customizing the Databricks Spark integration settings.

• : Consult this guide for information about connecting data users and setting up user impersonation.

• : This guide provides a list of Spark environment variables used to configure the integration.

• : This guide describes ephemeral overrides and how to configure them to reduce the risk that a user has overrides set to a cluster (or multiple clusters) that aren't currently up.

This page outlines the configuration for setting up project UDFs, which allow users to set their current project in Immuta through Spark. For details about the specific functions available and how to use them, see the .

1. Lower the web service cache timeout in Immuta:

   1. Click the App Settings icon and scroll to the HDFS Cache Settings section.

   2. Lower the Cache TTL of HDFS user names (ms) to 0.

   3. Click Save.

2. Raise the cache timeout on your Databricks cluster: In the Spark environment variables section, set the IMMUTA_CURRENT_PROJECT_CACHE_TIMEOUT_SECONDS and IMMUTA_PROJECT_CACHE_TIMEOUT_SECONDS to high values (like 10000).

   Note: These caches will be invalidated on cluster when a user calls immuta.set_current_project, so they can effectively be cached permanently on cluster to avoid periodically reaching out to the web service.

In the context of the Databricks Spark integration, Immuta uses the term ephemeral to describe data sources where the associated compute resources can vary over time. This means that the compute bound to these data sources is not fixed and can change. All Databricks data sources in Immuta are ephemeral.

Ephemeral overrides are specific to each data source and user. They effectively bind cluster compute resources to a data source for a given user. Immuta uses these overrides to determine which cluster compute to use when connecting to Databricks for various maintenance operations.

The operations that use the ephemeral overrides include

• Visibility checks on the data source for a particular user. These checks assess how to apply row-level policies for specific users.

• Stats collection triggered by a specific user.

• Validating a custom WHERE clause policy against a data source. When owners or governors create custom WHERE clause policies, Immuta uses compute resources to validate the SQL in the policy. In this case, the ephemeral overrides for the user writing the policy are used to contact a cluster for SQL validation.

• High cardinality column detection. Certain advanced policy types (e.g., minimization) in Immuta require a high cardinality column, and that column is computed on data source creation. It can be recomputed on demand and, if so, will use the ephemeral overrides for the user requesting computation.

Triggering an ephemeral override request

An ephemeral override request can be triggered when a user queries the securable corresponding to a data source in a Databricks cluster with the Spark plug-in configured. The actual triggering of this request depends on the .

Ephemeral overrides can also be set for a data source in the Immuta UI by navigating to a data source page, clicking on the data source actions button, and selecting Ephemeral overrides from the dropdown menu.

Ephemeral override requests made from a cluster for data sources and users where ephemeral overrides were set in the UI will not be successful.

If ephemeral overrides are never set (either through the user interface or the cluster configuration), the system will continue to use the connection details directly associated with the data source, which are set during .

Configuring overrides in Immuta-enabled clusters

Ephemeral overrides can be problematic in environments that have a dedicated cluster to handle maintenance activities, since ephemeral overrides can cause these operations to execute on a different cluster than the dedicated one.

To reduce the risk that a user has overrides set to a cluster (or multiple clusters) that aren't currently up, complete one of the following actions:

• Direct all clusters' HTTP paths for overrides to a cluster dedicated for metadata queries using the .

• Disable ephemeral overrides completely by setting the to false.

The MariaDB connection registers data from MariaDB in Immuta.

How-to guide

Reference guide

: This guide describes the design and components of the connection.

The PostgreSQL connection registers data from PostgreSQL in Immuta and enforces subscription policies on that data.

This getting started guide outlines how to connect PostgreSQL to Immuta.

How-to guide

Reference guides

• : This guide describes the design and components of the connection.

• : This guide provides an overview of the Immuta features that provide security for your users and that allow you to prove compliance and monitor for anomalies.

• : This guide provides an overview of how to protect securables with Immuta policies.

• : This guide provides an overview of how PostgreSQL users access data registered in Immuta.

Immuta offers several features to provide security for your users and to prove compliance and monitor for anomalies.

Security

Data processing and encryption

See the and the guides for information about transmission of policy decision data, encryption of data in transit and at rest, and encryption key management.

Authentication

Registering the connection

The PostgreSQL connection supports the following authentication methods to register a connection:

• Amazon Aurora and Amazon RDS deployments

  • Access using AWS IAM role (recommended): Immuta will assume this IAM role from Immuta's AWS account when interacting with the AWS API to perform any operations in your AWS account. This option allows you to provide Immuta with an IAM role from your AWS account that is granted a trust relationship with Immuta's IAM role.

  • Access using access key and secret access key: These credentials are used temporarily by Immuta to register the connection. The access key ID and secret access key provided must be for an AWS account with the permissions listed in the .

• Neon and PostgreSQL deployments

  • Username and password: These credentials are used temporarily by Immuta to register the connection. The credentials provided must be for an account with the permissions listed in the .

Identity providers for user authentication

The built-in Immuta IAM can be used as a complete solution for authentication and user entitlement. However, you can connect your existing identity management provider to Immuta to use that system for authentication and user entitlement instead.

Each of the supported identity providers includes a specific set of configuration options that enable Immuta to communicate with the IAM system and map the users, permissions, groups, and attributes into Immuta.

See the for a list of supported providers and details.

See the for details about user provisioning and mapping user accounts to Immuta.

Auditing and compliance

Immuta provides governance reports so that data owners and governors can monitor users' access to data and detect anomalies in behavior.

Immuta governance reports allow users with the GOVERNANCE Immuta permission to use a natural language builder to instantly create reports that delineate user activity across Immuta. These reports can be based on various entity types, including users, groups, projects, data sources, purposes, policy types, or connection types.

See the page for a list of report types and guidance.

Once data is registered through the PostgreSQL connection, you will access your data through your PostgreSQL client as you normally would. If you are subscribed to the data source, Immuta grants you access to the data in PostgreSQL.

When you submit a query, the PostgreSQL client submits the SQL query to the PostgreSQL server, which then processes the query and determines what data your role is allowed to see. Then, the PostgreSQL server queries the database and returns the query results to the PostgreSQL client, which then returns policy-enforced data to you.

The diagram below illustrates how Immuta, the PostgreSQL server, and PostgreSQL client interact to access data.

When you enable Unity Catalog, Immuta automatically migrates your existing Databricks data sources in Immuta to reference the legacy hive_metastore catalog to account for Unity Catalog's . New data sources will reference the Unity Catalog metastore you create and attach to your Databricks workspace.

Because the hive_metastore catalog is not managed by Unity Catalog, existing data sources in the hive_metastore cannot have Unity Catalog access controls applied to them. .

To allow Immuta to administer Unity Catalog access controls on that data, move the data to Unity Catalog and re-register those tables in Immuta by completing the steps below. If you don't move all data before configuring the integration, will protect your existing data sources throughout the migration process.

1. Ensure that all Databricks clusters that have Immuta installed are stopped and the Immuta configuration is removed from the cluster. Immuta-specific cluster configuration is no longer needed with the Databricks Unity Catalog integration.

2. Move all data into Unity Catalog before configuring Immuta with Unity Catalog. Existing data sources will need to be re-created after they are moved to Unity Catalog and the Unity Catalog integration is configured.

3. .

1. Click the App Settings icon in the navigation menu and scroll to the Global Integration Settings section.

2. Click the Enable Snowflake Low Row Access Policy Mode checkbox to enable the feature.

3. Confirm to allow Immuta to automatically disable impersonation for the Snowflake integration. If you do not confirm, you will not be able to enable Snowflake low row access policy mode.

4. Click Save.

Configure your Snowflake integration

If you already have a configured, you don't need to reconfigure your integration. Your Snowflake policies automatically refresh when you enable Snowflake low row access policy mode.

1. . Note that you will not be able to enable project workspaces or user impersonation with Snowflake low row access policy mode enabled.

2. Click Save and Confirm your changes.

The Oracle connection registers data from Oracle in Immuta.

How-to guide

Reference guide

: This guide describes the design and components of the connection.

The how-to guides linked on this page illustrate how to integrate Azure Synapse Analytics with Immuta. See the reference guide for information about the Azure Synapse Analytics integration.

Requirement: A running Dedicated SQL pool

The how-to guides linked on this page illustrate how to integrate Databricks Spark with Immuta.

Requirements

• If Databricks Unity Catalog is enabled in a Databricks workspace, you must use an Immuta cluster policy when you set up the Databricks Spark integration to create an Immuta-enabled cluster.

• If Databricks Unity Catalog is not enabled in your Databricks workspace, you must disable Unity Catalog in your Immuta tenant before proceeding with your configuration of Databricks Spark:

  1. Navigate to the App Settings page and click Integration Settings.

  2. Uncheck the Enable Unity Catalog checkbox.

  3. Click Save.

When the Databricks Spark plugin is running on a Databricks cluster, all Databricks users running jobs or queries are either a privileged user or a non-privileged user:

• Privileged users: Privileged users can effectively read from and write to any table or view in the cluster Metastore, or any file path accessible by the cluster, without restriction. Privileged users are either Databricks workspace admins or users specified in IMMUTA_SPARK_ACL_ALLOWLIST. Any user writing queries or jobs impersonating another user is a non-privileged user, even if they are impersonating a privileged user.

  Privileged users have effective authority to read from and write to any securable in the cluster metastore or file path, because in almost all cases Databricks clusters running with the Immuta Spark plug-in installed have disabled Hive metastore table access control. However, if Hive metastore table access control is enabled on the cluster, privileged users will have the authority granted to them that is specified by table access control.

• Non-privileged users: Non-privileged users are any users who are not privileged users, and all authorization for non-privileged users is determined by Immuta policies.

Whether a user is a privileged user or a non-privileged user, for a given query or job, is cached once first determined, based on IMMUTA_SPARK_ACL_PRIVILEGED_TIMEOUT_SECONDS environment variable. This caching can be disabled entirely by setting the value of that environment variable to 0.

Mapping Databricks users to Immuta

Usernames in Databricks must match the usernames in the connected Immuta tenant. By default, the Immuta Spark plugin checks the Databricks username against the username within Immuta's internal IAM to determine access. However, you can integrate your existing IAM with Immuta and use that instead of the default internal IAM. Ideally, you should use the same identity manager for Immuta that you use for Databricks. See the Immuta support matrix page for a list of supported identity providers and protocols.

It is possible within Immuta to have multiple users share the same username if they exist within different IAMs. In this case, the cluster can be configured to look up users from a specified IAM. To do this, the value of theIMMTUA_USER_MAPPING_IAMID Spark environment variable must be updated to be the targeted IAM ID configured within the Immuta tenant. The targeted IAM ID can be found on the App settings page. Each Databricks cluster can only be mapped to one IAM.

User impersonation

Databricks user impersonation allows a Databricks user to impersonate an Immuta user. With this feature,

• the Immuta user who is being impersonated does not have to have a Databricks account, but they must have an Immuta account.

• the Databricks user who is impersonating an Immuta user does not have to be associated with Immuta. For example, this could be a service account.

When acting under impersonation, the Databricks user loses their privileged access, so they can only access the tables the Immuta user has access to and only perform DDL commands when that user is acting under an allowed circumstance (such as workspaces, scratch paths, or non-Immuta reads/writes).

Use the IMMUTA_SPARK_DATABRICKS_ALLOWED_IMPERSONATION_USERS Spark environment variable to enable user impersonation.

When using Delta Lake, the API does not go through the normal Spark execution path. This means that Immuta's Spark extensions do not provide protection for the API. To solve this issue and ensure that Immuta has control over what a user can access, the Delta Lake API is blocked.

Spark SQL can be used instead to give the same functionality with all of Immuta's data protections.

Requests

Below is a table of the Delta Lake API with the Spark SQL that may be used instead.

See here for a complete list of the Delta SQL Commands.

Merging tables in workspaces

When a table is created in a project workspace, you can merge a different Immuta data source from that workspace into that table you created.

1. Create a table in the project workspace.

2. Create a temporary view of the Immuta data source you want to merge into that table.

3. Use that temporary view as the data source you add to the project workspace.

4. Run the following command:

   Copy

   MERGE INTO delta_native.target_native as target
   USING immuta_temp_view_data_source as source
   ON target.dr_number = source.dr_number
   WHEN MATCHED THEN
   UPDATE SET target.date_reported = source.date_reported

Immuta manages access to Snowflake tables by administering Snowflake row access policies and column masking policies on those tables, allowing users to query tables directly in Snowflake while dynamic policies are enforced.

Getting started

This getting started guide outlines how to integrate your Snowflake account with Immuta.

How-to guides

• Configure a Snowflake integration: Configure the Snowflake integration.

• Edit or remove an existing integration: Manage integration settings or delete your existing Snowflake integration.

• Integration settings:

  • Enable Snowflake table grants: Enable Snowflake table grants and configure the Snowflake role prefix.

  • Use Snowflake data sharing with Immuta: Use Snowflake data sharing with table grants or project workspaces.

  • Snowflake low row access policy mode: Enable Snowflake low row access policy mode.

  • Snowflake lineage tag propagation: Configure your Snowflake integration to automatically apply tags added to a Snowflake table to its descendant data source columns in Immuta.

Reference guides

• Snowflake integration reference guide: This reference guide describes the design and features of the Snowflake integration.

• Snowflake table grants: Snowflake table grants simplifies the management of privileges in Snowflake when using Immuta. Instead of manually granting users access to tables registered in Immuta, you allow Immuta to manage privileges on your Snowflake tables and views according to subscription policies. This guide describes the components of Snowflake table grants and how they are used in Immuta's Snowflake integration.

• Snowflake data sharing with Immuta: Organizations can share the policy-protected data of their Snowflake database with other Snowflake accounts with Immuta policies enforced in real time. This guide describes the components of using Immuta with Snowflake data shares.

• Snowflake low row access policy mode: The Snowflake low row access policy mode improves query performance in Immuta's Snowflake integration. To do so, this mode decreases the number of Snowflake row access policies Immuta creates and uses table grants to manage user access. This guide describes the design and requirements of this mode.

• Snowflake lineage tag propagation: Snowflake column lineage specifies how data flows from source tables or columns to the target tables in write operations. When Snowflake lineage tag propagation is enabled in Immuta, Immuta automatically applies tags added to a Snowflake table to its descendant data source columns in Immuta so you can build policies using those tags to restrict access to sensitive data.

• Warehouse sizing recommendations: Adjust the size and scale of clusters for your warehouse to manage workloads so that you can use Snowflake compute resources the most cost effectively.

Explanatory guide

Phased Snowflake onboarding: A phased onboarding approach to configuring the Snowflake integration ensures that your users will not be immediately affected by changes as you add data sources and policies. This guide describes the settings and requirements for implementing this phased approach.

Immuta is compatible with Snowflake Secure Data Sharing. Using both Immuta and Snowflake, organizations can share the policy-protected data of their Snowflake database with other Snowflake accounts with Immuta policies enforced in real time.

Prerequisites:

• Snowflake integration enabled

• Snowflake tables registered in Immuta as data sources

Create Immuta Policies to Protect the Data

Required Permission: Immuta: GOVERNANCE

Build Immuta data policies to fit your organization's compliance requirements.

It's important to understand that subscription policies are not relevant to Snowflake data shares, because the act of sharing the data is the subscription policy. Data policies can be enforced on the consuming account from the producer account on a share following these instructions.

Register the Snowflake Data Consumer with Immuta

Required Permission: Immuta: USER_ADMIN

To register the Snowflake data consumer in Immuta,

1. Create a new Immuta user.

2. Update the Immuta user's Snowflake username to match the account ID for the data consumer. This value is the output on the data consumer side when SELECT CURRENT_ACCOUNT() is run in Snowflake.

3. Give the Immuta user the appropriate attributes and groups for your organization's policies.

4. Subscribe the Immuta user to the data sources.

Create the Snowflake Data Share

Required Permission: Snowflake ACCOUNTADMIN

To share the policy-protected data source,

1. Create a Snowflake Data Share of the Snowflake table that has been registered in Immuta.

2. Grant reference usage on the Immuta database to the share you created:

   Copy

   GRANT REFERENCE_USAGE ON DATABASE "<Immuta database of the provider account>" TO SHARE "<DATA_SHARE>";

   Replace the content in angle brackets above with the name of your Immuta database and Snowflake data share.

Prerequisites

This upgrade step is necessary if you meet both of the following criteria:

• You have the Snowflake low row access policy mode enabled in private preview.

• You have user impersonation enabled.

If you do not meet this criteria, follow the instructions on the configuration guide.

Upgrade to Snowflake low row access policy mode

To upgrade to the generally available version of the feature, disable your Snowflake integration on the app settings page and then re-enable it.

The SQL Server connection registers data from SQL Server in Immuta.

How-to guide

Register a SQL Server connection

Reference guide

SQL Server connection reference guide: This guide describes the design and components of the connection.

While you're onboarding Snowflake data sources and designing policies, you don't want to disrupt your Snowflake users' existing workflows. Instead, you want to gradually onboard Immuta through a series of successive changes that will not impact your existing Snowflake users.

A phased onboarding approach to configuring the Snowflake integration ensures that your users will not be immediately affected by changes as you add data sources and configure policies.

Several features allow you to gradually onboard data sources and policies in Immuta:

• No subscription policies are applied at the time of data registration: No policy is applied at registration time unless an existing global policy applies to it; the table is registered in Immuta and waits for a policy to be applied, if ever.

  There are several benefits to this design:

  • All existing roles maintain access to the data and registration of the table or view with Immuta has zero impact on your data platform.

  • It gives you time to configure tags on the Immuta registered tables and views, either manually or through automatic means, such as identification, or an external catalog integration to include Snowflake tags.

  • It gives you time to assess and validate the sensitive data tags that were applied.

  • You can build only row and column controls with Immuta and let your existing roles manage table access instead of using Immuta subscription policies for table access.

• Snowflake table grants coupled with Snowflake low row access policy mode: With these features enabled, Immuta manages access to tables (subscription policies) through GRANTs. This works by assigning each user their own unique role created by Immuta and all table access is managed using that single role.

  Without these two features enabled, Immuta uses a Snowflake row access policy (RAP) to manage table access. A RAP only allows users to access rows in the table if they were explicitly granted access through an Immuta subscription policy; otherwise, the user sees no rows. This behavior means all existing Snowflake roles lose access to the table contents until explicitly granted access through Immuta subscription policies. Essentially, roles outside of Immuta don't control access anymore.

  By using table grants and the low row access policy mode, users and roles outside Immuta continue to work.

  There are two benefits to this approach:

  • All pre-existing Snowflake roles retain access to the data until you explicitly revoke access (outside Immuta).

  • It provides a way to test that Immuta GRANTs are working without impacting production workloads.

Requirements

The following configuration is required for phased Snowflake onboarding:

• Impersonation is disabled

• Project workspaces are disabled

If either of these capabilities is necessary for your use case, you cannot do phased Snowflake onboarding as described below.

Next

See the Getting started page for step-by-step guidance to implement phased Snowflake onboarding.

In this integration, Immuta policies are translated into Starburst rules and permissions and applied directly to tables within users’ existing catalogs.

Getting started

This guide outlines how to integrate Starburst with Immuta.

How-to guides

• Starburst (Trino) integration configuration guide: Configure the integration in Immuta.

• Map read and write access policies to Starburst (Trino) privileges: Configure how read and write access subscription policies translate to Starburst (Trino) privileges and apply to Starburst (Trino) data sources.

Reference guide

Starburst (Trino) integration reference guide: This guide describes the design and components of the integration.

If a Databricks cluster needs to be manually updated to reflect changes in the Immuta init script or cluster policies, you can remove and set up your integration again to get the updated policies and init script.

1. Log in to Immuta as an Application Admin.

2. Click the App Settings icon in the navigation menu and scroll to the Integration Settings section.

3. Your existing Databricks Spark integration should be listed here; expand it and note the configuration values. Now select Remove to remove your integration.

4. Click Add Integration and select Databricks Integration to add a new integration.

5. Enter your Databricks Spark integration settings again as configured previously.

6. Click Add Integration to add the integration, and then select Configure Cluster Policies to set up the updated cluster policies and init script.

7. Select the cluster policies you wish to use for your Immuta-enabled Databricks clusters.

8. Automatically push cluster policies and the init script (recommended) or manually update your cluster policies.

   • Automatically push cluster policies

     1. Select Automatically Push Cluster Policies and enter your privileged Databricks access token. This token must have privileges to write to cluster policies.

     2. Select Apply Policies to push the cluster policies and init script again.

     3. Click Save and Confirm to deploy your changes.

   • Manually update cluster policies

     1. Download the init script and the new cluster policies to your local computer.

     2. Click Save and Confirm to save your changes in Immuta.

     3. Log in to your Databricks workspace with your administrator account to set up cluster policies.

     4. Get the path you will upload the init script (immuta_cluster_init_script_proxy.sh) to by opening one of the cluster policy .json files and looking for the defaultValue of the field init_scripts.0.dbfs.destination. This should be a DBFS path in the form of dbfs:/immuta-plugin/hostname/immuta_cluster_init_script_proxy.sh.

     5. Click Data in the left pane to upload your init script to DBFS to the path you found above.

     6. To find your existing cluster policies you need to update, click Compute in the left pane and select the Cluster policies tab.

     7. Edit each of these cluster policies that were configured before and overwrite the contents of the JSON with the new cluster policy JSON you downloaded.

9. Restart any Databricks clusters using these updated policies for the changes to take effect.

This page provides an overview of the Redshift integration in Immuta. For a tutorial detailing how to enable this integration, see the installation guide.

Overview

Redshift is a policy push integration that allows Immuta to apply policies directly in Redshift. This allows data analysts to query Redshift views directly instead of going through a proxy and have per-user policies dynamically applied at query time.

Architecture

The Redshift integration will create views from the tables within the database specified when configured. Then, the user can choose the name for the schema where all the Immuta generated views will reside. Immuta will also create the schemas immuta_system, immuta_functions, and immuta_procedures to contain the tables, views, UDFs, and stored procedures that support the integration. Immuta then creates a system role and gives that system account the following privileges:

• ALL PRIVILEGES ON DATABASE IMMUTA_DB

• ALL PRIVILEGES ON ALL SCHEMAS IN DATABASE IMMUTA_DB

• USAGE ON FUTURE PROCEDURES IN SCHEMA IMMUTA_DB.IMMUTA_PROCEDURES

• USAGE ON LANGUAGE PLPYTHONU

Additionally the PUBLIC role will be granted the following privileges:

• USAGE ON DATABASE IMMUTA_DB

• TEMP ON DATABASE IMMUTA_DB

• USAGE ON SCHEMA IMMUTA_DB.IMMUTA_PROCEDURES

• USAGE ON SCHEMA IMMUTA_DB.IMMUTA_FUNCTIONS

• USAGE ON FUTURE FUNCTIONS IN SCHEMA IMMUTA_DB.IMMUTA_FUNCTIONS

• USAGE ON SCHEMA IMMUTA_DB.IMMUTA_SYSTEM

• SELECT ON TABLES TO public

Integration type

Immuta supports the Redshift integration as both multi-database and single-database integrations. In either integration type, Immuta supports a single integration with secure views in a single database per cluster.

Multi-database integration

If using a multi-database integration, you must use a Redshift cluster with an RA3 node because Immuta requires cross-database views.

Single-database integration

If using a single-database integration, all Redshift cluster types are supported. However, because cross-database queries are not supported in any types other than RA3, Immuta's views must exist in the same database as the raw tables. Consequently, the steps for configuring the integration for Redshift clusters with external tables differ slightly from those that don't have external tables. Allow Immuta to create secure views of your external tables through one of these methods:

• configure the integration with an existing database that contains the external tables: Instead of creating an immuta database that manages all schemas and views created when Redshift data is registered in Immuta, the integration adds the Immuta-managed schemas and views to an existing database in Redshift.

• configure the integration by creating a new immuta database and re-create all of your external tables in that database.

Policy enforcement

SQL statements are used to create all views, including a join to the secure view: immuta_system.user_profile. This secure view is a select from the immuta_system.profile table (which contains all Immuta users and their current groups, attributes, projects, and a list of valid tables they have access to) with a constraint immuta__userid = current_user() to ensure it only contains the profile row for the current user. The immuta_system.user_profile view is readable by all users, but will only display the data that corresponds to the user executing the query.

The Redshift integration uses webhooks to keep views up-to-date with Immuta data sources. When a data source or policy is created, updated, or disabled, a webhook will be called that will create, modify, or delete the dynamic view. The immuta_system.profile table is updated through webhooks when a user's groups or attributes change, they switch projects, they acknowledge a purpose, or when their data source access is approved or revoked. The profile table can only be read and updated by the Immuta system account.

Integration health status

The status of the integration is visible on the integrations tab of the Immuta application settings page. If errors occur in the integration, a banner will appear in the Immuta UI with guidance for remediating the error.

The definitions for each status and the state of configured data platform integrations is available in the response schema of the integrations API. However, the UI consolidates these error statuses and provides detail in the error messages.

Data flow

1. An Immuta Application Administrator configures the Redshift integration and registers Redshift warehouse and databases with Immuta.

2. Immuta creates a database inside the configured Redshift ecosystem that contains Immuta policy definitions and user entitlements.

3. A Data Owner registers Redshift tables in Immuta as data sources.

4. A Data Owner, Data Governor, or Administrator creates or changes a policy or user in Immuta.

5. Data source metadata, tags, user metadata, and policy definitions are stored in Immuta's Metadata Database.

6. The Immuta Web Service calls a stored procedure that modifies the user entitlements or policies.

7. A Redshift user who is subscribed to the data source in Immuta queries the corresponding table directly in Redshift through the immuta database and sees policy-enforced data.

Redshift Spectrum

Redshift Spectrum (Redshift external tables) allows Redshift users to query external data directly from files on Amazon S3. Because cross-database queries are not supported in Redshift Spectrum, Immuta's views must exist in the same database as the raw tables. Consequently, the steps for configuring the integration for Redshift clusters with external tables differ slightly from those that don't have external tables. Allow Immuta to create secure views of your external tables through one of these methods:

• configure the integration with an existing database that contains the external tables: Instead of creating an immuta database that manages all schemas and views created when Redshift data is registered in Immuta, the integration adds the Immuta-managed schemas and views to an existing database in Redshift

• configure the integration by creating a new immuta database and re-create all of your external tables in that database.

Once the integration is configured, Data Owners must register Redshift Spectrum data sources using the Immuta CLI or V2 API.

Immuta offers two integrations for Databricks:

• Databricks Unity Catalog integration: This integration supports working with database objects registered in Unity Catalog.

• Databricks Spark integration: This integration supports working with database objects registered in the legacy Hive metastore.

Which integration should you use?

To determine which integration you should use, consider which metastore you use:

• Legacy Hive metastore: Databricks recommends that you migrate all data from the legacy Hive metastore to Unity Catalog. However, when this migration is not possible, use the Databricks Spark integration to protect securables registered in the Hive metastore.

• Unity Catalog: To protect securables registered in the Unity Catalog metastore, use the Databricks Unity Catalog integration.

• Legacy Hive metastore and Unity Catalog: If you need to work with database objects registered in both the legacy Hive metastore and in Unity Catalog, metastore magic allows you to use both integrations.

Metastore magic

Databricks metastore magic allows you to migrate your data from the Databricks legacy Hive metastore to the Unity Catalog metastore while protecting data and maintaining your current processes in a single Immuta instance.

Databricks metastore magic is for organizations who intend to use the Databricks Unity Catalog integration, but must still protect tables in the Hive metastore until they can migrate all of their data to Unity Catalog.

Requirement

Unity Catalog support is enabled in Immuta.

Databricks metastores and Immuta policy enforcement

Databricks has two built-in metastores that contain metadata about your tables, views, and storage credentials:

• Legacy Hive metastore: Created at the workspace level. This metastore contains metadata of the registered securables in that workspace available to query.

• Unity Catalog metastore: Created at the account level and is attached to one or more Databricks workspaces. This metastore contains metadata of the registered securables available to query. All clusters on that workspace use the configured metastore and all workspaces that are configured to use a single metastore share those securables.

Databricks allows you to use the legacy Hive metastore and the Unity Catalog metastore simultaneously. However, Unity Catalog does not support controls on the Hive metastore, so you must attach a Unity Catalog metastore to your workspace and move existing databases and tables to the attached Unity Catalog metastore to use the governance capabilities of Unity Catalog.

Immuta's Databricks Spark integration and Unity Catalog integration enforce access controls on the Hive and Unity Catalog metastores, respectively. However, because these metastores have two distinct security models, users were discouraged from using both in a single Immuta instance before metastore magic; the Databricks Spark integration and Unity Catalog integration were unaware of each other, so using both concurrently caused undefined behavior.

Databricks metastore magic solution

Metastore magic reconciles the distinct security models of the legacy Hive metastore and the Unity Catalog metastore, allowing you to use multiple metastores (specifically, the Hive metastore or AWS Glue Data Catalog alongside Unity Catalog metastores) within a Databricks workspace and single Immuta instance and keep policies enforced on all your tables as you migrate them. The diagram below shows Immuta enforcing policies on registered tables across workspaces.

In clusters A and D, Immuta enforces policies on data sources in each workspace's Hive metastore and in the Unity Catalog metastore shared by those workspaces. In clusters B, C, and E (which don't have Unity Catalog enabled in Databricks), Immuta enforces policies on data sources in the Hive metastores for each workspace.

Enforce policies as you migrate

With metastore magic, the Databricks Spark integration enforces policies only on data in the Hive metastore, while the Unity Catalog integration enforces policies on tables in the Unity Catalog metastore. The table below illustrates this policy enforcement.

To enforce plugin-based policies on Hive metastore tables and Unity Catalog native controls on Unity Catalog metastore tables, enable the Databricks Spark integration and the Databricks Unity Catalog integration. Note that some Immuta policies are not supported in the Databricks Unity Catalog integration. See the Databricks Unity Catalog integration reference guide for details.

Enforcing policies on Databricks SQL

Databricks SQL cannot run the Databricks Spark plugin to protect tables, so Hive metastore data sources will not be policy enforced in Databricks SQL.

To enforce policies on data sources in Databricks SQL, use Hive metastore table access controls to manually lock down Hive metastore data sources and the Databricks Unity Catalog integration to protect tables in the Unity Catalog metastore. Table access control is enabled by default on SQL warehouses, and any Databricks cluster without the Immuta plugin must have table access control enabled.

1. In the Databricks Clusters UI, install your third-party library .jar or Maven artifact with Library Source Upload, DBFS, DBFS/S3, or Maven. Alternatively, use the Databricks libraries API.

2. In the Databricks Clusters UI, add the IMMUTA_SPARK_DATABRICKS_TRUSTED_LIB_URIS property as a Spark environment variable and set it to your artifact's URI:

IMMUTA_SPARK_DATABRICKS_TRUSTED_LIB_URIS=maven:/com.github.immuta.hadoop.immuta-spark-third-party-maven-lib-test:2020-11-17-144644

IMMUTA_SPARK_DATABRICKS_TRUSTED_LIB_URIS=dbfs:/immuta/bstabile/jars/immuta-spark-third-party-lib-test.jar

1. Once you've finished making your changes, restart the cluster.

2. Once the cluster is up, execute a command in a notebook. If the trusted library installation is successful, you should see driver log messages like this:

   Copy

   TrustedLibraryUtils: Successfully found all configured Immuta configured trusted libraries in Databricks.
   TrustedLibraryUtils: Wrote trusted libs file to [/databricks/immuta/immutaTrustedLibs.json]: true.
   TrustedLibraryUtils: Added trusted libs file with 1 entries to spark context.
   TrustedLibraryUtils: Trusted library installation complete.

In the PostgreSQL connection, Immuta administers PostgreSQL privileges on data registered in Immuta. Then, Immuta users who have been granted access to the tables can query them with policies enforced.

The sequence diagram below outlines the events that occur when an Immuta user who is subscribed to a data source queries it in PostgreSQL.

Registering a connection

PostgreSQL is configured and data is registered through connections, an Immuta feature that allows administrators to register data objects in a technology through a single connection to make data registration more scalable for your organization.

Once the PostgreSQL connection is registered, you can author subscription policies in Immuta to enforce access controls.

See the PostgreSQL connection reference guide for more details about registering a connection.

Protecting data

After tables are registered in Immuta, you can author subscription policies in Immuta to enforce access controls.

When a policy is applied to a data source, users who meet the conditions of the policy will be automatically subscribed to the data source. Then, Immuta issues a SQL statement in PostgreSQL that grants the SELECT privilege to users on those tables.

Consider the following example that illustrates how Immuta enforces a subscription policy that only allows users in the analysts group to access the yellow-table. When this policy is authored and applied to the data source, Immuta issues a SQL statement in PostgreSQL that grants the SELECT privilege on yellow-table to users (registered in Immuta) that are part of the analysts group.

In the image above, the user in the analysts group accesses yellow-table , while the user who is a part of the research group is denied access. See the Author a subscription policy page for guidance on applying a subscription policy to a data source. See the Subscription policy access types page for details about the subscription policy types supported and PostgreSQL privileges Immuta grants on tables registered as Immuta data sources.

In the AWS Lake Formation connection, Immuta orchestrates Lake Formation access controls on data registered in the Glue Data Catalog. Then, Immuta users who have been granted access to the Glue Data Catalog table or view can query it using one of these analytic engines:

• Amazon Athena

• Amazon EMR Spark

• Amazon Redshift Spectrum

The sequence diagram below outlines the events that occur when an Immuta user who is subscribed to a data source submits a query in their AWS analytic engine.

See the for more details about Lake Formation access controls.

Registering a connection

AWS Lake Formation is configured and data is registered through , an Immuta feature that allows administrators to register data objects in a technology through a single connection to make data registration more scalable for your organization.

Once the Lake Formation connection is registered, you can author policies in Immuta to orchestrate Lake Formation access controls.

See the for more details about registering a connection.

Protecting data

After Glue Data Catalog views and tables are registered in Immuta, you can author subscription policies in Immuta to orchestrate Lake Formation access controls. Once a subscription policy is applied, users can be subscribed to data sources in the following ways:

• Manually subscribed: If a data owner , Immuta issues a grant directly to the data object in AWS.

• Automatically subscribed through policy logic: When a policy is applied to a data source, users who meet the conditions of the policy will be automatically subscribed to the data source. Then, Immuta generates a Lake Formation tag and applies it to the corresponding data object in AWS and grants subscribers access to that tag, which in turn grants them access to the data. See the for details about this process.

Consider the following example that illustrates how Immuta enforces a subscription policy that only allows users in the analysts group to access the yellow-table. When this policy is authored and applied to the data source, Immuta generates a Lake Formation (LF) tag that is applied to the Glue Data Catalog yellow-table and permissions on that tag are granted to all AWS users (registered in Immuta) that are part of the analysts group.

In the image above, the user in the analysts group accesses yellow-table , while the user who is a part of the research group is denied access.

See the for guidance on applying a subscription policy to a data source. See the page for details about the subscription policy types supported and permissions Immuta grants on securables registered as Immuta data sources.

This page provides a tutorial for enabling the Azure Synapse Analytics integration on the Immuta app settings page. To configure this integration via the Immuta API, see the .

For an overview of the integration, see the documentation.

Requirement

A running Dedicated SQL pool is required.

Add an Azure Synapse Analytics integration

1. Click the App Settings icon in the navigation menu.

2. Click the Integrations tab.

3. Click the +Add Integration button and select Azure Synapse Analytics from the dropdown menu.

4. Complete the Host, Port, Immuta Database, and Immuta Schema fields.

5. Opt to check the Enable Impersonation box and customize the Impersonation Role name as needed. This will allow users to natively impersonate another user.

6. Opt to update the User Profile Delimiters. This will be necessary if any of the provided symbols are used in user profile information.

Select your configuration method

You have two options for configuring your Azure Synapse Analytic environment:

• : Grant Immuta one-time use of credentials to automatically configure your environment and the integration.

• : Run the Immuta script in your Azure Synapse Analytics environment yourself to configure the integration.

Automatic setup

Enter the username and password in the Privileged User Credentials section.

Manual setup

1. Select Manual.

2. Download, fill out the appropriate fields, and run the bootstrap master script and bootstrap script linked in the Setup section.

3. Enter the username and password in the Immuta System Account Credentials section. The username and password provided must be the credentials that were set in the bootstrap master script when you created the user.

Save the configuration

Click Save.

Register data

.

Edit an Azure Synapse Analytics integration

1. Click the App Settings icon in the navigation menu.

2. Navigate to the Integrations tab and click the down arrow next to the Azure Synapse Analytics Integration.

3. Edit the field you want to change. Note any field shadowed is not editable, and the integration must be disabled and re-installed to change it.

4. Enter Username and Password.

5. Click Save.

Remove an Azure Synapse Analytics integration

1. Click the App Settings icon in the navigation menu.

2. Navigate to the Integrations tab and click the down arrow next to the Azure Synapse Analytics Integration.

3. Click the checkbox to disable the integration.

4. Enter the username and password that were used to initially configure the integration.

5. Click Save.

This page describes the Azure Synapse Analytics integration, through which Immuta applies policies directly in Azure Synapse Analytics. For a tutorial on configuring Azure Synapse Analytics see the .

Overview

The Azure Synapse Analytics is a policy push integration that allows Immuta to apply policies directly in Azure Synapse Analytics Dedicated SQL pools without the need for users to go through a proxy. Instead, users can work within their existing Synapse Studio and have per-user policies dynamically applied at query time.

Architecture

This integration works on a per-Dedicated-SQL-pool basis: all of Immuta's policy definitions and user entitlements data need to be in the same pool as the target data sources because Dedicated SQL pools do not support cross-database joins. Immuta creates schemas inside the configured Dedicated SQL pool that contain policy-enforced views that users query.

When the integration is configured, the Application Admin specifies the

• Immuta Database: This is the pre-existing database Immuta uses. Immuta will create views from the tables contained in this database, and all schemas and views created by Immuta will exist in this database, such as the schemas immuta_system, immuta_functions, and the immuta_procedures that contain the tables, views, UDFs, and stored procedures that support the integration.

• Immuta Schema: The schema that Immuta manages. All views generated by Immuta for tables registered as data sources will be created in this schema.

• User Profile Delimiters: Since Azure Synapse Analytics dedicated SQL pools do not support array or hash objects, certain user access information is stored as delimited strings; the Application Admin can modify those delimiters to ensure they do not conflict with possible characters in strings.

For a tutorial on configuring the integration see the .

Data source naming convention

Synapse data sources are represented as views and are under one schema instead of a database, so their view names are a combination of their schema and table name, separated by an underscore.

For example, with a configuration that uses IMMUTA as the schema in the database dedicated_pool, the view name for the data source dedicated_pool.tpc.case would be dedicated_pool.IMMUTA.tpc_case.

You can see the view information on the data source details page under Connection Information.

Policy enforcement

This integration uses webhooks to keep views up-to-date with the corresponding Immuta data sources. When a data source or policy is created, updated, or disabled, a webhook is called that creates, modifies, or deletes the dynamic view in the Immuta schema. Note that only standard views are available because Azure Synapse Analytics Dedicated SQL pools do not support secure views.

Integration health status

The status of the integration is visible on the integrations tab of the Immuta application settings page. If errors occur in the integration, a banner will appear in the Immuta UI with guidance for remediating the error.

The definitions for each status and the state of configured data platform integrations is available in the . However, the UI consolidates these error statuses and provides detail in the error messages.

Data flow

1. An Immuta Application Administrator , registering their initial Synapse Dedicated SQL pool with Immuta.

2. Immuta creates Immuta schemas inside the configured Synapse Dedicated SQL pool.

3. A Data Owner in Immuta as data sources. A Data Owner, Data Governor, or Administrator or in Immuta.

4. Data source metadata, tags, user metadata, and policy definitions are stored in Immuta's Metadata Database.

5. The Immuta Web Service calls a stored procedure that modifies the user entitlements or policies and updates data source view definitions as necessary.

6. A Synapse user who is subscribed to the data source in Immuta in Synapse and sees policy-enforced data.

This page outlines how to enable access to DBFS in Databricks for non-sensitive data. Databricks administrators should place the desired configuration in the Spark environment variables.

DBFS FUSE mount

This Databricks feature mounts DBFS to the local cluster filesystem at /dbfs. Although disabled when using process isolation, this feature can safely be enabled if raw, unfiltered data is not stored in DBFS and all users on the cluster are authorized to see each other’s files. When enabled, the entirety of DBFS essentially becomes a scratch path where users can read and write files in /dfbs/path/to/my/file as though they were local files.

For example,

In Python,

Note: This solution also works in R and Scala.

Enable DBFS FUSE mount

To enable the DBFS FUSE mount, set this configuration in the Spark environment variables: IMMUTA_SPARK_DATABRICKS_DBFS_MOUNT_ENABLED=true.

Scala DBUtils (and %fs magic) with scratch paths

Scratch paths will work when performing arbitrary remote filesystem operations with fs magic or Scala dbutils.fs functions. For example,

Configure Scala DBUtils (and %fs magic) with scratch paths

To support %fs magic and Scala DBUtils with scratch paths, configure

Configure DBUtils in Python

To use dbutils in Python, set this configuration: immuta.spark.databricks.py4j.strict.enabled=false.

Example workflow

This section illustrates the workflow for getting a file from a remote scratch path, editing it locally with Python, and writing it back to a remote scratch path.

1. Get the file from remote storage:

2. Make a copy if you want to explicitly edit localScratchFile, as it will be read-only and owned by root:

3. Write the new file back to remote storage:

Contact your Immuta representative to enable this feature in your Immuta tenant.

Configure the Snowflake integration

1. Navigate to the App Setting page and click the Integration tab.

2. Click +Add Integration and select Snowflake from the dropdown menu.

3. Complete the Host, Port, and Default Warehouse fields.

4. Enable Query Audit.

5. Enable Lineage and complete the following fields:

   • Ingest Batch Sizes: This setting configures the number of rows Immuta ingests per batch when streaming Access History data from your Snowflake instance.

   • Table Filter: This filter determines which tables Immuta will ingest lineage for. Enter a regular expression that excludes / from the beginning and end to filter tables. Without this filter, Immuta will attempt to ingest lineage for every table on your Snowflake instance.

   • Tag Filter: This filter determines which tags to propagate using lineage. Enter a regular expression that excludes / from the beginning and end to filter tags. Without this filter, Immuta will ingest lineage for every tag on your Snowflake instance.

6. Select Manual or Automatic Setup and

Trigger Snowflake lineage sync job

Prerequisite

.

Trigger the lineage job

The Snowflake lineage sync endpoint triggers the lineage ingestion job that allows Immuta to propagate Snowflake tags added through lineage to Immuta data sources.

1. Copy the example and replace the Immuta URL and API key with your own.

2. Change the payload attribute values to your own, where

   • tableFilter (string): This regular expression determines which tables Immuta will ingest lineage for. Enter a regular expression that excludes / from the beginning and end to filter tables. Without this filter, Immuta will attempt to ingest lineage for every table on your Snowflake instance.

   • batchSize (integer): This parameter configures the number of rows Immuta ingests per batch when streaming Access History data from your Snowflake instance. Minimum 1.

   • lastTimestamp (string): Setting this parameter will only return lineage events later than the value provided. Use a format like 2022-06-29T09:47:06.012-07:00.

Next steps

Once the sync job is complete, you can complete the following steps:

Immuta is compatible with . Using both Immuta and Snowflake, organizations can share the policy-protected data of their Snowflake database with other Snowflake accounts with Immuta policies enforced in real time. This integration gives data consumers a live connection to the data and relieves data providers of the legal and technical burden of creating static data copies that leave their Snowflake environment.

Requirements:

• Snowflake Enterprise Edition or higher

• Immuta's

Configuration

This method requires that the data consumer account is registered as an Immuta user with the Snowflake user name equal to the consuming account.

At that point, the user that represents the account being shared with can have the appropriate attributes and groups assigned to them, relevant to the data policies that need to be enforced. Once that user has access to the share in the consuming account (not managed by Immuta), they can query the share with the data policies from the producer account enforced because Immuta is treating that account as if they are a single user in Immuta.

For a tutorial on this workflow, see the .

Benefits

Using Immuta with Snowflake Data Sharing allows the sharer to

• Only need limited knowledge of the context or goals of the existing policies in place: Because the sharer is not editing or creating policies to share their data, they only need a limited knowledge of how the policies work. Their main responsibility is making sure they properly represent the attributes of the data consumer (the account being shared to).

• Leave policies untouched.

The Snowflake low row access policy mode improves query performance in Immuta's Snowflake integration by decreasing the number of Immuta creates and by using table grants to manage user access.

Immuta manages access to Snowflake tables by administering and on those tables, allowing users to query them directly in Snowflake while policies are enforced.

Without Snowflake low row access policy mode enabled, row access policies are created and administered by Immuta in the following scenarios:

• are disabled and a subscription policy that does not automatically subscribe everyone to the data source is applied. Immuta administers Snowflake row access policies to filter out all the rows to restrict access to the entire table when the user doesn't have privileges to query it. However, if table grants are disabled and a subscription policy is applied that grants everyone access to the data source automatically, Immuta does not create a row access policy in Snowflake. See the for details about these policy types.

• is applied to a data source. A row access policy filters out all the rows of the table if users aren't acting under the purpose specified in the policy when they query the table.

• is applied to a data source. A row access policy filters out rows querying users don't have access to.

• is enabled. A row access policy is created for every Snowflake table registered in Immuta.

Reducing row access policies

Snowflake low row access policy mode is enabled by default to reduce the number of row access policies Immuta creates and improve query performance. Snowflake low row access policy mode requires

• .

• user impersonation to be disabled. User impersonation diminishes the performance of interactive queries because of the number of row access policies Immuta creates when it's enabled.

Requirements

Project-scoped purpose exceptions for Snowflake with low row access policy mode enabled

Project-scoped purpose exceptions for Snowflake integrations allow you to apply to Snowflake data sources in a project. As a result, users can only access that data when they are working within that specific project.

Masked joins for Snowflake with low row access policy mode enabled

This feature allows masked columns to be joined across data sources that belong to the same project. When data sources do not belong to a project, Immuta uses a unique salt per data source for hashing to prevent masked values from being joined. (See the guide for an explanation of that behavior.) However, once you add Snowflake data sources to a project and enable masked joins, Immuta uses a consistent salt across all the data sources in that project to allow the join.

For more information about masked joins and enabling them for your project, see the of documentation.

Limitations and considerations

• Project workspaces are not compatible with this feature.

• Impersonation is not supported when the Snowflake low row access policy mode is enabled.

1. Navigate to the App Settings page.

2. Scroll to the Global Integrations Settings section.

3. Ensure the Snowflake Table Grants checkbox is checked. It is enabled by default.

4. Opt to change the Role Prefix. Snowflake table grants creates a new Snowflake role for each Immuta user. To ensure these Snowflake role names do not collide with existing Snowflake roles, each Snowflake role created for Snowflake table grants requires a common prefix. When using multiple Immuta accounts within a single Snowflake account, the Snowflake table grants role prefix should be unique for each Immuta account. The prefix must adhere to and be less than 50 characters. Once the configuration is saved, the prefix cannot be modified; however, the Snowflake table grants feature can be disabled and re-enabled to change the prefix.

5. Finish configuring your integration by following one of these guidelines:

   • New Snowflake integration: Set up a new Snowflake integration by following the .

   • Existing Snowflake integration (automatic setup): You will be prompted to enter connection information for a Snowflake user. Immuta will execute the migration to Snowflake table grants using a connection established with this Snowflake user. The Snowflake user you provide here must have Snowflake privileges to run these .

   • Existing Snowflake integration (manual setup): Immuta will display a link to a migration script you must run in Snowflake and a link to a rollback script for use in the event of a failed migration. Important: Execute the migration script in Snowflake before clicking Save on the app settings page.

Once a Databricks securable is registered in Immut as a data source and you are subscribed to that data source, you must access that data through SQL:

df = spark.sql("select * from immuta.table")

import org.apache.spark.sql.SparkSession

val spark = SparkSession
  .builder()
  .appName("Spark SQL basic example")
  .config("spark.some.config.option", "some-value")
  .getOrCreate()
val sqlDF = spark.sql("SELECT * FROM immuta.table")

%sql
select * from immuta.table

library(SparkR)
df <- SparkR::sql("SELECT * from immuta.table")

See the sections below for more guidance on accessing data using Delta Lake, direct file reads in Spark for file paths, and user impersonation.

Delta Lake

When using Delta Lake, the API does not go through the normal Spark execution path. This means that Immuta's Spark extensions do not provide protection for the API. To solve this issue and ensure that Immuta has control over what a user can access, the Delta Lake API is blocked.

Spark SQL can be used instead to give the same functionality with all of Immuta's data protections. See the Delta API reference guide for a list of corresponding Spark SQL calls to use.

Spark direct file reads

In addition to supporting direct file reads through workspace and scratch paths, Immuta allows direct file reads in Spark for file paths. As a result, users who prefer to interact with their data using file paths or who have existing workflows revolving around file paths can continue to use these workflows without rewriting those queries for Immuta.

When reading from a path in Spark, the Immuta Databricks Spark plugin queries the Immuta Web Service to find Databricks data sources for the current user that are backed by data from the specified path. If found, the query plan maps to the Immuta data source and follows existing code paths for policy enforcement.

Users can read data from individual parquet files in a sub-directory and partitioned data from a sub-directory (or by using a where predicate). Expand the blocks below to view examples of reading data using these methods.

spark.read.format("parquet").load("s3:/my_bucket/path/to/my_parquet_table/partition_column=01/my_file.parquet")

spark.read.format("parquet").load("s3:/my_bucket/path/to/my_parquet_table/partition_column=01")

spark.read.format("parquet").load("s3:/my_bucket/path/to/my_parquet_table").where("partition_column=01")Read partitioned data from a sub-directory

Limitations

• Direct file reads for Immuta data sources only apply to data sources created from tables, not data sources created from views or queries.

• If more than one data source has been created for a path, Immuta will use the first valid data source it finds. It is therefore not recommended to use this integration when more than one data source has been created for a path.

• In Databricks, multiple input paths are supported as long as they belong to the same data source.

• CSV-backed tables are not currently supported.

• Loading a delta partition from a sub-directory is not recommended by Spark and is not supported in Immuta. Instead, use a where predicate:

  Copy

  # Not recommended by Spark and not supported in Immuta
  spark.read.format("delta").load("s3:/my_bucket/path/to/my_delta_table/partition_column=01")
  
  # Recommended by Spark and supported in Immuta.
  spark.read.format("delta").load("s3:/my_bucket/path/to/my_delta_table").where("partition_column=01")
  

User impersonation

User impersonation allows Databricks users to query data as another Immuta user. To impersonate another user, see the User impersonation page.

Requirement

• Amazon RDS for MariaDB

Permissions

The user registering the connection must have the permissions below.

• APPLICATION_ADMIN Immuta permission

• The MariaDB user registering the connection and running the script must be the root user or have the GRANT OPTION MariaDB privilege.

Create a database user account

1. Create a new database user in MariaDB to serve as the Immuta system account. Immuta will use this system account continuously to crawl the database you register. How you create this user depends on your database authentication method. Follow the instructions linked below to create this user:

   1. Password authentication: Follow the MariaDB documentation to create the database user in and assign that user a password.

   2. IAM database authentication:

      1. Create the database user in MariaDB.

      2. Create an IAM policy for IAM database authentication.

      3. Create the database account.

2. Grant this account the following MariaDB privileges. A sample command that provides all these privileges to all databases and views is provided below:

   1. SHOW DATABASES on all databases in the server

   2. SELECT on all databases, tables, and views in the server

   3. SHOW VIEW on all views in the server

   Copy

   GRANT SELECT, SHOW DATABASES, SHOW VIEW ON *.* TO '<user>'@'%';

Register a MariaDB connection

1. In Immuta, click Data and select Connections in the navigation menu.

2. Click the + Add Connection button.

3. Select the MariaDB tile.

4. Select RDS as the deployment method.

5. Enter the host connection information:

   1. Display Name: This is the name of your new connection. This name will be used in the API (connectionKey), in data source names from the host, and on the connections page.

   2. Hostname: URL of your MariaDB instance.

   3. Port: Port configured with MariaDB.

   4. Region: The region of the AWS account with your MariaDB instance.

6. Select an authentication method from the dropdown menu.

   1. AWS Access Key: Provide the access key ID and secret access key for the database account you created above.

   2. AWS Assumed Role (recommended): Immuta will assume this IAM role from Immuta's AWS account to request temporary credentials that it can use to perform operations in the registered MariaDB database. Before proceeding, contact your Immuta representative and provide your service principal's IAM role. Immuta will allowlist the service principal so that Immuta can successfully assume that role. Your Immuta representative will provide the account to add to your trust relationship. Then, complete the steps below.

      1. Enter the Role ARN of the database account you created above.

      2. Set the external ID provided in a condition on the trust relationship for the role specified above. See the AWS documentation for guidance.

   3. Username and Password: Enter the credentials for the MariaDB database user account you created above.

7. Click Save connection.

Requirement

• Amazon RDS for Oracle

Permissions

The user registering the connection must have the permissions below.

• APPLICATION_ADMIN Immuta permission

• The Oracle user running the script must have either of the following system privileges:

  • GRANT ANY ROLE

  • GRANT ANY PRIVILEGE

Create the database user

1. Create a new database user in Oracle to serve as the Immuta system account. Immuta will use this system account continuously to crawl the connection.

2. Grant this account the SELECT Oracle privilege on the system views listed below:

   • V$DATABASE

   • CDB_PDBS

   • SYS.DBA_USERS

   • SYS.DBA_TABLES

   • SYS.DBA_VIEWS

   • SYS.DBA_MVIEWS

   • SYS.DBA_TAB_COLUMNS

   • SYS.DBA_OBJECTS

   • SYS.DBA_CONSTRAINTS

   • SYS.DBA_CONS_COLUMNS

Register an Oracle connection

1. In Immuta, click Data and select Connections in the navigation menu.

2. Click the + Add Connection button.

3. Select the Oracle tile.

4. Select RDS as the deployment method.

5. Enter the host connection information:

   1. Display Name: This is the name of your new connection. This name will be used in the API (connectionKey), in data source names from the host, and on the connections page.

   2. Hostname: URL of your Oracle instance.

   3. Port: Port configured for Oracle.

   4. Database: The Oracle database you want to connect to. All databases in the host will be registered.

   5. Region: The region of the AWS account with your Oracle instance.

6. Enter the username and password of the Oracle database user you created above.

7. Click Save connection.

To edit or remove a Snowflake integration, you have two options:

• Automatic: Grant Immuta one-time use of credentials with the following privileges to automatically edit or remove the integration:

  • CREATE DATABASE ON ACCOUNT WITH GRANT OPTION

  • CREATE ROLE ON ACCOUNT WITH GRANT OPTION

  • CREATE USER ON ACCOUNT WITH GRANT OPTION

  • MANAGE GRANTS ON ACCOUNT WITH GRANT OPTION

• Manual: Run the Immuta script in your Snowflake environment as a user with the following privileges to edit or remove the integration:

  • CREATE DATABASE ON ACCOUNT WITH GRANT OPTION

  • CREATE ROLE ON ACCOUNT WITH GRANT OPTION

  • CREATE USER ON ACCOUNT WITH GRANT OPTION

  • MANAGE GRANTS ON ACCOUNT WITH GRANT OPTION

  • APPLY MASKING POLICY ON ACCOUNT WITH GRANT OPTION

  • APPLY ROW ACCESS POLICY ON ACCOUNT WITH GRANT OPTION

Edit a Snowflake integration

Select one of the following options for editing your integration:

• Automatic: Grant Immuta one-time use of credentials to automatically edit the integration.

• Manual: Run the Immuta script in your Snowflake environment yourself to edit the integration.

Automatic edit

1. Click the App Settings icon in the navigation menu.

2. Click the Integrations tab and click the down arrow next to the Snowflake integration.

3. Edit the field you want to change or check a checkbox of a feature you would like to enable. Note any field shadowed is not editable, and the integration must be disabled and re-installed to change it.

4. From the Select Authentication Method Dropdown, select either Username and Password or Key Pair Authentication:

   • Username and Password option: Complete the Username, Password, and Role fields.

   • Key Pair Authentication option:

     1. Complete the Username field.

     2. Click Key Pair (Required), and upload a Snowflake key pair file.

     3. Complete the Role field.

5. Click Save.

Manual edit

1. Click the App Settings icon in the navigation menu.

2. Click the Integrations tab and click the down arrow next to the Snowflake integration.

3. Edit the field you want to change or check a checkbox of a feature you would like to enable. Note any field shadowed is not editable, and the integration must be disabled and re-installed to change it.

4. Click edit script to download the script, and then run it in Snowflake.

5. Click Save.

Remove a Snowflake integration

Select one of the following options for deleting your integration:

• Automatic: Grant Immuta one-time use of credentials to automatically remove the integration and Immuta-managed resources from your Snowflake environment.

• Manual: Run the Immuta script in your Snowflake environment yourself to remove Immuta-managed resources and policies from Snowflake.

Automatic removal

1. Click the App Settings icon in the navigation menu.

2. Click the Integrations tab and click the down arrow next to the Snowflake integration.

3. Click the checkbox to disable the integration.

4. Enter the Username, Password, and Role that was entered when the integration was configured.

5. Click Save.

Manual removal

1. Click the App Settings icon in the navigation menu.

2. Click the Integrations tab and click the down arrow next to the Snowflake integration.

3. Click the checkbox to disable the integration.

4. Click cleanup script to download the script.

5. Click Save.

6. Run the cleanup script in Snowflake.

The how-to guides linked on this page illustrate how to use PostgreSQL with Immuta. See the reference guide for information about the PostgreSQL connection.

The how-to guides linked on this page illustrate how to integrate Snowflake with Immuta. See the reference guide for information about the Snowflake integration.

Requirements

• Snowflake enterprise edition

• Access to a Snowflake account that can create a Snowflake user

This integration enforces policies on Databricks securables registered in the legacy Hive metastore. Once these securables are registered as Immuta data sources, users can query policy-enforced data on Databricks clusters.

The guides in this section outline how to integrate Databricks Spark with Immuta.

Getting started

This getting started guide outlines how to integrate Databricks with Immuta.

How-to guides

• Configure a Databricks Spark integration: Configure the Databricks Spark integration.

• Manually update your Databricks cluster: Manually update your cluster to reflect changes in the Immuta init script or cluster policies.

• Install a trusted library: Register a Databricks library with Immuta as a trusted library to avoid Immuta security manager errors when using third-party libraries.

• Project UDFs cache settings: Raise the caching on-cluster and lower the cache timeouts for the Immuta web service to allow use of project UDFs in Spark jobs.

• Run R and Scala spark-submit jobs on Databricks: Run R and Scala spark-submit jobs on your Databricks cluster.

• DBFS access: Access DBFS in Databricks for non-sensitive data.

• Troubleshooting: Resolve errors in the Databricks Spark configuration.

Reference guides

• Databricks Spark integration configuration: This guide describes the design and components of the integration.

• Security and compliance: This guide provides an overview of the Immuta features that provide security for your users and Databricks clusters and that allow you to prove compliance and monitor for anomalies.

• Registering and protecting data: This guide provides an overview of registering Databricks securables and protecting them with Immuta policies.

• Accessing data: This guide provides an overview of how Databricks users access data registered in Immuta.

This page describes the Azure Synapse integration, configuration options, and features. See the Azure Synapse integration page for a tutorial on enabling the integration and these features through the app settings page.

Feature Availability

Prerequisite

• A running Dedicated SQL pool

Authentication Method

Immuta only supports the SQL authentication option for Azure Synapse Analytics to configure the integration and create data sources. The Microsoft Entra authentication option is unsupported. See the SQL Authentication in Azure Synapse Analytics documentation for details.

Tag Ingestion

Immuta cannot ingest tags from Synapse, but you can connect any of these supported external catalogs to work with your integration.

User Impersonation

Impersonation allows users to query data as another Immuta user in Synapse. To enable user impersonation, see the User Impersonation page.

Multiple Integrations

A user can configure multiple integrations of Synapse to a single Immuta tenant.

Limitations

• Immuta does not support the following masking types in this integration because of limitations with Dedicated SQL pools (linked below). Any column assigned one of these masking types will be masked to NULL:

  • Reversible Masking: Synapse UDFs currently only support SQL, but Immuta needs to execute code (such as JavaScript or Python) to support this masking feature. See the Synapse Documentation for details.

  • Format Preserving Masking: Synapse UDFs currently only support SQL, but Immuta needs to execute code (such as JavaScript or Python) to support this masking feature. See the Synapse Documentation for details.

  • Regex: The built in string replace function does not support full regex. See the Synapse Documentation for details.

• The delimiters configured when enabling the integration cannot be changed once they are set. To change the delimiters, the integration has to be disabled and re-enabled.

• If the generated view name is more than 128 characters, then the view name is shortened to 128 characters. This could cause collisions between view names if the shortened version is the same for two different data sources.

• For proper updates, the Dedicated SQL pools have to be running when changes are made to users or data sources in Immuta.

The warehouse you select when configuring the Snowflake integration uses compute resources to set up the integration, register data sources, orchestrate policies, and run jobs like identification. Snowflake credit charges are based on the size of and amount of time the warehouse is active, not the number of queries run.

This document prescribes how and when to adjust the size and scale of clusters for your warehouse to manage workloads so that you can use Snowflake compute resources the most cost effectively.

In general, increase the size of and number of clusters for the warehouse to handle heavy workloads and multiple queries. Workloads are typically lighter after data sources are onboarded and policies are established in Immuta, so compute resources can be reduced after those workloads complete.

Integration and data source registration warehouse use

The Snowflake integration uses warehouse compute resources to sync policies created in Immuta to the Snowflake objects registered as data sources and, if configured, to run and . Follow the guidelines below to adjust the warehouse size and scale according to your needs.

• Increase the of and of clusters for the warehouse during large policy syncs, updates, and changes.

• Enable to optimize resource use in Snowflake. In the Snowflake UI, the lowest auto suspend time setting is 5 minutes. However, through SQL query, you can set auto_suspend to 61 seconds (since the minimum uptime for a warehouse is 60 seconds). For example,

• Identification uses compute resources for each table it runs on. Consider when registering data sources if you have an or a tagging strategy in place.

• Register data before creating global policies. Immuta does not apply a subscription policy on registered data unless an existing global policy applies to it, which allows Immuta to only pull metadata instead of also applying policies when data sources are created. Registering data before policies are created reduces the workload and the Snowflake compute resources needed.

• Begin onboarding with a small dataset of tables, and then review and monitor query performance in the . Adjust the virtual warehouse accordingly to handle heavier loads.

• uses the compute warehouse that was employed during the initial ingestion to periodically monitor the schema for changes. If you expect a low number of new tables or minimal changes to the table structure, consider scaling down the warehouse size.

• Resize the warehouse after after data sources are registered and policies are established. For example,

For more details and guidance about warehouse sizing, see the .

Identifying bulk jobs and heavy workloads

Even after your integration is configured, data sources are registered, and policies are established, changes to those data sources or policies may initiate heavy workloads. Follow the guidelines below to adjust your warehouse size and scale according to your needs.

• Review your to identify query performance and bottlenecks.

• Check how many credits queries have consumed:

• After reviewing query performance and cost, implement to adjust your warehouse.

Requirements

The requirements depend on your deployment type:

• Supported Azure SQL Server versions:

  • Azure SQL Database

  • Azure SQL Managed Instance

  • SQL Server on Azure VMs. Immuta supports the following:

    • SQL Server 2025 Preview

    • SQL Server 2022

    • SQL Server 2019

    • SQL Server 2017

    • SQL Server 2016

    • SQL Server 2014

    • SQL Server 2012

• Supported SQL Server on Amazon RDS versions:

  • SQL Server 2022 (16.0.4185.3)

  • SQL Server 2019 (15.0.4430.1)

  • SQL Server 2017 (14.0.3485.1)

  • SQL Server 2016 (13.0.6455.2)

Permissions

The user registering the connection must have the permissions below.

• APPLICATION_ADMIN Immuta permission

• The user registering the connection must have the following system privileges depending on your deployment type:

  • Azure SQL Server

  • SQL Server on AWS RDS

    • master user

Create the system account user

Create a new system account user for Immuta. Immuta will use the credentials of this system user to connect to SQL Server, ingest the data objects, and continually crawl the registered connection. See instructions below based on your deployment method:

Register a SQL Server connection

1. In your SQL Server environment, create an Immuta database that Immuta can use to connect to your SQL Server instance to register the connection and maintain state with SQL Server.

   Having this separate database for Immuta prevents custom ETL processes or jobs deleting the database you use to register the connection, which would break the connection.

2. In Immuta, click Data and select Connections in the navigation menu.

3. Click the + Add Connection button.

4. Select the SQL Server tile.

5. Select your deployment method:

   1. Azure SQL Server

   2. RDS

   3. Self-Managed

6. Enter the host connection information:

   1. Display Name: This is the name of your new connection. This name will be used in the API (connectionKey), in data source names from the host, and on the connections page.

   2. Hostname: URL of your SQL Server instance.

   3. Port: Port configured for SQL Server.

   4. Database: The SQL Server database you created for Immuta. All databases in the host will be registered.

7. Select an authentication method from the dropdown menu:

   1. Username and Password: Enter the credentials of the .

   2. Azure AD Access Token: Enter the and credentials of the .

8. Click Save connection.

The how-to guides linked on this page illustrate how to integrate Starburst (Trino) with Immuta. See the for information about the Starburst (Trino) integration.

This guide illustrates how to run R and Scala spark-submit jobs on Databricks, including prerequisites and caveats.

R spark-submit

Prerequisites

Before you can run spark-submit jobs on Databricks, complete the following steps.

1. Initialize the Spark session:

   1. Enter these settings into the R submit script to allow the R script to access Immuta data sources, scratch paths, and workspace tables: immuta.spark.acl.assume.not.privileged="true" and spark.hadoop.immuta.databricks.config.update.service.enabled="false".

   2. Once the script is written, upload the script to a location in dbfs/S3/ABFS to give the Databricks cluster access to it.

2. Because of how some user properties are populated in Databricks, load the SparkR library in a separate cell before attempting to use any SparkR functions.

Create the R spark submit Job

To create the R spark-submit job,

1. Go to the Databricks jobs page.

2. Create a new job, and select Configure spark-submit.

3. Set up the parameters:

   Copy

    [
    "--conf","spark.driver.extraJavaOptions=-Djava.security.manager=com.immuta.security.ImmutaSecurityManager -Dimmuta.security.manager.classes.config=file:///databricks/immuta/allowedCallingClasses.json -Dimmuta.spark.encryption.fpe.class=com.immuta.spark.encryption.ff1.ImmutaFF1Service",
    "--conf","spark.executor.extraJavaOptions=-Djava.security.manager=com.immuta.security.ImmutaSecurityManager -Dimmuta.security.manager.classes.config=file:///databricks/immuta/allowedCallingClasses.json -Dimmuta.spark.encryption.fpe.class=com.immuta.spark.encryption.ff1.ImmutaFF1Service",
    "--conf","spark.databricks.repl.allowedLanguages=python,sql,scala,r",
    "dbfs:/path/to/script.R",
    "arg1", "arg2", "..."
    ]

   Note: The path dbfs:/path/to/script.R can be in S3 or ABFS (on Azure Databricks), assuming the cluster is configured with access to that path.

4. Edit the cluster configuration, and change the Databricks Runtime to be a supported version.

5. Configure the Spark environment variables section as you normally would for an Immuta cluster.

Scala spark-submit

Prerequisites

Before you can run spark-submit jobs on Databricks you must initialize the Spark session with the settings outlined below.

1. Configure the Spark session with immuta.spark.acl.assume.not.privileged="true" and spark.hadoop.immuta.databricks.config.update.service.enabled="false".

   Note: Stop your Spark session (spark.stop()) at the end of your job or the cluster will not terminate.

2. The spark submit job needs to be launched using a different classloader which will point at the designated user JARs directory. The following Scala template can be used to handle launching your submit code using a separate classloader:

   Copy

   package com.example.job
   
   import java.net.URLClassLoader
   import java.io.File
   
   import org.apache.spark.sql.SparkSession
   
   object ImmutaSparkSubmitExample {
   def main(args: Array[String]): Unit = {
       val jarDir = new File("/databricks/immuta/jars/")
       val urls = jarDir.listFiles.map(_.toURI.toURL)
   
       // Configure a new ClassLoader which will load jars from the additional jars directory
       val cl = new URLClassLoader(urls)
       val jobClass = cl.loadClass(classOf[ImmutaSparkSubmitExample].getName)
       val job = jobClass.newInstance
       jobClass.getMethod("runJob").invoke(job)
   }
   }
   
   class ImmutaSparkSubmitExample {
   
   def getSparkSession(): SparkSession = {
       SparkSession.builder()
       .appName("Example Spark Submit")
       .enableHiveSupport()
       .config("immuta.spark.acl.assume.not.privileged", "true")
       .config("spark.hadoop.immuta.databricks.config.update.service.enabled", "false")
       .getOrCreate()
   }
   
   def runJob(): Unit = {
       val spark = getSparkSession
       try {
       val df = spark.table("immuta.<YOUR DATASOURCE>")
   
       // Run Immuta Spark queries...
   
       } finally {
       spark.stop()
       }
   }
   }

Create the Scala spark-submit Job

To create the Scala spark-submit job,

1. Build and upload your JAR to dbfs/S3/ABFS where the cluster has access to it.

2. Select Configure spark-submit, and configure the parameters:

   Copy

    [
    "--conf","spark.driver.extraJavaOptions=-Djava.security.manager=com.immuta.security.ImmutaSecurityManager -Dimmuta.security.manager.classes.config=file:///databricks/immuta/allowedCallingClasses.json -Dimmuta.spark.encryption.fpe.class=com.immuta.spark.encryption.ff1.ImmutaFF1Service",
    "--conf","spark.executor.extraJavaOptions=-Djava.security.manager=com.immuta.security.ImmutaSecurityManager -Dimmuta.security.manager.classes.config=file:///databricks/immuta/allowedCallingClasses.json -Dimmuta.spark.encryption.fpe.class=com.immuta.spark.encryption.ff1.ImmutaFF1Service",
    "--conf","spark.databricks.repl.allowedLanguages=python,sql,scala,r",
    "--class","org.youorg.package.MainClass",
    "dbfs:/path/to/code.jar",
    "arg1", "arg2", "..."
    ]

   Note: The fully-qualified class name of the class whose main function will be used as the entry point for your code in the --class parameter.

   Note: The path dbfs:/path/to/code.jar can be in S3 or ABFS (on Azure Databricks) assuming the cluster is configured with access to that path.

3. Edit the cluster configuration, and change the Databricks Runtime to a supported version.

4. Include IMMUTA_INIT_ADDITIONAL_JARS_URI=dbfs:/path/to/code.jar in the "Environment Variables" (where dbfs:/path/to/code.jar is the path to your jar) so that the jar is uploaded to all the cluster nodes.

Caveats

• The user mapping works differently from notebooks because spark-submit clusters are not configured with access to the Databricks SCIM API. The cluster tags are read to get the cluster creator and match that user to an Immuta user.

• Privileged users (Databricks admins and allowlisted users) must be tied to an Immuta user and given access through Immuta to access data through spark-submit jobs because the setting immuta.spark.acl.assume.not.privileged="true" is used.

• There is an option of using the immuta.api.key setting with an Immuta API key generated on the Immuta profile page.

• Currently when an API key is generated it invalidates the previous key. This can cause issues if a user is using multiple clusters in parallel, since each cluster will generate a new API key for that Immuta user. To avoid these issues, manually generate the API key in Immuta and set the immuta.api.key on all the clusters or use a specified job user for the submit job.

Immuta offers several features to provide security for your users and Databricks clusters and to prove compliance and monitor for anomalies.

Authentication

Configuring the integration and registering data

Immuta supports the following authentication methods to configure the Databricks Spark integration and register data sources:

• OAuth machine-to-machine (M2M): Immuta uses the Client Credentials Flow to integrate with Databricks OAuth machine-to-machine authentication, which allows Immuta to authenticate with Databricks using a client secret. Once Databricks verifies the Immuta service principal’s identity using the client secret, Immuta is granted a temporary OAuth token to perform token-based authentication in subsequent requests. When that token expires (after one hour), Immuta requests a new temporary token. See the Databricks OAuth machine-to-machine (M2M) authentication page for more details.

• Personal access token (PAT): This token gives Immuta temporary permission to push the cluster policies to the configured Databricks workspace and overwrite any cluster policy templates previously applied to the workspace when configuring the integration or to register securables as Immuta data sources.

User authentication

The built-in Immuta IAM can be used as a complete solution for authentication and fine-grained user entitlement. However, you can connect your existing identity management provider to Immuta to use that system for authentication and fine-grained user entitlement instead.

Each of the supported identity providers includes a specific set of configuration options that enable Immuta to communicate with the IAM system and map the users, permissions, groups, and attributes into Immuta.

See the Identity managers guide for a list of supported providers and details.

See the Setting up users guide for details and instructions on mapping Databricks user accounts to Immuta.

Cluster security

Data processing and encryption

See the Data processing and the Encryption and masking practices guides for more information about transmission of policy decision data, encryption of data in transit and at rest, and encryption key management.

Protecting the Immuta configuration

Non-administrator users on an Immuta-enabled Databricks cluster must not have access to view or modify Immuta configuration, as this poses a security loophole around Immuta policy enforcement. Databricks secrets allow you to securely apply environment variables to Immuta-enabled clusters.

Databricks secrets can be used in the environment variables configuration section for a cluster by referencing the secret path instead of the actual value of the environment variable.

See the Installation and compliance guide for details and instructions on using Databricks secrets.

Scala cluster security

There are limitations to isolation among users in Scala jobs on a Databricks cluster. When data is broadcast, cached (spilled to disk), or otherwise saved to SPARK_LOCAL_DIR, it's impossible to distinguish between which user’s data is composed in each file/block. To address this vulnerability, Immuta suggests that you

• limit Scala clusters to Scala jobs only and

• require equalized projects, which will force all users to act under the same set of attributes, groups, and purposes with respect to their data access. This requirement guarantees that data being dropped into SPARK_LOCAL_DIR will have policies enforced and that those policies will be homogeneous for all users on the cluster. Since each user will have access to the same data, if they attempt to manually access other users' cached/spilled data, they will only see what they have access to via equalized permissions on the cluster. If project equalization is not turned on, users could dig through that directory and find data from another user with heightened access, which would result in a data leak.

See the Installation and compliance guide for more details and configuration instructions.

Auditing and compliance

Immuta provides auditing features and governance reports so that data owners and governors can monitor users' access to data and detect anomalies in behavior.

You can view the information in these audit logs on dashboards or export the full audit logs to S3 and ADLS for long-term backup and processing with log data processors and tools. This capability fosters convenient integrations with log monitoring services and data pipelines.

See the Audit documentation for details about these capabilities and how they work with the Databricks Spark integration.

Databricks query audit

Immuta captures the code or query that triggers the Spark plan in Databricks, making audit records more useful in assessing what users are doing.

To audit what triggers the Spark plan, Immuta hooks into Databricks where notebook cells and JDBC queries execute and saves the cell or query text. Then, Immuta pulls this information into the audits of the resulting Spark jobs.

Immuta will audit queries that come from interactive notebooks, notebook jobs, and JDBC connections, but will not audit Scala or R submit jobs. Furthermore, Immuta only audits Spark jobs that are associated with Immuta tables. Consequently, Immuta will not audit a query in a notebook cell that does not trigger a Spark job, unless IMMUTA_SPARK_AUDI_ALL_QUERIES is set to true.

See the Databricks Spark query audit logs page for examples of saved queries and the resulting audit records. To exclude query text from audit events, see the App settings page.

Auditing all queries

Immuta supports auditing all queries run on a Databricks cluster, regardless of whether users touch Immuta-protected data or not.

See the Installation and compliance guide for details and instructions.

Auditing queries run while impersonating another user

When a query is run by a user impersonating another user, the extra.impersonationUser field in the audit log payload is populated with the Databricks username of the user impersonating another user. The userId field will return the Immuta username of the user being impersonated:

{
  "id": "query-a20e-493e-id-c1ada0a23a26",
  [...]
  "userId": "<immuta_username>",
  [...]
  "extra": {
    [...]
    "impersonationUser": "<databricks_username>"
  }
  [...]
}

See the Setting up users guide for details about user impersonation.

Governance reports

Immuta governance reports allow users with the GOVERNANCE Immuta permission to use a natural language builder to instantly create reports that delineate user activity across Immuta. These reports can be based on various entity types, including users, groups, projects, data sources, purposes, policy types, or connection types.

See the Governance report types page for a list of report types and guidance.

The how-to guides linked on this page illustrate how to integrate Databricks Unity Catalog with Immuta. See the reference guide for information about the Databricks Unity Catalog integration.

Requirements:

• Unity Catalog metastore created and attached to a Databricks workspace. Immuta supports configuring a single metastore for each configured integration, and that metastore may be attached to multiple Databricks workspaces.

• Unity Catalog enabled on your Databricks cluster or SQL warehouse. All SQL warehouses have Unity Catalog enabled if your workspace is attached to a Unity Catalog metastore.