Immuta integrates with your data platforms and external catalogs so you can register your data and effectively manage access controls on that data.

This section includes concept, reference, and how-to guides for configuring your data platform integration, registering data sources, and connecting your external catalog so that you can discover, monitor, and protect sensitive data using Immuta's flagship modules: Discover, Detect, and Secure.

Immuta integrations

This reference guide outlines the features, policies, and audit capabilities supported by each integration.

Snowflake

This section includes how-to and reference guides for Snowflake and how it integrates with Immuta.

Databricks Unity Catalog

This section includes how-to and reference guides for Databricks Unity Catalog and how it integrates with Immuta.

Databricks Spark

This section includes how-to and reference guides for Databricks Spark and how it integrates with Immuta.

Starburst (Trino)

This section includes how-to and reference guides for Starburst (Trino) and how it integrates with Immuta.

Redshift

This section includes how-to and reference guides for Redshift and how it integrates with Immuta.

Azure Synapse Analytics

This section includes how-to and reference guides for Azure Synapse Analytics and how it integrates with Immuta.

Amazon S3

This page includes how-to and reference content for Amazon S3 and how it integrates with Immuta.

Google BigQuery

This page includes how-to and reference content for Google BigQuery and how it integrates with Immuta.

Catalogs

This section covers the various data catalogs Immuta integrates with.

Queries Immuta runs in remote platforms

This reference guide outlines the actions and features that trigger Immuta queries in your remote platform that may incur cost.

Immuta does not require users to learn a new API or language to access protected data. Instead, Immuta integrates with existing tools and ongoing work while remaining invisible to downstream consumers.

The following data platforms integrate with Immuta:

• Snowflake integration: With this integration, policies administered in Immuta are pushed down into Snowflake as Snowflake governance features (row access policies and masking policies).

• Databricks:

  • Databricks Unity Catalog integration: This integration allows you to manage multiple Databricks workspaces through Unity Catalog while protecting your data with Immuta policies. Instead of manually creating UDFs or granting access to each table in Databricks, you can author your policies in Immuta and have Immuta manage and enforce Unity Catalog access-control policies on your data in Databricks clusters or SQL warehouse.

  • Databricks Spark integration: This integration enforces policies on Databricks tables registered as data sources in Immuta, allowing users to query policy-enforced data on Databricks clusters (including job clusters). Immuta policies are applied to the plan that Spark builds for users' queries, all executed directly against Databricks tables.

• Google BigQuery: In this integration, Immuta generates policy-enforced views in your configured Google BigQuery dataset for tables registered as Immuta data sources.

• Starburst (Trino) integration: The Starburst (Trino) integration allows you to access policy-protected data directly in your Starburst (Trino) catalogs without rewriting queries or changing your workflows. Immuta policies are translated into Starburst (Trino) rules and permissions and applied directly to tables within your existing catalogs.

• Redshift integration: With the Redshift integration, Immuta applies policies directly in Redshift. This allows data analysts to query their data directly in Redshift instead of going through a proxy.

• Azure Synapse Analytics integration: The Azure Synapse Analytics integration allows Immuta to apply policies directly in Azure Synapse Analytics dedicated SQL pools without needing users to go through a proxy. Instead, users can work within their existing Synapse Studio and have per-user policies dynamically applied at query time.

• Amazon S3 integration: The Amazon S3 integration allows users to apply subscription policies to data in S3 to restrict what prefixes, buckets, or objects users can access. To enforce access controls on this data, Immuta creates S3 grants that are administered by S3 Access Grants, an AWS feature that defines access permissions to data in S3.

Feature support

The table below outlines the features supported by each of Immuta's integrations.

Policy support

Certain policies are unsupported or supported with caveats*, depending on the integration:

*Supported with caveats:

• On Databricks data sources, joins will not be allowed on data protected with replace with NULL or constant policies.

• Databricks Unity Catalog ARRAY, MAP, or STRUCT type columns only support masking with NULL.

• On Starburst data sources, the Immuta @iam function for WHERE clause policies can block the creation of views.

For details about each of these policies, see the Policies in Immuta page.

Audit support for platform queries

The table below outlines what information is included in the query audit logs for each integration where query audit is supported.

Legend:

To edit or remove a Snowflake integration, you have two options:

• Automatic: Grant Immuta one-time use of credentials to automatically edit or remove the integration.

  The credentials provided must have the following permissions:

  • CREATE DATABASE ON ACCOUNT WITH GRANT OPTION

  • CREATE ROLE ON ACCOUNT WITH GRANT OPTION

  • CREATE USER ON ACCOUNT WITH GRANT OPTION

  • MANAGE GRANTS ON ACCOUNT WITH GRANT OPTION

• Manual: Run the Immuta script in your Snowflake environment yourself to edit or remove the integration.

  The specified role used to run the bootstrap needs to have the following privileges:

  • CREATE DATABASE ON ACCOUNT WITH GRANT OPTION

  • CREATE ROLE ON ACCOUNT WITH GRANT OPTION

  • CREATE USER ON ACCOUNT WITH GRANT OPTION

  • MANAGE GRANTS ON ACCOUNT WITH GRANT OPTION

  • APPLY MASKING POLICY ON ACCOUNT WITH GRANT OPTION

  • APPLY ROW ACCESS POLICY ON ACCOUNT WITH GRANT OPTION

Edit a Snowflake integration

Select one of the following options for editing your integration:

• Automatic: Grant Immuta one-time use of credentials to automatically edit the integration.

• Manual: Run the Immuta script in your Snowflake environment yourself to edit the integration.

Automatic edit

1. Click the App Settings icon in the left sidebar.

2. Click the Integrations tab and click the down arrow next to the Snowflake integration.

3. Edit the field you want to change or check a checkbox of a feature you would like to enable. Note any field shadowed is not editable, and the integration must be disabled and re-installed to change it.

4. From the Select Authentication Method Dropdown, select either Username and Password or Key Pair Authentication:

   • Username and Password option: Complete the Username, Password, and Role fields.

   • Key Pair Authentication option:

     1. Complete the Username field.

     2. Click Key Pair (Required), and upload a Snowflake key pair file.

     3. Complete the Role field.

5. Click Save.

Manual edit

1. Click the App Settings icon in the left sidebar.

2. Click the Integrations tab and click the down arrow next to the Snowflake integration.

3. Edit the field you want to change or check a checkbox of a feature you would like to enable. Note any field shadowed is not editable, and the integration must be disabled and re-installed to change it.

4. Click edit script to download the script, and then run it in Snowflake.

5. Click Save.

Remove a Snowflake integration

Select one of the following options for deleting your integration:

• Automatic: Grant Immuta one-time use of credentials to automatically remove the integration and Immuta-managed resources from your Snowflake environment.

• Manual: Run the Immuta script in your Snowflake environment yourself to remove Immuta-managed resources and policies from Snowflake.

Automatic removal

1. Click the App Settings icon in the left sidebar.

2. Click the Integrations tab and click the down arrow next to the Snowflake integration.

3. Click the checkbox to disable the integration.

4. Enter the Username, Password, and Role that was entered when the integration was configured.

5. Click Save.

Manual removal

1. Click the App Settings icon in the left sidebar.

2. Click the Integrations tab and click the down arrow next to the Snowflake integration.

3. Click the checkbox to disable the integration.

4. Click cleanup script to download the script.

5. Click Save.

6. Run the cleanup script in Snowflake.

Immuta manages access to Snowflake tables by administering Snowflake row access policies and column masking policies on those tables, allowing users to query tables directly in Snowflake while dynamic policies are enforced.

Getting started

This getting started guide outlines how to integrate your Snowflake account with Immuta.

How-to guides

• Configure a Snowflake integration: Configure the Snowflake integration.

• Edit or remove an existing integration: Manage integration settings or delete your existing Snowflake integration.

• Integration settings:

  • Enable Snowflake table grants: Enable Snowflake table grants and configure the Snowflake role prefix.

  • Use Snowflake data sharing with Immuta: Use Snowflake data sharing with table grants or project workspaces.

  • Snowflake low row access policy mode: Enable Snowflake low row access policy mode.

  • Snowflake lineage tag propagation: Configure your Snowflake integration to automatically apply tags added to a Snowflake table to its descendant data source columns in Immuta.

Reference guides

• Phased Snowflake onboarding approach: A phased onboarding approach to configuring the Snowflake integration ensures that your users will not be immediately affected by changes as you add data sources and policies. This guide describes the settings and requirements for implementing this phased approach.

• Snowflake integration reference guide: This reference guide describes the design and features of the Snowflake integration.

• Integration health statuses: This reference guide provides descriptions of the possible statuses of a configured integration.

• Snowflake table grants: Snowflake table grants simplifies the management of privileges in Snowflake when using Immuta. Instead of manually granting users access to tables registered in Immuta, you allow Immuta to manage privileges on your Snowflake tables and views according to subscription policies. This guide describes the components of Snowflake table grants and how they are used in Immuta's Snowflake integration.

• Snowflake data sharing with Immuta: Organizations can share the policy-protected data of their Snowflake database with other Snowflake accounts with Immuta policies enforced in real time. This guide describes the components of using Immuta with Snowflake data shares.

• Snowflake low row access policy mode: The Snowflake low row access policy mode improves query performance in Immuta's Snowflake integration. To do so, this mode decreases the number of Snowflake row access policies Immuta creates and uses table grants to manage user access. This guide describes the design and requirements of this mode.

• Snowflake lineage tag propagation: Snowflake column lineage specifies how data flows from source tables or columns to the target tables in write operations. When Snowflake lineage tag propagation is enabled in Immuta, Immuta automatically applies tags added to a Snowflake table to its descendant data source columns in Immuta so you can build policies using those tags to restrict access to sensitive data.

• Warehouse sizing recommendations: Adjust the size and scale of clusters for your warehouse to manage workloads so that you can use Snowflake compute resources the most cost effectively.

The how-to guides linked on this page illustrate how to integrate Snowflake with Immuta.

Requirement: Snowflake Enterprise Edition

Configure your Snowflake integration

These guides provide information on the recommended features to enable with Snowflake.

1. Configure your Snowflake integration with the following features enabled:

   • Snowflake table grants (enabled by default)

   • Snowflake low row access policy mode (enabled by default)

   • Snowflake native query audit (enabled by default)

2. Select None as your default subscription policy.

3. Integrate an IAM with Immuta.

4. Map external user IDs from Snowflake to Immuta.

Register metadata

These guides provide instructions for organizing your Snowflake data to align with your governance structure.

1. Register Snowflake data in Immuta as data sources.

2. Recommended: Organize your data sources into domains and assign domain permissions to accountable teams.

Detect your user activity

These guides provide instructions for auditing and detecting your users' activity, or see the Detect use case for a comprehensive guide on the benefits of these features and other recommendations.

1. Set up audit export to S3 or ADLS Gen2 for your Snowflake audit logs.

2. View the Detect dashboards to see the activity of your registered users on registered tables.

Discover your data

These guides provide instructions for discovering, classifying, and tagging your data.

1. Enable sensitive data discovery (SDD).

2. Register a subset of your tables to configure and validate SDD.

3. Configure SDD to discover entities of interest for your policy needs.

4. Validate that the SDD tags are applied correctly.

5. Register your remaining tables at the schema level with schema monitoring turned on.

6. Implement classification to categorize and tag sensitive data.

Secure your data

These guides provide instructions for configuring and securing your data with governance policies, or see the Secure use cases for a comprehensive guide on creating policies to fit your organization's use case.

1. Create a global subscription policy.

2. Validate the policy. You do not have to validate every policy you create in Immuta; instead, examine a few to validate the behavior you expect to see:

   1. Validate that the Immuta users impacted now have an Immuta role in Snowflake dedicated to them.

   2. Validate that when acting under the Immuta role those users have access to the table(s) in question.

   3. Validate that users without access in Immuta can still access the table with a different Snowflake role that has access.

   4. Validate that a user with SECONDARY ROLES ALL enabled retains access if

      • they were not granted access by Immuta and

      • they have a role that provides them access, even if they are not currently acting under that role.

3. Create a global data policy.

4. Validate that a user with a role that can access the table in question (whether it's an Immuta role or not) sees the impact of that data policy.

5. Once all Immuta policies are in place, remove or alter old roles.

To migrate from the private preview version of table grants (available before September 2022) to the GA version, complete the steps below.

1. Navigate to the App Settings page.

2. Scroll to the Global Integrations Settings section.

3. Uncheck the Snowflake Table Grants checkbox to disable the feature.

4. Click Save. Wait for about 1 minute per 1000 users. This gives time for Immuta to drop all the previously created user roles.

5. Use the Enable Snowflake table grants tutorial to re-enable the feature.

Immuta is compatible with . Using both Immuta and Snowflake, organizations can share the policy-protected data of their Snowflake database with other Snowflake accounts with Immuta policies enforced in real time.

Prerequisites:

Create Immuta Policies to Protect the Data

Required Permission: Immuta: GOVERNANCE

to fit your organization's compliance requirements.

It's important to understand that subscription policies are not relevant to Snowflake data shares, because the act of sharing the data is the subscription policy. Data policies can be enforced on the consuming account from the producer account on a share following these instructions.

Register the Snowflake Data Consumer with Immuta

Required Permission: Immuta: USER_ADMIN

To register the Snowflake data consumer in Immuta,

1. .

2. to match the account ID for the data consumer. This value is the output on the data consumer side when SELECT CURRENT_ACCOUNT() is run in Snowflake.

3. for your organization's policies.

4. .

Create the Snowflake Data Share

Required Permission: Snowflake ACCOUNTADMIN

To share the policy-protected data source,

2. Grant reference usage on the Immuta database to the share you created:

   Replace the content in angle brackets above with the name of your Immuta database and Snowflake data share.

Prerequisites

This upgrade step is necessary if you meet both of the following criteria:

• You have the Snowflake low row access policy mode enabled in private preview.

• You have user impersonation enabled.

If you do not meet this criteria, follow the instructions on the .

Upgrade to Snowflake low row access policy mode

To upgrade to the generally available version of the feature, on the app settings page and then re-enable it.

1. Navigate to the App Settings page.

2. Scroll to the Global Integrations Settings section.

3. Ensure the Snowflake Table Grants checkbox is checked. It is enabled by default.

4. Opt to change the Role Prefix. Snowflake table grants creates a new Snowflake role for each Immuta user. To ensure these Snowflake role names do not collide with existing Snowflake roles, each Snowflake role created for Snowflake table grants requires a common prefix. When using multiple Immuta accounts within a single Snowflake account, the Snowflake table grants role prefix should be unique for each Immuta account. The prefix must adhere to and be less than 50 characters. Once the configuration is saved, the prefix cannot be modified; however, the Snowflake table grants feature can be disabled and re-enabled to change the prefix.

5. Finish configuring your integration by following one of these guidelines:

   • New Snowflake integration: Set up a new Snowflake integration by following the .

   • Existing Snowflake integration (automatic setup): You will be prompted to enter connection information for a Snowflake user. Immuta will execute the migration to Snowflake table grants using a connection established with this Snowflake user. The Snowflake user you provide here must have Snowflake privileges to run these .

   • Existing Snowflake integration (manual setup): Immuta will display a link to a migration script you must run in Snowflake and a link to a rollback script for use in the event of a failed migration. Important: Execute the migration script in Snowflake before clicking Save on the app settings page.

Immuta helps you achieve the following outcomes in your data platform:

• Simplify Operations: Immuta’s dynamic access control and policy management require 93x fewer data policies to manage access control in your data platform according to the . It is simple and scalable, which improves change management and lowers the total cost of ownership of cloud data management.

• Improve data security: Immuta helps prove compliance with rules and regulations, even when securing hundreds of thousands of tables. An Immuta customer, , migrated all critical analytics workloads to the cloud in less than 12 months, including over 100 terabytes from more than 2,500 sources.

• Unlock data’s value: Immuta helps organizations get access to more data 100x faster, which translates to improved productivity. An Immuta customer, enabled faster access to data, resulting in a 60x increase in data usage and greater productivity.

How does Immuta do it?

Immuta provides three modules to create a full data security platform suite.

Sensitive data discovery and classification:

Discover sensitive data from millions of fields without manual effort. With over 60 pre-built and domain-specific identifiers, you can tailor data classification to your unique business needs based on your desired confidence level.

Continuous data security monitoring:

Leverage timely insights into data access and user activity with anomaly indicators for faster analysis and proactive actions.

Data security and access control:

Immuta’s attribute-based access control (ABAC) delivers scalable data access without role explosion, and dynamic data masking ensures the right users can access the right data.

1. Click the App Settings icon in the sidebar and scroll to the Global Integration Settings section.

2. Click the Enable Snowflake Low Row Access Policy Mode checkbox to enable the feature.

3. Confirm to allow Immuta to automatically disable impersonation for the Snowflake integration. If you do not confirm, you will not be able to enable Snowflake low row access policy mode.

4. Click Save.

Configure your Snowflake integration

If you already have a Snowflake integration configured, you don't need to reconfigure your integration. Your Snowflake policies automatically refresh when you enable Snowflake low row access policy mode.

1. Configure your Snowflake integration. Note that you will not be able to enable project workspaces or user impersonation with Snowflake low row access policy mode enabled.

2. Click Save and Confirm your changes.

Immuta is compatible with Snowflake Secure Data Sharing. Using both Immuta and Snowflake, organizations can share the policy-protected data of their Snowflake database with other Snowflake accounts with Immuta policies enforced in real time. This integration gives data consumers a live connection to the data and relieves data providers of the legal and technical burden of creating static data copies that leave their Snowflake environment.

Requirements:

• Snowflake Enterprise Edition or higher

• Immuta's table grants feature

Configuration

This method requires that the data consumer account is registered as an Immuta user with the Snowflake user name equal to the consuming account.

At that point, the user that represents the account being shared with can have the appropriate attributes and groups assigned to them, relevant to the data policies that need to be enforced. Once that user has access to the share in the consuming account (not managed by Immuta), they can query the share with the data policies from the producer account enforced because Immuta is treating that account as if they are a single user in Immuta.

For a tutorial on this workflow, see the Using Snowflake Data Sharing page.

Benefits

Using Immuta with Snowflake Data Sharing allows the sharer to

• Only need limited knowledge of the context or goals of the existing policies in place: Because the sharer is not editing or creating policies to share their data, they only need a limited knowledge of how the policies work. Their main responsibility is making sure they properly represent the attributes of the data consumer (the account being shared to).

• Leave policies untouched.

This page details how to configure the Snowflake integration using the legacy workflow. To configure the Snowflake integration and register data sources using the simplified workflow, see this how-to guide.

Permissions

Automatic setup permissions

When performing an automated installation, Immuta requires temporary, one-time use of credentials with the following permissions:

• CREATE DATABASE ON ACCOUNT WITH GRANT OPTION

• CREATE ROLE ON ACCOUNT WITH GRANT OPTION

• CREATE USER ON ACCOUNT WITH GRANT OPTION

• MANAGE GRANTS ON ACCOUNT WITH GRANT OPTION

• APPLY MASKING POLICY ON ACCOUNT WITH GRANT OPTION

• APPLY ROW ACCESS POLICY ON ACCOUNT WITH GRANT OPTION

These permissions will be used to create and configure a new IMMUTA database within the specified Snowflake instance. The credentials are not stored or saved by Immuta, and Immuta doesn’t retain access to them after initial setup is complete.

You can create a new account for Immuta to use that has these permissions, or you can grant temporary use of a pre-existing account. By default, the pre-existing account with appropriate permissions is ACCOUNTADMIN. If you create a new account, it can be deleted after initial setup is complete.

Alternatively, you can create the IMMUTA database within the specified Snowflake instance manually using the manual setup option.

Manual setup permissions

The specified role used to run the bootstrap needs to have the following privileges:

• CREATE DATABASE ON ACCOUNT WITH GRANT OPTION

• CREATE ROLE ON ACCOUNT WITH GRANT OPTION

• CREATE USER ON ACCOUNT WITH GRANT OPTION

• MANAGE GRANTS ON ACCOUNT WITH GRANT OPTION

• APPLY MASKING POLICY ON ACCOUNT WITH GRANT OPTION

• APPLY ROW ACCESS POLICY ON ACCOUNT WITH GRANT OPTION

It will create a user called IMMUTA_SYSTEM_ACCOUNT, and grant the following privileges to that user:

• APPLY MASKING POLICY ON ACCOUNT

• APPLY ROW ACCESS POLICY ON ACCOUNT

• Additional grants associated with the IMMUTA database

Configure the integration

1. Opt to configure private connectivity for Snowflake.

2. Click the App Settings icon in the navigation panel.

3. Click the Integrations tab.

4. Click the +Add Native Integration button and select Snowflake from the dropdown menu.

5. Complete the Host, Port, and Default Warehouse fields.

6. Opt to check the Enable Project Workspace box. This will allow for managed write access within Snowflake. Note: Project workspaces still use Snowflake views, so the default role of the account used to create the data sources in the project must be added to the Excepted Roles List. This option is unavailable when table grants is enabled.

7. Opt to check the Enable Impersonation box and customize the Impersonation Role to allow users to natively impersonate another user. You cannot edit this choice after you configure the integration.

8. Snowflake query audit is enabled by default; you can disable it by clicking the Enable Native Query Audit checkbox.

   1. Configure the audit frequency by scrolling to Integrations Settings and find the Snowflake Audit Sync Schedule section.

   2. Enter how often, in hours, you want Immuta to ingest audit events from Snowflake as an integer between 1 and 24.

   3. Continue with your integration configuration.

Select your configuration method

You have two options for configuring your Snowflake environment:

• Automatic setup: Grant Immuta one-time use of credentials to automatically configure your Snowflake environment and the integration.

• Manual setup: Run the Immuta script in your Snowflake environment yourself to configure your Snowflake environment and the integration.

Automatic setup

Required permissions: When performing an automated installation, Immuta requires temporary, one-time use of credentials with the Snowflake permissions listed above.

From the Select Authentication Method Dropdown, select one of the following authentication methods:

• Username and Password: Complete the Username, Password, and Role fields.

• Key Pair Authentication:

  1. Complete the Username field.

  2. When using a private key, enter the private key file password in the Additional Connection String Options. Use the following format: PRIV_KEY_FILE_PWD=<your_pw>

  3. Click Key Pair (Required), and upload a Snowflake key pair file.

  4. Complete the Role field.

Manual setup

Required permissions: The specified role used to run the bootstrap needs to have the Snowflake permissions listed above.

It will create a user called IMMUTA_SYSTEM_ACCOUNT, and grant the following privileges to that user:

• APPLY MASKING POLICY ON ACCOUNT

• APPLY ROW ACCESS POLICY ON ACCOUNT

• Additional grants associated with the IMMUTA database

Run the script

1. Select Manual.

2. Use the Dropdown Menu to select your Authentication Method:

   • Username and password: Enter the Username and Password and set them in the bootstrap script for the Immuta system account credentials.

   • Key pair authentication: Upload the Key Pair file and when using a private key, enter the private key file password in the Additional Connection String Options. Use the following format: PRIV_KEY_FILE_PWD=<your_pw>

   • Snowflake External OAuth:

     1. Create a security integration for your Snowflake External OAuth. Note that if you have an existing security integration, then the Immuta system role must be added to the existing EXTERNAL_OAUTH_ALLOWED_ROLES_LIST. The Immuta system role will be the Immuta database provided above with _SYSTEM. If you used the default database name it will be IMMUTA_SYSTEM.

     2. Fill out the Token Endpoint. This is where the generated token is sent.

     3. Fill out the Client ID. This is the subject of the generated token.

     4. Select the method Immuta will use to obtain an access token:

        • Certificate

          1. Keep the Use Certificate checkbox enabled.

          2. Opt to fill out the Resource field with a URI of the resource where the requested token will be used.

          3. Enter the x509 Certificate Thumbprint. This identifies the corresponding key to the token and is often abbreviated as `x5t` or is called `sub` (Subject).

          4. Upload the PEM Certificate, which is the client certificate that is used to sign the authorization request.

        • Client secret

          1. Uncheck the Use Certificate checkbox.

          2. Enter the Scope (string). The scope limits the operations and roles allowed in Snowflake by the access token. See the OAuth 2.0 scopes documentation for details about scopes.

          3. Enter the Client Secret (string). Immuta uses this secret to authenticate with the authorization server when it requests a token.

3. In the Setup section, click bootstrap script to download the script. Then, fill out the appropriate fields and run the bootstrap script in Snowflake.

Select available warehouses (optional)

If you enabled a Snowflake workspace, select Warehouses from the dropdown menu that will be available to project owners when creating native Snowflake workspaces. Select from a list of all the warehouses available to the privileged account entered above. Note that any warehouse accessible by the PUBLIC role does not need to be explicitly added.

Select excepted roles and users

Enter the Excepted Roles/User List. Each role or username (both case-sensitive) in this list should be separated by a comma.

Save the configuration

Click Save.

Opt to enable Snowflake tag ingestion

To allow Immuta to automatically import table and column tags from Snowflake, enable Snowflake tag ingestion in the external catalog section of the Immuta app settings page.

1. Navigate to the App Settings page.

2. Scroll to 2 External Catalogs, and click Add Catalog.

3. Enter a Display Name and select Snowflake from the dropdown menu.

4. Enter the Account.

5. Enter the Authentication information: Username, Password, Port, Default Warehouse, and Role.

6. Opt to enter the Proxy Host, Proxy Port, and Encrypted Key File Passphrase.

7. Opt to Upload Certificates.

8. Click the Test Connection button.

9. Click the Test Data Source Link.

10. Once both tests are successful, click Save.

Register data

Register Snowflake data in Immuta.

The Snowflake low row access policy mode improves query performance in Immuta's Snowflake integration by decreasing the number of Snowflake row access policies Immuta creates and by using table grants to manage user access.

Immuta manages access to Snowflake tables by administering Snowflake row access policies and column masking policies on those tables, allowing users to query them directly in Snowflake while policies are enforced.

Without Snowflake low row access policy mode enabled, row access policies are created and administered by Immuta in the following scenarios:

• Table grants are disabled and a subscription policy that does not automatically subscribe everyone to the data source is applied. Immuta administers Snowflake row access policies to filter out all the rows to restrict access to the entire table when the user doesn't have privileges to query it. However, if table grants are disabled and a subscription policy is applied that grants everyone access to the data source automatically, Immuta does not create a row access policy in Snowflake. See the subscription policies page for details about these policy types.

• Purpose-based policy is applied to a data source. A row access policy filters out all the rows of the table if users aren't acting under the purpose specified in the policy when they query the table.

• Row-level security policy is applied to a data source. A row access policy filters out rows querying users don't have access to.

• User impersonation is enabled. A row access policy is created for every Snowflake table registered in Immuta.

Reducing row access policies

Snowflake low row access policy mode is enabled by default to reduce the number of row access policies Immuta creates and improve query performance. Snowflake low row access policy mode requires

• table grants to be enabled.

• user impersonation to be disabled. User impersonation diminishes the performance of interactive queries because of the number of row access policies Immuta creates when it's enabled.

Requirements

• Snowflake integration enabled

• Snowflake table grants enabled

Project-scoped purpose exceptions for Snowflake with low row access policy mode enabled

Project-scoped purpose exceptions for Snowflake integrations allow you to apply purpose-based policies to Snowflake data sources in a project. As a result, users can only access that data when they are working within that specific project.

Masked joins for Snowflake with low row access policy mode enabled

This feature allows masked columns to be joined across data sources that belong to the same project. When data sources do not belong to a project, Immuta uses a unique salt per data source for hashing to prevent masked values from being joined. (See the Why use masked joins? guide for an explanation of that behavior.) However, once you add Snowflake data sources to a project and enable masked joins, Immuta uses a consistent salt across all the data sources in that project to allow the join.

For more information about masked joins and enabling them for your project, see the Masked joins section of documentation.

Limitations and considerations

• Project workspaces are not compatible with this feature.

• Impersonation is not supported when the Snowflake low row access policy mode is enabled.

Contact your Immuta representative to enable this feature in your Immuta tenant.

Configure the Snowflake integration

1. Navigate to the App Setting page and click the Integration tab.

2. Click +Add Native Integration and select Snowflake from the dropdown menu.

3. Complete the Host, Port, and Default Warehouse fields.

4. Enable Native Query Audit.

5. Enable Native Lineage and complete the following fields:

   • Ingest Batch Sizes: This setting configures the number of rows Immuta ingests per batch when streaming Access History data from your Snowflake instance.

   • Table Filter: This filter determines which tables Immuta will ingest lineage for. Enter a regular expression that excludes / from the beginning and end to filter tables. Without this filter, Immuta will attempt to ingest lineage for every table on your Snowflake instance.

   • Tag Filter: This filter determines which tags to propagate using lineage. Enter a regular expression that excludes / from the beginning and end to filter tags. Without this filter, Immuta will ingest lineage for every tag on your Snowflake instance.

6. Select Manual or Automatic Setup and follow the steps in this guide to configure the Snowflake integration

Trigger Snowflake lineage sync job

Prerequisite

Authenticate with the Immuta API.

Trigger the lineage job

The Snowflake lineage sync endpoint triggers the native lineage ingestion job that allows Immuta to propagate Snowflake tags added through lineage to Immuta data sources.

1. Copy the example and replace the Immuta URL and API key with your own.

2. Change the payload attribute values to your own, where

   • tableFilter (string): This regular expression determines which tables Immuta will ingest lineage for. Enter a regular expression that excludes / from the beginning and end to filter tables. Without this filter, Immuta will attempt to ingest lineage for every table on your Snowflake instance.

   • batchSize (integer): This parameter configures the number of rows Immuta ingests per batch when streaming Access History data from your Snowflake instance. Minimum 1.

   • lastTimestamp (string): Setting this parameter will only return lineage events later than the value provided. Use a format like 2022-06-29T09:47:06.012-07:00.

   Copy

   curl -X 'POST' \
       'https://www.organization.immuta.com/lineage/ingest/snowflake' \
       -H 'accept: application/json' \
       -H 'Content-Type: application/json' \
       -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f' \
       -d '{
       "tableFilter": "MY_DATABASE\\MY_SCHEMA\\..*",
       "batchSize": 1,
       "lastTimestamp": "2022-06-29T09:47:06.012-07:00"
       }'

Next steps

Once the sync job is complete, you can complete the following steps:

• Register Snowflake data sources

• Build policies

While you're onboarding Snowflake data sources and designing policies, you don't want to disrupt your Snowflake users' existing workflows. Instead, you want to gradually onboard Immuta through a series of successive changes that will not impact your existing Snowflake users.

A phased onboarding approach to configuring the Snowflake integration ensures that your users will not be immediately affected by changes as you add data sources and configure policies.

Several features allow you to gradually onboard data sources and policies in Immuta:

• Subscription policy of “None” by default: By default, no policy is applied at registration time; instead of applying a restrictive policy immediately upon registration, the table is registered in Immuta and waits for a policy to be applied, if ever.

  There are several benefits to this design:

  • All existing roles maintain access to the data and registration of the table or view with Immuta has zero impact on your data platform.

  • It gives you time to configure tags on the Immuta registered tables and views, either manually or through automatic means, such as Immuta’s sensitive data detection (SDD), or an external catalog integration to include Snowflake tags.

  • It gives you time to assess and validate the sensitive data tags that were applied.

  • You can build only row and column controls with Immuta and let your existing roles manage table access instead of using Immuta subscription policies for table access.

• Snowflake table grants coupled with Snowflake low row access policy mode: With these features enabled, Immuta manages access to tables (subscription policies) through GRANTs. This works by assigning each user their own unique role created by Immuta and all table access is managed using that single role.

  Without these two features enabled, Immuta uses a Snowflake row access policy (RAP) to manage table access. A RAP only allows users to access rows in the table if they were explicitly granted access through an Immuta subscription policy; otherwise, the user sees no rows. This behavior means all existing Snowflake roles lose access to the table contents until explicitly granted access through Immuta subscription policies. Essentially, roles outside of Immuta don't control access anymore.

  By using table grants and the low row access policy mode, users and roles outside Immuta continue to work.

  There are two benefits to this approach:

  • All pre-existing Snowflake roles retain access to the data until you explicitly revoke access (outside Immuta).

  • It provides a way to test that Immuta GRANTs are working without impacting production workloads.

Requirements

The following configuration is required for phased Snowflake onboarding:

• Impersonation is disabled

• Project workspaces are disabled

If either of these capabilities is necessary for your use case, you cannot do phased Snowflake onboarding as described below.

Next

See the Getting started page for step-by-step guidance to implement phased Snowflake onboarding.

The warehouse you select when configuring the Snowflake integration uses compute resources to set up the integration, register data sources, orchestrate policies, and run jobs like sensitive data discovery. Snowflake credit charges are based on the size of and amount of time the warehouse is active, not the number of queries run.

This document prescribes how and when to adjust the size and scale of clusters for your warehouse to manage workloads so that you can use Snowflake compute resources the most cost effectively.

In general, increase the size of and number of clusters for the warehouse to handle heavy workloads and multiple queries. Workloads are typically lighter after data sources are onboarded and policies are established in Immuta, so compute resources can be reduced after those workloads complete.

Integration and data source registration warehouse use

The Snowflake integration uses warehouse compute resources to sync policies created in Immuta to the Snowflake objects registered as data sources and, if enabled, to run sensitive data discovery and schema monitoring. Follow the guidelines below to adjust the warehouse size and scale according to your needs.

• Increase the size of and number of clusters for the warehouse during large policy syncs, updates, and changes.

• Enable auto-suspend and auto-resume to optimize resource use in Snowflake. In the Snowflake UI, the lowest auto suspend time setting is 5 minutes. However, through SQL query, you can set auto_suspend to 61 seconds (since the minimum uptime for a warehouse is 60 seconds). For example,

  Copy

  ALTER WAREHOUSE "WH_NAME" SET WAREHOUSE_SIZE = 'XSMALL' AUTO_SUSPEND = 61 AUTO_RESUME = TRUE MIN_CLUSTER_COUNT = 1 MAX_CLUSTER_COUNT = 2 SCALING_POLICY = 'STANDARD' COMMENT = '';

• Sensitive data discovery uses compute resources for each table registered if it is enabled. Consider disabling sensitive data discovery when registering data sources if you have an external catalog available or a tagging strategy in place.

• Register data before creating global policies. By default, Immuta does not apply a subscription policy on registered data (unless an existing global policy applies to it), which allows Immuta to only pull metadata instead of also applying policies when data sources are created. Registering data before policies are created reduces the workload and the Snowflake compute resources needed.

• Begin onboarding with a small dataset of tables, and then review and monitor query performance in the Snowflake Query Monitor. Adjust the virtual warehouse accordingly to handle heavier loads.

• Schema monitoring uses the compute warehouse that was employed during the initial ingestion to periodically monitor the schema for changes. If you expect a low number of new tables or minimal changes to the table structure, consider scaling down the warehouse size.

• Resize the warehouse after after data sources are registered and policies are established. For example,

ALTER WAREHOUSE "INTEGRATION_WH" SET WAREHOUSE_SIZE = 'XSMALL' AUTO_SUSPEND = 120 AUTO_RESUME = TRUE MIN_CLUSTER_COUNT = 1 MAX_CLUSTER_COUNT = 2 SCALING_POLICY = 'STANDARD'; 

For more details and guidance about warehouse sizing, see the Snowflake Warehouse Considerations documentation.

Identifying bulk jobs and heavy workloads

Even after your integration is configured, data sources are registered, and policies are established, changes to those data sources or policies may initiate heavy workloads. Follow the guidelines below to adjust your warehouse size and scale according to your needs.

• Review your Snowflake query history to identify query performance and bottlenecks.

• Check how many credits queries have consumed:

  Copy

  SELECT h.* FROM "SNOWFLAKE"."ACCOUNT_USAGE"."QUERY_HISTORY" h
  INNER JOIN "SNOWFLAKE"."ACCOUNT_USAGE"."SESSIONS" s
  ON s.session_id = h.session_id
  WHERE GET(parse_json(s.client_environment), 'APPLICATION') = 'IMMUTA' limit 25;

• After reviewing query performance and cost, implement strategies above to adjust your warehouse.

If a Databricks cluster needs to be manually updated to reflect changes in the Immuta init script or cluster policies, you can remove and set up your integration again to get the updated policies and init script.

1. Log in to Immuta as an Application Admin.

2. Click the App Settings icon in the left sidebar and scroll to the Integration Settings section.

3. Your existing Databricks Spark integration should be listed here; expand it and note the configuration values. Now select Remove to remove your integration.

4. Click Add Native Integration and select Databricks Integration to add a new integration.

5. Enter your Databricks Spark integration settings again as configured previously.

6. Click Add Native Integration to add the integration, and then select Configure Cluster Policies to set up the updated cluster policies and init script.

7. Select the cluster policies you wish to use for your Immuta-enabled Databricks clusters.

8. Automatically push cluster policies and the init script (recommended) or manually update your cluster policies.

   • Automatically push cluster policies

     1. Select Automatically Push Cluster Policies and enter your privileged Databricks access token. This token must have privileges to write to cluster policies.

     2. Select Apply Policies to push the cluster policies and init script again.

     3. Click Save and Confirm to deploy your changes.

   • Manually update cluster policies

     1. Download the init script and the new cluster policies to your local computer.

     2. Click Save and Confirm to save your changes in Immuta.

     3. Log in to your Databricks workspace with your administrator account to set up cluster policies.

     4. Get the path you will upload the init script (immuta_cluster_init_script_proxy.sh) to by opening one of the cluster policy .json files and looking for the defaultValue of the field init_scripts.0.dbfs.destination. This should be a DBFS path in the form of dbfs:/immuta-plugin/hostname/immuta_cluster_init_script_proxy.sh.

     5. Click Data in the left pane to upload your init script to DBFS to the path you found above.

     6. To find your existing cluster policies you need to update, click Compute in the left pane and select the Cluster policies tab.

     7. Edit each of these cluster policies that were configured before and overwrite the contents of the JSON with the new cluster policy JSON you downloaded.

9. Restart any Databricks clusters using these updated policies for the changes to take effect.

Snowflake column lineage specifies how data flows from source tables or columns to the target tables in write operations. When Snowflake lineage tag propagation is enabled in Immuta, Immuta automatically applies tags added to a Snowflake table to its descendant data source columns in Immuta so you can build policies using those tags to restrict access to sensitive data.

Snowflake Access History tracks user read and write operations. Snowflake column lineage extends this Access History to specify how data flows from source columns to the target columns in write operations, allowing data stewards to understand how sensitive data moves from ancestor tables to target tables so that they can

• trace data back to its source to validate the integrity of dashboards and reports,

• identify who performed write operations to meet compliance requirements,

• evaluate data quality and pinpoint points of failure, and

• tag sensitive data on source tables without having tag columns on their descendant tables.

However, tagging sensitive data doesn’t innately protect that data in Snowflake; users need Immuta to disseminate these lineage tags automatically to descendant tables registered in Immuta so data stewards can build policies using the semantic and business context captured by those tags to restrict access to sensitive data. When Snowflake lineage tag propagation is enabled, Immuta propagates tags applied to a data source to its descendant data source columns in Immuta, which keeps your data inventory in Immuta up-to-date and allows you to protect your data with policies without having to manually tag every new Snowflake data source you register in Immuta.

Data flow

1. An application administrator enables the feature on the Immuta app settings page.

2. Snowflake lineage metadata (column names and tags) for the Snowflake tables is stored in the metadata database.

3. A data owner creates a new data source (or adds a new column to a Snowflake table) that initiates a job that applies all tags for each column from its ancestor columns.

4. A data owner or governor adds a tag to a column in Immuta that has descendants, which initiates a job that propagates the tag to all descendants.

5. An audit record is created that includes which tags were applied and from which columns those tags originated.

Snowflake access history view and Immuta lineage job

The Snowflake Account Usage ACCESS_HISTORY view contains column lineage information.

To appropriately propagate tags to descendant data sources, Immuta fetches Access History metadata to determine what column tags have been updated, stores this metadata in the Immuta metadata database, and then applies those tags to relevant descendant columns of tables registered in Immuta.

Consider the following example using the Customer, Customer 2, and Customer 3 tables that were all registered in Immuta as data sources.

• Customer: source table

• Customer 2: descendant of Customer

• Customer 3: descendant of Customer 2

If the Discovered.Electronic Mail Address tag is added to the Customer data source in Immuta, that tag will propagate through lineage to the Customer 2 and Customer 3 data sources.

Data source registration

After an application administrator has enabled Snowflake lineage tag propagation, data owners can register data in Immuta and have tags in Snowflake propagated from ancestor tables to descendant data sources. Whenever new tags are added to those tables in Immuta, those upstream tags will propagate to descendant data sources.

By default all tags are propagated, but these tags can be filtered on the app settings page or using the Immuta API.

Managing tags

Lineage tag propagation works with any tag added to the data dictionary. Tags can be manually added, synced from an external catalog, or discovered by SDD. Consider the following example using the Customer, Customer 2, and Customer 3 tables that were all registered in Immuta as data sources.

• Customer: source table

• Customer 2: descendant of Customer

• Customer 3: descendant of Customer 2

Immuta added the Discovered.Electronic Mail Address tag to the Customer data source, and that tag propagated through lineage to the Customer 2 and Customer 3 data sources.

Removing the tag from the Customer 2 table soft deletes it from the Customer 2 data source. When a tag is deleted, downstream lineage tags are removed, unless another parent data source still has that tag. The tag remains visible, but it will not be re-added if a future propagation event specifies the same tag again. Immuta prevents you from removing Snowflake object tags from data sources. You can only remove Immuta-managed tags. To remove Snowflake object tags from tables, you must remove them in Snowflake.

However the Discovered.Electronic Mail Address tag still applies to the Customer 3 data source because Customer still has the tag applied. The only way a tag will be removed from descendant data sources is if no other ancestor of the descendant still prescribes the tag.

If the Snowflake lineage tag propagation feature is disabled, tags will remain on Immuta data sources.

Sensitive data discovery

Sensitive data discovery will still run on data sources and can be manually triggered. Tags applied through sensitive data discovery will propagate as tags added through lineage to descendant Immuta data sources.

Snowflake lineage audit

Immuta audit records include Snowflake lineage tag events when a tag is added or removed.

The example audit record below illustrates the SNOWFLAKE_TAGS.pii tag successfully propagating from the Customer table to Customer 2:

{
  "id": "c8e020cb-232c-4ba9-a0d8-f3a84ba6808d",
  "dateTime": "1670355170336",
  "month": 1475,
  "profileId": 1,
  "userId": "immuta_system_account",
  "dataSourceId": 2,
  "dataSourceName": "Customer 2",
  "count": 1,
  "recordType": "nativeLineageDataSourceTagUpdate",
  "success": true,
  "component": "dataSource",
  "extra": {
    "sourceColumn": {
      "nativeColumnName": "\"MY_DATABASE\".\"PUBLIC\".\"CUSTOMER\".\"C_FIRST_NAME\"",
      "dataSourceId": 1,
      "columnName": "c_first_name"
    },
    "dataSourceId": 2,
    "columnName": "c_first_name",
    "tagPropagationDirection": "downstream",
    "tags": [
      {
        "name": "SNOWFLAKE_TAGS.pii",
        "source": "immuta-us-east-1"
      }
    ]
  },
  "newAuditServiceFields": {
    "actorIp": null,
    "sessionId": null
  },
  "createdAt": "2022-12-06T19:32:50.372Z",
  "updatedAt": "2022-12-06T19:32:50.372Z"
}

Limitations

• Without tableFilter set, Immuta will ingest lineage for every table on the Snowflake instance.

• Tag propagation based on lineage is not retroactive. For example, if you add a table, add tags to that table, and then run the lineage ingestion job, tags will not get propagated. However, if you add a table, run the lineage ingestion job, and then add tags to the table, the tags will get propagated.

• The native lineage job needs to pull in lineage data before any tag is applied in Immuta. When Immuta gets new lineage information from Snowflake, Immuta does not update existing tags in Immuta.

• There can be up to a 3-hour delay in Snowflake for a lineage event to make it into the ACCESS_HISTORY view.

• Immuta does not ingest lineage information for views.

• Snowflake only captures lineage events for CTAS, CLONE, MERGE, and INSERT write operations. Snowflake does not capture lineage events for DROP, RENAME, ADD, or SWAP. Instead of using these latter operations, you need to recreate a table with the same name if you need to make changes.

• Immuta cannot enforce coherence of your Snowflake lineage. If a column, table, or schema in the middle of the lineage graph gets dropped, Immuta will not do anything unless a table with that same name gets recreated. This means a table that gets dropped but not recreated could live in Immuta’s system indefinitely.

Snowflake table grants simplifies the management of privileges in Snowflake when using Immuta. Instead of having to manually grant users access to tables registered in Immuta, you allow Immuta to manage privileges on your Snowflake tables and views according to subscription policies. Then, users subscribed to a data source in Immuta can view and query the Snowflake table, while users who are not subscribed to the data source cannot view or query the Snowflake table.

Snowflake privileges

Enabling Snowflake table grants gives the following privileges to the Immuta Snowflake role:

• MANAGE GRANTS ON ACCOUNT allows the Immuta Snowflake role to grant and revoke SELECT privileges on Snowflake tables and views that have been added as data sources in Immuta.

• CREATE ROLE ON ACCOUNT allows for the creation of a Snowflake role for each user in Immuta, enabling fine-grained, attribute-based access controls to determine which tables are available to which individuals.

Table grants role

Since table privileges are granted to roles and not to users in Snowflake, Immuta's Snowflake table grants feature creates a new Snowflake role for each Immuta user. This design allows Immuta to manage table grants through fine-grained access controls that consider the individual attributes of users.

Each Snowflake user with an Immuta account will be granted a role that Immuta manages. The naming convention for this role is <IMMUTA>_USER_<username>, where

• <IMMUTA> is the prefix you specified when enabling the feature on the Immuta app settings page.

• <username> is the user's Immuta username.

Querying Snowflake tables managed by Immuta

Users are granted access to each Snowflake table or view automatically when they are subscribed to the corresponding data source in Immuta.

Users have two options for querying Snowflake tables that are managed by Immuta:

• Use the role that Immuta creates and manages. (For example, USE ROLE IMMUTA_USER_<username>. See the section above for details about the role and name conventions.) If the current active primary role is used to query tables, USAGE on a Snowflake warehouse must be granted to the Immuta-managed Snowflake role for each user.

• USE SECONDARY ROLES ALL, which allows users to use the privileges from all roles that they have been granted, including IMMUTA_USER_<username>, in addition to the current active primary role. Users may also set a value for DEFAULT_SECONDARY_ROLES as an object property on a Snowflake user. To learn more about primary roles and secondary roles in Snowflake, see Snowflake documentation.

Applying GRANTs and REVOKEs at scale

Immuta uses an algorithm to determine the most optimal way to group users in a role hierarchy in order to optimize the number of GRANTs (or REVOKES) executed in Snowflake. This is done by determining the least amount of possible permutations of access across tables and users based on the policies in place; then, those become intermediate roles in the hierarchy that each user is added to, based on the intermediate roles they belong to.

As an example, take the below users and data sources they have access to. To do this naively by individually granting every user to the tables they have access to would result in 37 grants:

Conversely, using the Immuta algorithm, we can optimize the number of grants in the same scenario down to 29:

It’s important to consider a few things here:

1. If the permutations of access are small, there will be a huge optimization realized (very few intermediate roles). If every user has their own unique permutation of access, the optimization will be negligible (an intermediate role per user). It is most common that the number of permutations of access will be many multiples smaller than the actual user count, so there should be large optimizations. In other words, a much smaller number of intermediate roles and the number of total overall grants reduced, since the tables are granted to roles and roles to users.

2. This only happens once up front. After that, changes are incremental based on policy changes and user attribute changes (smaller updates), unless there’s a policy that makes a sweeping change across all users. The addition of new users who have access becomes much more straightforward also due to the fact above. User’s access will be granted via the intermediate role, and, therefore, a lot of the work is front loaded in the intermediate role creation.

Limitations

• Project workspaces are not supported when Snowflake table grants is enabled.

• If an Immuta tenant is connected to an external IAM and that external IAM has a username identical to another username in Immuta's built-in IAM, those users will have the same Snowflake role, leading both to see the same data.

• Sometimes the role generated can contain special characters such as @ because it's based on the user name configured from your identity manager. Because of this, it is recommended that any code references to the Immuta-generated role be enclosed with double quotes.

This integration allows you to manage and access data in your Databricks account across all of your workspaces. With Immuta’s Databricks Unity Catalog integration, you can write your policies in Immuta and have them enforced automatically by Databricks across data in your Unity Catalog metastore.

This getting started guide outlines how to integrate Databricks Unity Catalog with Immuta.

How-to guides

• : Configure the Databricks Unity Catalog integration.

• : Migrate from the legacy Databricks Spark integrations to the Databricks Unity Catalog integration.

Reference guides

• : This guide describes the design and components of the integration.

• : This reference guide provides descriptions of the possible statuses of a configured integration.

The how-to guides linked on this page illustrate how to integrate Databricks Unity Catalog with Immuta.

Requirements:

• Unity Catalog and attached to a Databricks workspace. Immuta supports configuring a single metastore for each configured integration, and that metastore may be attached to multiple Databricks workspaces.

• Unity Catalog enabled on your Databricks cluster or SQL warehouse. All SQL warehouses have Unity Catalog enabled if your workspace is attached to a Unity Catalog metastore.

Configure your Databricks Unity Catalog integration

These guides provide information on the recommended features to enable with Databricks Unity Catalog.

1. with the following feature enabled: (enabled by default)

2. Select None as your .

3. .

4. .

Register metadata

These guides provide instructions for organizing your Databricks Unity Catalog data to align with your governance structure.

These guides provide instructions for discovering, classifying, and tagging your data.

3. Validate the policies. You do not have to validate every policy you create in Immuta; instead, examine a few to validate the behavior you expect to see.

4. Once all Immuta policies are in place, remove or alter old permissions and revoke access to the ungoverned tables.

allows you to manage and access data in your Databricks account across all of your workspaces. With Immuta’s Databricks Unity Catalog integration, you can write your policies in Immuta and have them enforced automatically by Databricks across data in your Unity Catalog metastore.

This page details how to configure the integration. To configure the Databricks Unity Catalog integration and register data sources using the , see this .

Permissions

Several different accounts are used to set up and maintain the Databricks Unity Catalog integration. The permissions required for each are outlined below.

• Immuta account (required): This user configures the integration on the app settings page in Immuta. To access the app settings page, this user needs the following permission:

  • APPLICATION_ADMIN Immuta permission

• Databricks service principal (required): This service principal is used continuously by Immuta to orchestrate Unity Catalog policies and maintain state between Immuta and Databricks. In the , Immuta also uses this service principal to create the Immuta-managed catalog. This service principal needs the following Databricks privileges:

  • CREATE CATALOG privilege on the Unity Catalog metastore. This is only required if you have Immuta the integration in Databricks for you. If a separate user will run the Immuta script in Databricks to manually configure the integration, that Databricks user account needs this privilege instead.

  • OWNER permission on the Immuta catalog you configure.

  • OWNER privilege on one of the securables below so that Immuta can administer Unity Catalog row-level and column-level security controls.

    • on catalogs with schemas and tables registered as Immuta data sources. This permission could also be applied by granting OWNER on a catalog to a Databricks group that includes the Immuta service principal to allow for multiple owners.

    • on schemas with tables registered as Immuta data sources.

    • on all tables registered as Immuta data sources - if the OWNER permission cannot be applied at the catalog- or schema-level. In this case, each table registered as an Immuta data source must individually have the OWNER permission granted to the Immuta service principal.

  • USE CATALOG and USE SCHEMA on parent catalogs and schemas of tables registered as Immuta data sources so that the Immuta service principal can SELECT and MODIFY securables within the parent catalog and schema.

  • SELECT and MODIFY on all tables registered as Immuta data sources so that the Immuta service principal can grant and revoke access to tables and apply Unity Catalog row- and column-level security controls.

  • For native query audit (optional)

    • USE CATALOG on the system catalog

    • USE SCHEMA on the system.access schema

    • SELECT on the following system tables:

      • system.access.audit

      • system.access.table_lineage

      • system.access.column_lineage

• Databricks account (recommended): This user account can manually configure the integration in Databricks to create the Immuta-managed catalog. To do so, this account requires the following Databricks privileges:

  • CREATE CATALOG on the Unity Catalog metastore

  • ACCOUNT ADMIN on the Unity Catalog metastore for native query audit (optional)

Requirements

Before you configure the Databricks Unity Catalog integration, ensure that you have fulfilled the following requirements:

• Unity Catalog enabled on your Databricks cluster or SQL warehouse. All SQL warehouses have Unity Catalog enabled if your workspace is attached to a Unity Catalog metastore. Immuta recommends linking a SQL warehouse to your Immuta tenant rather than a cluster for both performance and availability reasons.

Migrate data to Unity Catalog

1. Ensure that all Databricks clusters that have Immuta installed are stopped and the Immuta configuration is removed from the cluster. Immuta-specific cluster configuration is no longer needed with the Databricks Unity Catalog integration.

Create the Databricks service principal

Opt to enable native query audit for Unity Catalog

3. • USE CATALOG on the system catalog

   • USE SCHEMA on the system.access schema

   • SELECT on the following system tables:

     • system.access.audit

     • system.access.table_lineage

     • system.access.column_lineage

Configure the Databricks Unity Catalog integration

You have two options for configuring your Databricks Unity Catalog integration:

Automatic setup

1. Click the App Settings icon in the left sidebar.

2. Click the Integrations tab.

3. Click + Add Native Integration and select Databricks Unity Catalog from the dropdown menu.

4. Complete the following fields:

   • Server Hostname is the hostname of your Databricks workspace.

   • HTTP Path is the HTTP path of your Databricks cluster or SQL warehouse.

   • Immuta Catalog is the name of the catalog Immuta will create to store internal entitlements and other user data specific to Immuta. This catalog will only be readable for the Immuta service principal and should not be granted to other users. The catalog name may only contain letters, numbers, and underscores and cannot start with a number.

5. If using a proxy server with Databricks Unity Catalog, click the Enable Proxy Support checkbox and complete the Proxy Host and Proxy Port fields. The username and password fields are optional.

6. Opt to fill out the Exemption Group field with the name of a group in Databricks that will be excluded from having data policies applied and must not be changed from the default value. Create this account-level group for privileged users and service accounts that require an unmasked view of data before configuring the integration in Immuta.

7. Opt to scope the query audit ingestion by entering in Unity Catalog Workspace IDs. Enter a comma-separated list of the workspace IDs that you want Immuta to ingest audit records for. If left empty, Immuta will audit all tables and users in Unity Catalog.

8. 2. Enter how often, in hours, you want Immuta to ingest audit events from Unity Catalog as an integer between 1 and 24.

   3. Continue with your integration configuration.

9. Select your authentication method from the dropdown:

   • OAuth machine-to-machine (M2M):

     • AWS Databricks:

       • Fill out the Token Endpoint with the full URL of the identity provider. This is where the generated token is sent. The default value is https://<your workspace name>.cloud.databricks.com/oidc/v1/token.

       • Enter the Client Secret you created above. Immuta uses this secret to authenticate with the authorization server when it requests a token.

     • Azure Databricks:

       • Within Immuta, fill out the Token Endpoint with the full URL of the identity provider. This is where the generated token is sent. The default value is https://<your workspace name>.azuredatabricks.net/oidc/v1/token.

       • Enter the Client Secret you created above. Immuta uses this secret to authenticate with the authorization server when it requests a token.

10. Click Save.

Manual setup

1. Click the App Settings icon in the left sidebar.

2. Click the Integrations tab.

3. Click + Add Native Integration and select Databricks Unity Catalog from the dropdown menu.

4. Complete the following fields:

   • Server Hostname is the hostname of your Databricks workspace.

   • HTTP Path is the HTTP path of your Databricks cluster or SQL warehouse.

   • Immuta Catalog is the name of the catalog Immuta will create to store internal entitlements and other user data specific to Immuta. This catalog will only be readable for the Immuta service principal and should not be granted to other users. The catalog name may only contain letters, numbers, and underscores and cannot start with a number.

5. If using a proxy server with Databricks Unity Catalog, click the Enable Proxy Support checkbox and complete the Proxy Host and Proxy Port fields. The username and password fields are optional.

6. Opt to fill out the Exemption Group field with the name of a group in Databricks that will be excluded from having data policies applied and must not be changed from the default value. Create this account-level group for privileged users and service accounts that require an unmasked view of data before configuring the integration in Immuta.

7. Opt to scope the query audit ingestion by entering in Unity Catalog Workspace IDs. Enter a comma-separated list of the workspace IDs that you want Immuta to ingest audit records for. If left empty, Immuta will audit all tables and users in Unity Catalog.

8. 2. Enter how often, in hours, you want Immuta to ingest audit events from Unity Catalog as an integer between 1 and 24.

   3. Continue with your integration configuration.

9. Select your authentication method from the dropdown:

   • OAuth machine-to-machine (M2M):

     • AWS Databricks:

       • Fill out the Token Endpoint with the full URL of the identity provider. This is where the generated token is sent. The default value is https://<your workspace name>.cloud.databricks.com/oidc/v1/token.

       • Enter the Client Secret you created above. Immuta uses this secret to authenticate with the authorization server when it requests a token.

     • Azure Databricks:

       • Within Immuta, fill out the Token Endpoint with the full URL of the identity provider. This is where the generated token is sent. The default value is https://<your workspace name>.azuredatabricks.net/oidc/v1/token.

       • Enter the Client Secret you created above. Immuta uses this secret to authenticate with the authorization server when it requests a token.

10. Select the Manual toggle and copy or download the script. You can modify the script to customize your storage location for tables, schemas, or catalogs.

11. Run the script in Databricks.

12. Click Save.

Map Databricks users to Immuta

If the usernames in Immuta do not match usernames in Databricks, map each Databricks username to each Immuta user account to ensure Immuta properly enforces policies using one of the methods linked below:

Opt to enable Databricks Unity Catalog tag ingestion

Requirement: A Databricks Unity Catalog integration must be configured for tags to be ingested.

To allow Immuta to automatically import table and column tags from Databricks Unity Catalog, enable Databricks Unity Catalog tag ingestion in the external catalog section of the Immuta app settings page.

1. Navigate to the App Settings page.

2. Scroll to 2 External Catalogs, and click Add Catalog.

3. Enter a Display Name and select Databricks Unity Catalog from the dropdown menu.

4. Click Save and confirm your changes.

Register data

This integration enforces policies on Databricks tables registered as data sources in Immuta, allowing users to query policy-enforced data on Databricks clusters (including job clusters). Immuta policies are applied to the plan that Spark builds for users' queries, all executed directly against Databricks tables.

The guides in this section outline how to integrate Databricks Spark with Immuta.

How-to guides

• : Configure the Databricks Spark integration.

• : Access DBFS in Databricks for non-sensitive data.

• : Allow Immuta users to access tables that are not protected by Immuta.

• : Hide the Immuta database from users in Databricks, since user queries do not need to reference it.

• : Run R and Scala spark-submit jobs on your Databricks cluster.

• : Raise the caching on-cluster and lower the cache timeouts for the Immuta web service to allow use of project UDFs in Spark jobs.

• : Use an existing Hive external metastore instead of the built-in metastore.

Reference guides

• : This guide describes the design and components of the integration.

• : This reference guide provides descriptions of the possible statuses of a configured integration.

• Configuration settings: These guides describe various integration settings that can be configured, including , cluster policies, and .

• : This guide describes Immuta's support of Databricks change data feed.

• : The trusted libraries feature allows Databricks cluster administrators to avoid Immuta security manager errors when using third-party libraries. This guide describes the feature and its configuration.

• : When using Delta Lake, the API does not go through the normal Spark execution path. This means that Immuta's Spark extensions do not provide protection for the API. To solve this issue and ensure that Immuta has control over what a user can access, the Delta Lake API is blocked. This reference guide outlines the Spark SQL options that can be substituted for the Delta Lake API.

• : Immuta allows direct file reads in Spark for file paths. This guide describes that process.

This guide details the manual installation method for enabling native access to Databricks with Immuta policies enforced. Before proceeding, ensure your Databricks workspace, instance, and permissions meet the guidelines outlined in the .

Requirements

• If Databricks Unity Catalog is enabled in a Databricks workspace, you must use an Immuta cluster policy when you set up the Databricks Spark integration to create an Immuta-enabled cluster. .

• If Databricks Unity Catalog is not enabled in your Databricks workspace, you must disable Unity Catalog in your Immuta tenant before proceeding with your configuration of Databricks Spark:

  1. Navigate to the App Settings page and click Integration Settings.

  2. Uncheck the Enable Unity Catalog checkbox.

  3. Click Save.

1 - Download and Configure Immuta Artifacts

1. Navigate to the page. If you are prompted to log in and need basic authentication credentials, contact your Immuta support professional.

2. Navigate to the Databricks folder for the newest Immuta version. Ex: https://archives.immuta.com/hadoop/databricks/2024.2.1/.

3. Download the .jar file (Immuta plugin) as well as the other scripts listed below, which will load the plugin at cluster startup.

   The immuta-benchmark-suite.dbc is a collection of notebooks packaged as a .dbc file. After you have added cluster policies to your cluster, you can import this file into Databricks to run performance tests and compare a regular Databricks cluster to one protected by Immuta. Detailed instructions are available in the first notebook, which will require an Immuta and non-Immuta cluster to generate test data and perform queries.

4. Specify the following properties as Spark environment variables or in the optional immuta_conf.xml file. If the same property is specified in both locations, the Spark environment variable takes precedence. The variable names are the config names in all upper case with _ instead of .. For example, to set the value of immuta.base.url via an environment variable, you would set the following in the Environment Variables section of cluster configuration: IMMUTA_BASE_URL=https://immuta.mycompany.com

   • immuta.system.api.key: Obtain this value from the under HDFS > System API Key. You will need to be a user with the APPLICATION_ADMIN role to complete this action.

   • immuta.base.url: The full URL for the target Immuta tenant Ex: https://immuta.mycompany.com.

   • immuta.user.mapping.iamid: If users authenticate to Immuta using an IAM different from Immuta's built-in IAM, you need to update the configuration file to reflect the ID of that IAM. The IAM ID is shown within the Immuta App Settings page within the Identity Management section. See Databricks to Immuta User Mapping for more details.

2 - Stage Immuta Artifacts

When configuring the Databricks cluster, a path will need to be provided to each of the artifacts downloaded/created in the previous step. To do this, those artifacts must be hosted somewhere that your Databricks instance can access. The following methods can be used for this step:

These artifacts will be downloaded to the required location within the clusters file-system by the init script downloaded in the previous step. In order for the init script to find these files, a URI will have to be provided through environment variables configured on the cluster. Each method's URI structure and setup is explained below.

AWS/S3

URI Structure: s3://[bucket]/[path]

2. Upload the configuration file, JSON file, and JAR file to an S3 bucket that the role from step 1 has access to.

Authenticating with Access Keys or Session Tokens (Optional)

If you wish to authenticate using access keys, add the following items to the cluster's environment variables:

If you've assumed a role and received a session token, that can be added here as well:

Azure

ADL Gen 2

URI Structure: abfs(s)://[container]@[account].dfs.core.windows.net/[path]

Environment Variables:

If you want to authenticate using an account key, add the following to your cluster's environment variables:

If you want to authenticate using an Azure SAS token, add the following to your cluster's environment variables:

ADL Gen 1

URI Structure: adl://[account].azuredatalakestore.net/[path]

Environment Variables:

If authenticating as a Microsoft Entra ID user,

If authenticating using a service principal,

HTTPS

URI Structure: http(s)://[host](:port)/[path]

Artifacts are available for download from Immuta using basic authentication. Your basic authentication credentials can be obtained from your Immuta support professional.

Environment Variables (Optional)

DBFS

URI Structure: dbfs:/[path]

Since any user has access to everything in DBFS:

1. The artifacts can be stored anywhere in DBFS.

2. It's best to have a cluster-specific place for your artifacts in DBFS if you are testing to avoid overwriting or reusing someone else's artifacts accidentally.

3 - Protect Immuta Environment Variables with Databricks Secrets

Databricks secrets can be used in the Environment Variables configuration section for a cluster by referencing the secret path rather than the actual value of the environment variable. For example, if a user wanted to make the following value secret

they could instead create a Databricks secret and reference it as the value of that variable. For instance, if the secret scope my_secrets was created, and the user added a secret with the key my_secret_env_var containing the desired sensitive environment variable, they would reference it in the Environment Variables section:

Then, at runtime, {{secrets/my_secrets/my_secret_env_var}} would be replaced with the actual value of the secret if the owner of the cluster has access to that secret.

4 - Create and Configure the Cluster

Cluster creation in an Immuta-enabled organization or Databricks workspace should be limited to administrative users to avoid allowing users to create non-Immuta enabled clusters.

2. Select the Custom Access mode.

3. Opt to adjust the Autopilot Options and Worker Type settings. The default values provided here may be more than what is necessary for non-production or smaller use-cases. To reduce resource usage you can enable/disable autoscaling, limit the size and number of workers, and set the inactivity timeout to a lower value.

4. In the Advanced Options section, click the Instances tab.
5. Click the Spark tab. In Spark Config field, add your configuration.

   • Cluster Configuration Requirements:
7. Click the Init Scripts tab and set the following configurations:

   • Destination: Specify the service you used to host the Immuta artifacts.

   • File Path: Specify the full URI to the immuta_cluster_init_script.sh.

   • Add the new key/value to the configuration.

8. Click the Permissions tab and configure the following setting:

   • Who has access: Users or groups will need to have the permission Can Attach To to execute queries against Immuta configured data sources.

9. (Re)start the cluster.

Additional Hadoop Configuration File (Optional)

As mentioned in the "Environment Variables" section of the cluster configuration, there may be some cases where it is necessary to add sensitive configuration to SparkSession.sparkContext.hadoopConfiguration in order to read the data composing Immuta data sources.

As an example, when accessing external tables stored in Azure Data Lake Gen 2, Spark must have credentials to access the target containers/filesystems in ADLg2, but users must not have access to those credentials. In this case, an additional configuration file may be provided with a storage account key that the cluster may use to access ADLg2.

The additional configuration file looks very similar to the Immuta Configuration file referenced above. Some example configuration files for accessing different storage layers are below.

Amazon S3

Azure Data Lake Gen 2

Azure Data Lake Gen 1

Azure Blob Storage

5 - Register data

6 - Query Immuta Data

When the Immuta enabled Databricks cluster has been successfully started, users will see a new database labeled "immuta". This database is the virtual layer provided to access data sources configured within the connected Immuta tenant.

Before users can query an Immuta data source, an administrator must give the user Can Attach To permissions on the cluster and GRANT the user access to the immuta database.

The following SQL query can be run as an administrator within a journal to give the user access to "Immuta":

Creating a Databricks Data Source

Databricks to Immuta User Mapping

By default, the IAM used to map users between Databricks and Immuta is the BIM (Immuta's internal IAM). The Immuta Spark plugin will check the Databricks username against the username within the BIM to determine access. For a basic integration, this means the users email address in Databricks and the connected Immuta tenant must match.

Prerequisites

• Databricks instance: Premium tier workspace and

• Databricks instance has network level access to Immuta tenant

• Databricks instance is either publicly accessible or has been configured for .

• Access to

• Permissions and access to download (outside Internet access) or transfer files to the host machine

Recommended Databricks Workspace Configurations:

Note: Azure Databricks authenticates users with Microsoft Entra ID. Be sure to configure your Immuta tenant with an IAM that uses the same user ID as does Microsoft Entra ID. Immuta's Spark security plugin will look to match this user ID between the two systems. See this for details.

Supported Databricks Runtime Versions

Use the table below to determine which version of Immuta supports your Databricks Runtime version:

Supported Databricks Cluster Configurations

The table below outlines the integrations supported for various Databricks cluster configurations. For example, the only integration available to enforce policies on a cluster configured to run on Databricks Runtime 9.1 is the Databricks Spark integration.

Legend:

Supported Access Mode and Languages

Immuta supports the Custom access mode.

• Supported Languages:

  • Python

  • SQL

  • R (requires advanced configuration; work with your Immuta support professional to use R)

  • Scala (requires advanced configuration; work with your Immuta support professional to use Scala)

Databricks Installation Overview

The Immuta Databricks Spark integration injects an Immuta plugin into the SparkSQL stack at cluster startup. The Immuta plugin creates an "immuta" database that is available for querying and intercepts all queries executed against it. For these queries, policy determinations will be obtained from the connected Immuta tenant and applied before returning the results to the user.

The Databricks cluster init script provided by Immuta downloads the Immuta artifacts onto the target cluster and puts them in the appropriate locations on local disk for use by Spark. Once the init script runs, the Spark application running on the Databricks cluster will have the appropriate artifacts on its CLASSPATH to use Immuta for policy enforcement.

The cluster init script uses environment variables in order to

• Determine the location of the required artifacts for downloading.

• Authenticate with the service/storage containing the artifacts.

Note: Each target system/storage layer (HTTPS, for example) can only have one set of environment variables, so the cluster init script assumes that any artifact retrieved from that system uses the same environment variables.

Limitations

Installation Methods

There are two installation options for Databricks. Click a link below to navigate to a tutorial for your chosen method:

• 1. Adding the integration on the App Settings page.

  2. Downloading or automatically pushing cluster policies to your Databricks workspace.

  3. Creating or restarting your cluster.

• 1. Downloading and configuring Immuta artifacts.

  2. Staging Immuta artifacts somewhere the cluster can read from during its startup procedures.

  3. Protecting Immuta environment variables with Databricks Secrets.

  4. Creating and configuring the cluster to start with the init script and load Immuta into its SparkSQL environment.

Debugging Immuta Installation Issues

For easier debugging of the Immuta Databricks installation, enable cluster init script logging. In the cluster page in Databricks for the target cluster, under Advanced Options -> Logging, change the Destination from NONE to DBFS and change the path to the desired output location. Note: The unique cluster ID will be added onto the end of the provided path.

For debugging issues between the Immuta web service and Databricks, you can view the Spark UI on your target Databricks cluster. On the cluster page, click the Spark UI tab, which shows the Spark application UI for the cluster. If you encounter issues creating Databricks data sources in Immuta, you can also view the JDBC/ODBC Server portion of the Spark UI to see the result of queries that have been sent from Immuta to Databricks.

Using the Validation and Debugging Notebook

The Validation and Debugging Notebook (immuta-validation.ipynb) is packaged with other Databricks release artifacts (for manual installations), or it can be downloaded from the App Settings page when configuring native Databricks through the Immuta UI. This notebook is designed to be used by or under the guidance of an Immuta Support Professional.

1. Import the notebook into a Databricks workspace by navigating to Home in your Databricks instance.

2. Click the arrow next to your name and select Import.

3. Once you have executed commands in the notebook and populated it with debugging information, export the notebook and its contents by opening the File menu, selecting Export, and then selecting DBC Archive.

When you enable Unity Catalog, Immuta automatically migrates your existing Databricks data sources in Immuta to reference the legacy hive_metastore catalog to account for Unity Catalog's . New data sources will reference the Unity Catalog metastore you create and attach to your Databricks workspace.

Because the hive_metastore catalog is not managed by Unity Catalog, existing data sources in the hive_metastore cannot have Unity Catalog access controls applied to them. .

To allow Immuta to administer Unity Catalog access controls on that data, move the data to Unity Catalog and re-register those tables in Immuta by completing the steps below. If you don't move all data before configuring the integration, will protect your existing data sources throughout the migration process.

1. Ensure that all Databricks clusters that have Immuta installed are stopped and the Immuta configuration is removed from the cluster. Immuta-specific cluster configuration is no longer needed with the Databricks Unity Catalog integration.

2. Move all data into Unity Catalog before configuring Immuta with Unity Catalog. Existing data sources will need to be re-created after they are moved to Unity Catalog and the Unity Catalog integration is configured.

3. .

This guide details the simplified installation method for enabling native access to Databricks with Immuta policies enforced.

Ensure your Databricks workspace, instance, and permissions meet the guidelines outlined in the before you begin.

Requirements

• If Databricks Unity Catalog is enabled in a Databricks workspace, you must use an Immuta cluster policy when you set up the Databricks Spark integration to create an Immuta-enabled cluster. See the section below for guidance.

• If Databricks Unity Catalog is not enabled in your Databricks workspace, you must disable Unity Catalog in your Immuta tenant before proceeding with your configuration of Databricks Spark:

  1. Navigate to the App Settings page and click Integration Settings.

  2. Uncheck the Enable Unity Catalog checkbox.

  3. Click Save.

1 - Add the Integration on the App Settings Page

1. Log in to Immuta and click the App Settings icon in the left sidebar.

2. Scroll to the System API Key subsection under HDFS and click Generate Key.

3. Click Save and then Confirm.

4. Scroll to the Integration Settings section.

5. Click + Add Native Integration and select Databricks Integration from the dropdown menu.

6. Complete the Hostname field.

7. Enter a Unique ID for the integration. By default, your Immuta tenant URL populates this field. This ID is used to tie the set of cluster policies to your Immuta tenant and allows multiple Immuta tenants to access the same Databricks workspace without cluster policy conflicts.

8. Select your configured Immuta IAM from the dropdown menu.

9. Choose one of the following options for your data access model:

   • Protected until made available by policy: All tables are hidden until a user is permissioned through an Immuta policy. This is how most databases work and assumes least privileged access and also means you will have to register all tables with Immuta.

   • Available until protected by policy: All tables are open until explicitly registered and protected by Immuta. This makes a lot of sense if most of your tables are non-sensitive and you can pick and choose which to protect.

10. Select the Storage Access Type from the dropdown menu.

11. Opt to add any Additional Hadoop Configuration Files.

12. Click Add Native Integration.

2 - Configure Cluster Policies

Several cluster policies are available on the App Settings page when configuring this integration:

Click a link above to read more about each of these cluster policies before continuing with the tutorial.

1. Click Configure Cluster Policies.

2. Select one or more cluster policies in the matrix by clicking the Select button(s).

3. Opt to check the Enable Unity Catalog checkbox to generate cluster policies that will enable Unity Catalog on your cluster. This option is only available when Databricks runtime 11.3 is selected.

4. Opt to make changes to these cluster policies by clicking Additional Policy Changes and editing the text field.

5. Use one of the two Installation Types described below to apply the policies to your cluster:

   • Automatically push cluster policies: This option allows you to automatically push the cluster policies to the configured Databricks workspace. This will overwrite any cluster policy templates previously applied to this workspace.

     1. Select the Automatically Push Cluster Policies radio button.

     2. Enter your Admin Token. This token must be for a user who can create cluster policies in Databricks.

     3. Click Apply Policies.

   • Manually push cluster policies: Enabling this option will allow you to manually push the cluster policies to the configured Databricks workspace. There will be various files to download and manually push to the configured Databricks workspace.

     1. Select the Manually Push Cluster Policies radio button.

     2. Click Download Init Script.

     3. Follow the steps in the Instructions to upload the init script to DBFS section.

     4. Click Download Policies, and then manually add these Cluster Policies in Databricks.

6. Opt to click the Download the Benchmarking Suite to compare a regular Databricks cluster to one protected by Immuta. Detailed instructions are available in the first notebook, which will require an Immuta and non-Immuta cluster to generate test data and perform queries.

7. Click Close, and then click Save and Confirm.

3 - Add Policies to Your Cluster

2. In the Policy dropdown, select the Cluster Policies you pushed or manually added from Immuta.

3. Select the Custom Access mode.

4. Opt to adjust Autopilot Options and Worker Type settings: The default values provided here may be more than what is necessary for non-production or smaller use-cases. To reduce resource usage you can enable/disable autoscaling, limit the size and number of workers, and set the inactivity timeout to a lower value.

5. Opt to configure the Instances tab in the Advanced Options section:
6. Click Create Cluster.

4 - Register data

5 - Query Immuta Data

1. Before users can query an Immuta data source, an administrator must give the user Can Attach To permissions on the cluster.

Example Queries

In this integration, Immuta manages access to Snowflake tables by administering Snowflake and on those tables, allowing users to query tables directly in Snowflake while dynamic policies are enforced.

Like with all Immuta integrations, Immuta can inject its ABAC model into policy building and administration to remove policy management burden and significantly reduce role explosion.

Architecture

When an administrator configures the Snowflake integration with Immuta, Immuta creates an IMMUTA database and schemas (immuta_procedures, immuta_policies, and immuta_functions) within Snowflake to contain policy definitions and user entitlements. Immuta then creates a system role and gives that system account the following privileges:

• APPLY MASKING POLICY

• APPLY ROW ACCESS POLICY

• ALL PRIVILEGES ON DATABASE "IMMUTA" WITH GRANT OPTION

• ALL PRIVILEGES ON ALL SCHEMAS IN DATABASE "IMMUTA" WITH GRANT OPTION

• USAGE ON FUTURE PROCEDURES IN SCHEMA "IMMUTA".immuta_procedures WITH GRANT OPTION

• USAGE ON WAREHOUSE

• OWNERSHIP ON SCHEMA "IMMUTA".immuta_policies TO ROLE "IMMUTA_SYSTEM" COPY CURRENT GRANTS

• OWNERSHIP ON SCHEMA "IMMUTA".immuta_procedures TO ROLE "IMMUTA_SYSTEM" COPY CURRENT GRANTS

• OWNERSHIP ON SCHEMA "IMMUTA".immuta_functions TO ROLE "IMMUTA_SYSTEM" COPY CURRENT GRANTS

• OWNERSHIP ON SCHEMA "IMMUTA".public TO ROLE "IMMUTA_SYSTEM" COPY CURRENT GRANTS

Optional features, like automatic object tagging, native query auditing, etc., require additional permissions to be granted to the Immuta system account, are listed in the section.

Policy enforcement

For a user to query Immuta-protected data, they must meet two qualifications:

1. They must be subscribed to the Immuta data source.

After a user has met these qualifications they can query Snowflake tables directly.

Comply with column length and precision requirements in a Snowflake masking policy

Consider these columns in a data source that have the following masking policies applied:

• Column A (VARCHAR(6)): Mask using hashing for everyone

• Column B (VARCHAR(5)): Mask using a constant REDACTED for everyone

• Column C (VARCHAR(6)): Mask by making null for everyone

• Column D (NUMBER(3, 0)): Mask by rounding to the nearest 10 for everyone

Querying this data source in Snowflake would return the following values:

Query performance

Registering data sources

Register Snowflake data sources using a dedicated Snowflake role. Avoid using individual user accounts for data source onboarding. Instead, create a service account (Snowflake user account TYPE=SERVICE) with SELECT access for onboarding data sources. No policies will apply to that account, ensuring that your integration works with the following use cases:

Snowflake bulk data source creation

Bulk data source creation is the more efficient process when loading more than 5000 data sources from Snowflake and allows for data sources to be registered in Immuta before running sensitive data discovery or applying policies.

Resource allocations

Based on performance tests that create 100,000 data sources, Immuta recommends a SaaS XL environment.

Limitations

• Performance gains are limited when enabling sensitive data discovery at the time of data source creation.

• External catalog integrations are not recognized during bulk data source creation. Users must manually trigger a catalog sync for tags to appear on the data source through the data source's health check.

Excepted roles/users

Excepted roles and users are assigned when the integration is installed, and no policies will apply to these users' queries, despite any Immuta policies enforced on the tables they are querying. Credentials used to register a data source in Immuta will be automatically added to this excepted list for that Snowflake table. Consequently, roles and users added to this list and used to register data sources in Immuta should be limited to service accounts.

Immuta excludes the listed roles and users from policies by wrapping all policies in a CASE statement that will check if a user is acting under one of the listed usernames or roles. If a user is, then the policy will not be acted on the queried table. If the user is not, then the policy will be executed like normal. Immuta does not distinguish between role and username, so if you have a role and user with the exact same name, both the user and any user acting under that role will have full access to the data sources and no policies will be enforced for them.

Data flow

2. Immuta creates a database inside the configured Snowflake warehouse that contains Immuta policy definitions and user entitlements.

6. The Immuta web service calls a stored procedure that modifies the user entitlements or policies.

9. A Snowflake user who is subscribed to the data source in Immuta queries the corresponding table directly in Snowflake and sees policy-enforced data.

Authentication methods

The Snowflake integration supports the following authentication methods to configure the integration and create data sources:

• Username and password: Users can authenticate with their Snowflake username and password.

Snowflake External OAuth

Workflow

1. An Immuta application administrator configures the Snowflake integration or creates a data source.

2. Immuta creates a custom token and sends it to the authorization server.

3. The authorization server confirms the information sent from Immuta and issues an access token to Immuta.

4. Immuta sends the access token it received from the authorization server to Snowflake.

5. Snowflake authenticates the token and grants access to the requested resources from Immuta.

6. The integration is connected and users can query data.

Supported Snowflake features

The Immuta Snowflake integration supports the following Snowflake features:

Supported Immuta features

The Snowflake integration supports the Immuta features outlined below. Click the links provided for more details.

Immuta project workspaces

Caveat

To use project workspaces with the Snowflake integration, the default role of the account used to create data sources in the project must be added to the "Excepted Roles/Users List." If the role is not added, you will not be able to query the equalized view using the project role in Snowflake.

Tag ingestion

You can enable Snowflake tag ingestion so that Immuta will ingest Snowflake object tags from your Snowflake instance into Immuta and add them to the appropriate data sources.

The Snowflake tags' key and value pairs will be reflected in Immuta as two levels: the key will be the top level and the value the second. As Snowflake tags are hierarchical, Snowflake tags applied to a database will also be applied to all of the schemas in that database, all of the tables within those schemas, and all of the columns within those tables. For example: If a database is tagged PII, all of the tables and columns in that database will also be tagged PII.

Caveats

Native query audit

Multiple Snowflake instances

Caveats

• There can only be one integration connection with Immuta per host.

• The host of the data source must match the host of the integration for the view to be created.

• Projects can only be configured to use one Snowflake host.

Limitations

• Once a Snowflake integration is disabled in Immuta, the user must remove the access that was granted in Snowflake. If that access is not revoked, users will be able to access the raw table in Snowflake.

• Migration must be done using the credentials and credential method (automatic or bootstrap) used to configure the integration.

• When configuring one Snowflake instance with multiple Immuta tenants, the user or system account that enables the integration on the app settings page must be unique for each Immuta tenant.

• You cannot add a masking policy to an external table column while creating the external table because a masking policy cannot be attached to a virtual column.

• Snowflake tables from imported databases are not supported. Instead, create a view of the table and register that view as a data source.

Custom WHERE clause limitations

Requirements for a custom WHERE policy

1. All column names must be fully qualified: Any column names that are unqualified (i.e., just the column name) will default to a column of the data source the policy is being applied to (if one matches the name).

2. The Immuta system account must have SELECT privileges on all tables/views referenced in a subquery: The Immuta system role name is specified by the user, and the role is created when the Snowflake instance is integrated.

Subquery limitations

Any subqueries that error in Snowflake will also error in Immuta.

1. Including one or more subqueries in the Immuta policy condition may cause errors in Snowflake. If an error occurs, it may happen during policy creation or at query-time. To avoid these errors, limit the number of subqueries, limit the number of JOIN operations, and simplify WHERE clause conditions.

2. For more information on the Snowflake subquery limitations see

Immuta’s integration with Unity Catalog allows you to enforce fine-grained access controls on Unity Catalog securable objects with Immuta policies. Instead of manually creating UDFs or granting access to each table in Databricks, you can author your policies in Immuta and have Immuta manage and orchestrate Unity Catalog access-control policies on your data in Databricks clusters or SQL warehouses:

• Subscription policies: Immuta subscription policies automatically grant and revoke access to specific Databricks securable objects.

• : Immuta data policies enforce row- and column-level security.

Unity Catalog object model

Unity Catalog uses the following hierarchy of data objects:

• Metastore: Created at the account level and is attached to one or more Databricks workspaces. The metastore contains metadata of all the catalogs, schemas, and tables available to query. All clusters on that workspace use the configured metastore and all workspaces that are configured to use a single metastore share those objects.

• Catalog: Sits on top of schemas (also called databases) and tables to manage permissions across a set of schemas

• Schema: Organizes tables and views

• Table-etc: Table (managed or external tables), view, volume, model, and function

For details about the Unity Catalog object model, see the .

Feature support

The Databricks Unity Catalog integration supports

• :

  • applying column masks and row filters on specific securable objects

  • applying subscription polices on tables and views

• enforcing Unity Catalog access controls, even if Immuta becomes disconnected

• allowing non-Immuta reads and writes

• using Photon

• using a proxy server

Architecture

Immuta uses this service principal to run queries that set up user-defined functions (UDFs) and other data necessary for policy enforcement. Upon enabling the native integration, Immuta will create a catalog that contains these schemas:

• immuta_system: Contains internal Immuta data.

• immuta_policies_n: Contains policy UDFs.

When policies require changes to be pushed to Unity Catalog, Immuta updates the internal tables in the immuta_system schema with the updated policy information. If necessary, new UDFs are pushed to replace any out-of-date policies in the immuta_policies_n schemas and any row filters or column masks are updated to point at the new policies. Many of these operations require compute on the configured Databricks cluster or SQL warehouse, so compute must be available for these policies to succeed.

Workspace-catalog binding

Use cases

Typical use cases for binding a catalog to specific workspaces include

1. Ensuring users can only access production data from a production workspace environment.

   For example, you may have production data in a prod_catalog, as well as a production workspace you are introducing to your organization. Binding the prod_catalog to the prod_workspace ensures that workspace admins and users can only access prod_catalog from the prod_workspace environment.

2. Ensuring users can only process sensitive data from a specific workspace. Limiting the environments from which users can access sensitive data helps better secure your organization’s data. Limiting access to one workspace also simplifies any monitoring, auditing, and understanding of which users are accessing specific data. This would entail a similar setup as the example above.

3. Giving users read-only access to production data from a developer workspace.

   This enables your organization to effectively conduct development and testing, while minimizing risk to production data. All user access to this catalog from this workspace can be specified as read-only, ensuring developers can access the data they need for testing without risk of any unwanted updates.

Additional workspace connections

Policy enforcement

Immuta’s Unity Catalog integration applies Databricks table-, row-, and column-level security controls that are enforced natively within Databricks. Immuta's management of these Databricks security controls is automated and ensures that they synchronize with Immuta policy or user entitlement changes.

• Row-level security: Immuta applies SQL UDFs to restrict access to rows for querying users.

• Column-level security: Immuta applies column-mask SQL UDFs to tables for querying users. These column-mask UDFs run for any column that requires masking.

The Unity Catalog integration supports the following policy types:

• • Conditional masking

  • Constant

  • Custom masking

  • Hashing

  • Null

  • Rounding (date and numeric rounding)

• • Matching (only show rows where)

    • Custom WHERE

    • Never

    • Where user

    • Where value in column

  • Minimization

  • Time-based restrictions

Project-scoped purpose exceptions for Databricks Unity Catalog

Databricks Unity Catalog views

If you are using views in Databricks Unity Catalog, one of the following must be true for project-scoped purpose exceptions to apply to the views in Databricks:

• The view and underlying table are registered as Immuta data sources and added to a project: If a view and its underlying table are both added as Immuta data sources, both of these assets must be added to the project for the project-scoped purpose exception to apply. If a view and underlying table are both added as data sources but the table is not added to an Immuta project, the purpose exception will not apply to the view because Databricks does not support fine-grained access controls on views.

• Only the underlying table is registered as an Immuta data source and added to a project: If only the underlying table is registered as an Immuta data source but the view is not registered, the purpose exception will apply to both the table and corresponding view in Databricks. Views are the only Databricks object that will have Immuta policies applied to them even if they're not registered as Immuta data sources (as long as their underlying tables are registered).

Masked joins for Databricks Unity Catalog

Policy exemption groups

Some users may need to be exempt from masking and row-level policy enforcement. When you add user accounts to the configured exemption group in Databricks, Immuta will not enforce policies for those users. Exemption groups are created when the Unity Catalog integration is configured, and no policies will apply to these users' queries, despite any policies enforced on the tables they query.

The principal used to register data sources in Immuta will be automatically added to this exemption group for that Databricks table. Consequently, users added to this list and used to register data sources in Immuta should be limited to service accounts.

Policy support with hive_metastore

When enabling Unity Catalog support in Immuta, the catalog for all Databricks data sources will be updated to point at the default hive_metastore catalog. Internally, Databricks exposes this catalog as a proxy to the workspace-level Hive metastore that schemas and tables were kept in before Unity Catalog. Since this catalog is not a real Unity Catalog catalog, it does not support any Unity Catalog policies. Therefore, Immuta will ignore any data sources in the hive_metastore in any Databricks Unity Catalog integration, and policies will not be applied to tables there.

Authentication methods

The Databricks Unity Catalog integration supports the following authentication methods to configure the integration and create data sources:

Immuta data sources in Unity Catalog

External data connectors and query-federated tables

Native query audit

Tag ingestion

You can enable tag ingestion to allow Immuta to ingest Databricks Unity Catalog table and column tags so that you can use them in Immuta policies to enforce access controls. When you enable this feature, Immuta uses the credentials and connection information from the Databricks Unity Catalog integration to pull tags from Databricks and apply them to data sources as they are registered in Immuta. If Databricks data sources preexist the Databricks Unity Catalog tag ingestion enablement, those data sources will automatically sync to the catalog and tags will apply. Immuta checks for changes to tags in Databricks and syncs Immuta data sources to those changes every 24 hours.