Immuta helps you achieve the following outcomes in your data platform:

• Simplify Operations: Immuta’s dynamic access control and policy management require 93x fewer data policies to manage access control in your data platform according to the GigaOm study. It is simple and scalable, which improves change management and lowers the total cost of ownership of cloud data management.

• Improve data security: Immuta helps prove compliance with rules and regulations, even when securing hundreds of thousands of tables. An Immuta customer, Swedbank, migrated all critical analytics workloads to the cloud in less than 12 months, including over 100 terabytes from more than 2,500 sources.

• Unlock data’s value: Immuta helps organizations get access to more data 100x faster, which translates to improved productivity. An Immuta customer, Thomson Reuters enabled faster access to data, resulting in a 60x increase in data usage and greater productivity.

How does Immuta do it?

Immuta provides three modules to create a full data security platform suite.

Sensitive data discovery and classification: Immuta Discover

Discover sensitive data from millions of fields without manual effort. With over 60 pre-built and domain-specific identifiers, you can tailor data classification to your unique business needs based on your desired confidence level.

Continuous data security monitoring: Immuta Detect

Leverage timely insights into data access and user activity with anomaly indicators for faster analysis and proactive actions.

Data security and access control: Immuta Secure

Immuta’s attribute-based access control (ABAC) delivers scalable data access without role explosion, and dynamic data masking ensures the right users can access the right data.

Immuta manages access to Snowflake tables by administering Snowflake row access policies and column masking policies on those tables, allowing users to query tables directly in Snowflake while dynamic policies are enforced.

Getting started

This getting started guide outlines how to integrate your Snowflake account with Immuta.

How-to guides

• Configure a Snowflake integration: Configure the Snowflake integration.

• Edit or remove an existing integration: Manage integration settings or delete your existing Snowflake integration.

• Integration settings:

  • Enable Snowflake table grants: Enable Snowflake table grants and configure the Snowflake role prefix.

  • Use Snowflake data sharing with Immuta: Use Snowflake data sharing with table grants or project workspaces.

  • Snowflake low row access policy mode: Enable Snowflake low row access policy mode.

  • Snowflake lineage tag propagation: Configure your Snowflake integration to automatically apply tags added to a Snowflake table to its descendant data source columns in Immuta.

Reference guides

• Phased Snowflake onboarding approach: A phased onboarding approach to configuring the Snowflake integration ensures that your users will not be immediately affected by changes as you add data sources and policies. This guide describes the settings and requirements for implementing this phased approach.

• Snowflake integration reference guide: This reference guide describes the design and features of the Snowflake integration.

• Snowflake table grants: Snowflake table grants simplifies the management of privileges in Snowflake when using Immuta. Instead of manually granting users access to tables registered in Immuta, you allow Immuta to manage privileges on your Snowflake tables and views according to subscription policies. This guide describes the components of Snowflake table grants and how they are used in Immuta's Snowflake integration.

• Snowflake data sharing with Immuta: Organizations can share the policy-protected data of their Snowflake database with other Snowflake accounts with Immuta policies enforced in real time. This guide describes the components of using Immuta with Snowflake data shares.

• Snowflake low row access policy mode: The Snowflake low row access policy mode improves query performance in Immuta's Snowflake integration. To do so, this mode decreases the number of Snowflake row access policies Immuta creates and uses table grants to manage user access. This guide describes the design and requirements of this mode.

• Snowflake lineage tag propagation: Snowflake column lineage specifies how data flows from source tables or columns to the target tables in write operations. When Snowflake lineage tag propagation is enabled in Immuta, Immuta automatically applies tags added to a Snowflake table to its descendant data source columns in Immuta so you can build policies using those tags to restrict access to sensitive data.

• Warehouse sizing recommendations: Adjust the size and scale of clusters for your warehouse to manage workloads so that you can use Snowflake compute resources the most cost effectively.

To edit or remove a Snowflake integration, you have two options:

• Automatic: Grant Immuta one-time use of credentials to automatically edit or remove the integration.

  The credentials provided must have the following permissions:

  • CREATE DATABASE ON ACCOUNT WITH GRANT OPTION

  • CREATE ROLE ON ACCOUNT WITH GRANT OPTION

  • CREATE USER ON ACCOUNT WITH GRANT OPTION

  • MANAGE GRANTS ON ACCOUNT WITH GRANT OPTION

• Manual: Run the Immuta script in your Snowflake environment yourself to edit or remove the integration.

  The specified role used to run the bootstrap needs to have the following privileges:

  • CREATE DATABASE ON ACCOUNT WITH GRANT OPTION

  • CREATE ROLE ON ACCOUNT WITH GRANT OPTION

  • CREATE USER ON ACCOUNT WITH GRANT OPTION

  • MANAGE GRANTS ON ACCOUNT WITH GRANT OPTION

  • APPLY MASKING POLICY ON ACCOUNT WITH GRANT OPTION

  • APPLY ROW ACCESS POLICY ON ACCOUNT WITH GRANT OPTION

Edit a Snowflake integration

Select one of the following options for editing your integration:

• Automatic: Grant Immuta one-time use of credentials to automatically edit the integration.

• Manual: Run the Immuta script in your Snowflake environment yourself to edit the integration.

Automatic edit

1. Click the App Settings icon in the left sidebar.

2. Click the Integrations tab and click the down arrow next to the Snowflake integration.

3. Edit the field you want to change or check a checkbox of a feature you would like to enable. Note any field shadowed is not editable, and the integration must be disabled and re-installed to change it.

4. From the Select Authentication Method Dropdown, select either Username and Password or Key Pair Authentication:

   • Username and Password option: Complete the Username, Password, and Role fields.

   • Key Pair Authentication option:

     1. Complete the Username field.

     2. Click Key Pair (Required), and upload a Snowflake key pair file.

     3. Complete the Role field.

5. Click Save.

Manual edit

1. Click the App Settings icon in the left sidebar.

2. Click the Integrations tab and click the down arrow next to the Snowflake integration.

3. Edit the field you want to change or check a checkbox of a feature you would like to enable. Note any field shadowed is not editable, and the integration must be disabled and re-installed to change it.

4. Click edit script to download the script, and then run it in Snowflake.

5. Click Save.

Remove a Snowflake integration

Select one of the following options for deleting your integration:

• Automatic: Grant Immuta one-time use of credentials to automatically remove the integration and Immuta-managed resources from your Snowflake environment.

• Manual: Run the Immuta script in your Snowflake environment yourself to remove Immuta-managed resources and policies from Snowflake.

Automatic removal

1. Click the App Settings icon in the left sidebar.

2. Click the Integrations tab and click the down arrow next to the Snowflake integration.

3. Click the checkbox to disable the integration.

4. Enter the Username, Password, and Role that was entered when the integration was configured.

5. Click Save.

Manual removal

1. Click the App Settings icon in the left sidebar.

2. Click the Integrations tab and click the down arrow next to the Snowflake integration.

3. Click the checkbox to disable the integration.

4. Click cleanup script to download the script.

5. Click Save.

6. Run the cleanup script in Snowflake.

1. Navigate to the App Settings page.

2. Scroll to the Global Integrations Settings section.

3. Ensure the Snowflake Table Grants checkbox is checked. It is enabled by default.

4. Opt to change the Role Prefix. Snowflake table grants creates a new Snowflake role for each Immuta user. To ensure these Snowflake role names do not collide with existing Snowflake roles, each Snowflake role created for Snowflake table grants requires a common prefix. When using multiple Immuta accounts within a single Snowflake account, the Snowflake table grants role prefix should be unique for each Immuta account. The prefix must adhere to Snowflake identifier requirements and be less than 50 characters. Once the configuration is saved, the prefix cannot be modified; however, the Snowflake table grants feature can be disabled and re-enabled to change the prefix.

5. Finish configuring your integration by following one of these guidelines:

   • New Snowflake integration: Set up a new Snowflake integration by following the configuration tutorial.

   • Existing Snowflake integration (automatic setup): You will be prompted to enter connection information for a Snowflake user. Immuta will execute the migration to Snowflake table grants using a connection established with this Snowflake user. The Snowflake user you provide here must have Snowflake privileges to run these privilege grants.

   • Existing Snowflake integration (manual setup): Immuta will display a link to a migration script you must run in Snowflake and a link to a rollback script for use in the event of a failed migration. Important: Execute the migration script in Snowflake before clicking Save on the app settings page.

Immuta is compatible with Snowflake Secure Data Sharing. Using both Immuta and Snowflake, organizations can share the policy-protected data of their Snowflake database with other Snowflake accounts with Immuta policies enforced in real time.

Prerequisites:

• Snowflake integration enabled

• Snowflake tables registered in Immuta as data sources

Create Immuta Policies to Protect the Data

Required Permission: Immuta: GOVERNANCE

Build Immuta data policies to fit your organization's compliance requirements.

It's important to understand that subscription policies are not relevant to Snowflake data shares, because the act of sharing the data is the subscription policy. Data policies can be enforced on the consuming account from the producer account on a share following these instructions.

Register the Snowflake Data Consumer with Immuta

Required Permission: Immuta: USER_ADMIN

To register the Snowflake data consumer in Immuta,

1. Create a new Immuta user.

2. Update the Immuta user's Snowflake username to match the account ID for the data consumer. This value is the output on the data consumer side when SELECT CURRENT_ACCOUNT() is run in Snowflake.

3. Give the Immuta user the appropriate attributes and groups for your organization's policies.

4. Subscribe the Immuta user to the data sources.

Create the Snowflake Data Share

Required Permission: Snowflake ACCOUNTADMIN

To share the policy-protected data source,

1. Create a Snowflake Data Share of the Snowflake table that has been registered in Immuta.

2. Grant reference usage on the Immuta database to the share you created:

   Copy

   GRANT REFERENCE_USAGE ON DATABASE "<Immuta database of the provider account>" TO SHARE "<DATA_SHARE>";

   Replace the content in angle brackets above with the name of your Immuta database and Snowflake data share.

Prerequisites

This upgrade step is necessary if you meet both of the following criteria:

• You have the Snowflake low row access policy mode enabled in private preview.

• You have user impersonation enabled.

If you do not meet this criteria, follow the instructions on the configuration guide.

Upgrade to Snowflake low row access policy mode

To upgrade to the generally available version of the feature, disable your Snowflake integration on the app settings page and then re-enable it.

Immuta does not require users to learn a new API or language to access protected data. Instead, Immuta integrates with existing tools and ongoing work while remaining invisible to downstream consumers.

The following data platforms integrate with Immuta:

• Snowflake integration: With this integration, policies administered in Immuta are pushed down into Snowflake as Snowflake governance features (row access policies and masking policies).

• Databricks:

  • Databricks Unity Catalog integration: This integration allows you to manage multiple Databricks workspaces through Unity Catalog while protecting your data with Immuta policies. Instead of manually creating UDFs or granting access to each table in Databricks, you can author your policies in Immuta and have Immuta manage and enforce Unity Catalog access-control policies on your data in Databricks clusters or SQL warehouse.

  • Databricks Spark integration: This integration enforces policies on Databricks tables registered as data sources in Immuta, allowing users to query policy-enforced data on Databricks clusters (including job clusters). Immuta policies are applied to the plan that Spark builds for users' queries, all executed directly against Databricks tables.

• Google BigQuery: In this integration, Immuta generates policy-enforced views in your configured Google BigQuery dataset for tables registered as Immuta data sources.

• Starburst (Trino) integration: The Starburst (Trino) integration allows you to access policy-protected data directly in your Starburst (Trino) catalogs without rewriting queries or changing your workflows. Immuta policies are translated into Starburst (Trino) rules and permissions and applied directly to tables within your existing catalogs.

• Redshift integration: With the Redshift integration, Immuta applies policies directly in Redshift. This allows data analysts to query their data directly in Redshift instead of going through a proxy.

• Azure Synapse Analytics integration: The Azure Synapse Analytics integration allows Immuta to apply policies directly in Azure Synapse Analytics dedicated SQL pools without needing users to go through a proxy. Instead, users can work within their existing Synapse Studio and have per-user policies dynamically applied at query time.

• Amazon S3 integration: The Amazon S3 integration allows users to apply subscription policies to data in S3 to restrict what prefixes, buckets, or objects users can access. To enforce access controls on this data, Immuta creates S3 grants that are administered by S3 Access Grants, an AWS feature that defines access permissions to data in S3.

Feature support

The table below outlines the features supported by each of Immuta's integrations.

Policy support

Certain policies are unsupported or supported with caveats*, depending on the integration:

*Supported with caveats:

• On Databricks data sources, joins will not be allowed on data protected with replace with NULL or constant policies.

• Databricks Unity Catalog ARRAY, MAP, or STRUCT type columns only support masking with NULL.

• On Starburst data sources, the Immuta @iam function for WHERE clause policies can block the creation of views.

For details about each of these policies, see the Policies in Immuta page.

Audit support for platform queries

The table below outlines what information is included in the query audit logs for each integration where query audit is supported.

Legend:

Snowflake column lineage specifies how data flows from source tables or columns to the target tables in write operations. When Snowflake lineage tag propagation is enabled in Immuta, Immuta automatically applies tags added to a Snowflake table to its descendant data source columns in Immuta so you can build policies using those tags to restrict access to sensitive data.

Snowflake Access History tracks user read and write operations. Snowflake column lineage extends this Access History to specify how data flows from source columns to the target columns in write operations, allowing data stewards to understand how sensitive data moves from ancestor tables to target tables so that they can

• trace data back to its source to validate the integrity of dashboards and reports,

• identify who performed write operations to meet compliance requirements,

• evaluate data quality and pinpoint points of failure, and

• tag sensitive data on source tables without having tag columns on their descendant tables.

However, tagging sensitive data doesn’t innately protect that data in Snowflake; users need Immuta to disseminate these lineage tags automatically to descendant tables registered in Immuta so data stewards can build policies using the semantic and business context captured by those tags to restrict access to sensitive data. When Snowflake lineage tag propagation is enabled, Immuta propagates tags applied to a data source to its descendant data source columns in Immuta, which keeps your data inventory in Immuta up-to-date and allows you to protect your data with policies without having to manually tag every new Snowflake data source you register in Immuta.

Data flow

1. An application administrator enables the feature on the Immuta app settings page.

2. Snowflake lineage metadata (column names and tags) for the Snowflake tables is stored in the metadata database.

3. A data owner creates a new data source (or adds a new column to a Snowflake table) that initiates a job that applies all tags for each column from its ancestor columns.

4. A data owner or governor adds a tag to a column in Immuta that has descendants, which initiates a job that propagates the tag to all descendants.

5. An audit record is created that includes which tags were applied and from which columns those tags originated.

Snowflake access history view and Immuta lineage job

The Snowflake Account Usage ACCESS_HISTORY view contains column lineage information.

To appropriately propagate tags to descendant data sources, Immuta fetches Access History metadata to determine what column tags have been updated, stores this metadata in the Immuta metadata database, and then applies those tags to relevant descendant columns of tables registered in Immuta.

Consider the following example using the Customer, Customer 2, and Customer 3 tables that were all registered in Immuta as data sources.

• Customer: source table

• Customer 2: descendant of Customer

• Customer 3: descendant of Customer 2

If the Discovered.Electronic Mail Address tag is added to the Customer data source in Immuta, that tag will propagate through lineage to the Customer 2 and Customer 3 data sources.

Data source registration

After an application administrator has enabled Snowflake lineage tag propagation, data owners can register data in Immuta and have tags in Snowflake propagated from ancestor tables to descendant data sources. Whenever new tags are added to those tables in Immuta, those upstream tags will propagate to descendant data sources.

By default all tags are propagated, but these tags can be filtered on the app settings page or using the Immuta API.

Managing tags

Lineage tag propagation works with any tag added to the data dictionary. Tags can be manually added, synced from an external catalog, or discovered by SDD. Consider the following example using the Customer, Customer 2, and Customer 3 tables that were all registered in Immuta as data sources.

• Customer: source table

• Customer 2: descendant of Customer

• Customer 3: descendant of Customer 2

Immuta added the Discovered.Electronic Mail Address tag to the Customer data source, and that tag propagated through lineage to the Customer 2 and Customer 3 data sources.

Removing the tag from the Customer 2 table soft deletes it from the Customer 2 data source. When a tag is deleted, downstream lineage tags are removed, unless another parent data source still has that tag. The tag remains visible, but it will not be re-added if a future propagation event specifies the same tag again. Immuta prevents you from removing Snowflake object tags from data sources. You can only remove Immuta-managed tags. To remove Snowflake object tags from tables, you must remove them in Snowflake.

However the Discovered.Electronic Mail Address tag still applies to the Customer 3 data source because Customer still has the tag applied. The only way a tag will be removed from descendant data sources is if no other ancestor of the descendant still prescribes the tag.

If the Snowflake lineage tag propagation feature is disabled, tags will remain on Immuta data sources.

Sensitive data discovery

Snowflake lineage audit

Immuta audit records include Snowflake lineage tag events when a tag is added or removed.

The example audit record below illustrates the SNOWFLAKE_TAGS.pii tag successfully propagating from the Customer table to Customer 2:

Limitations

• Without tableFilter set, Immuta will ingest lineage for every table on the Snowflake instance.

• Tag propagation based on lineage is not retroactive. For example, if you add a table, add tags to that table, and then run the lineage ingestion job, tags will not get propagated. However, if you add a table, run the lineage ingestion job, and then add tags to the table, the tags will get propagated.

• The native lineage job needs to pull in lineage data before any tag is applied in Immuta. When Immuta gets new lineage information from Snowflake, Immuta does not update existing tags in Immuta.

• There can be up to a 3-hour delay in Snowflake for a lineage event to make it into the ACCESS_HISTORY view.

• Immuta does not ingest lineage information for views.

• Snowflake only captures lineage events for CTAS, CLONE, MERGE, and INSERT write operations. Snowflake does not capture lineage events for DROP, RENAME, ADD, or SWAP. Instead of using these latter operations, you need to recreate a table with the same name if you need to make changes.

• Immuta cannot enforce coherence of your Snowflake lineage. If a column, table, or schema in the middle of the lineage graph gets dropped, Immuta will not do anything unless a table with that same name gets recreated. This means a table that gets dropped but not recreated could live in Immuta’s system indefinitely.

Immuta integrates with your data platforms and external catalogs so you can register your data and effectively manage access controls on that data.

This section includes concept, reference, and how-to guides for configuring your data platform integration, registering data sources, and connecting your external catalog so that you can discover, monitor, and protect sensitive data using Immuta's flagship modules: , , and .

This reference guide outlines the features, policies, and audit capabilities supported by each integration.

This section includes how-to and reference guides for Snowflake and how it integrates with Immuta.

This section includes how-to and reference guides for Databricks Unity Catalog and how it integrates with Immuta.

This section includes how-to and reference guides for Databricks Spark and how it integrates with Immuta.

This section includes how-to and reference guides for Starburst (Trino) and how it integrates with Immuta.

This section includes how-to and reference guides for Redshift and how it integrates with Immuta.

This section includes how-to and reference guides for Azure Synapse Analytics and how it integrates with Immuta.

This page includes how-to and reference content for Amazon S3 and how it integrates with Immuta.

This page includes how-to and reference content for Google BigQuery and how it integrates with Immuta.

This section covers the various data catalogs Immuta integrates with.

This reference guide outlines the actions and features that trigger Immuta queries in your remote platform that may incur cost.

In this integration, Immuta manages access to Snowflake tables by administering Snowflake and on those tables, allowing users to query tables directly in Snowflake while dynamic policies are enforced.

Like with all Immuta integrations, Immuta can inject its ABAC model into policy building and administration to remove policy management burden and significantly reduce role explosion.

Architecture

When an administrator configures the Snowflake integration with Immuta, Immuta creates an IMMUTA database and schemas (immuta_procedures, immuta_policies, and immuta_functions) within Snowflake to contain policy definitions and user entitlements. Immuta then creates a system role and gives that system account the following privileges:

• APPLY MASKING POLICY

• APPLY ROW ACCESS POLICY

• ALL PRIVILEGES ON DATABASE "IMMUTA" WITH GRANT OPTION

• ALL PRIVILEGES ON ALL SCHEMAS IN DATABASE "IMMUTA" WITH GRANT OPTION

• USAGE ON FUTURE PROCEDURES IN SCHEMA "IMMUTA".immuta_procedures WITH GRANT OPTION

• USAGE ON WAREHOUSE

• OWNERSHIP ON SCHEMA "IMMUTA".immuta_policies TO ROLE "IMMUTA_SYSTEM" COPY CURRENT GRANTS

• OWNERSHIP ON SCHEMA "IMMUTA".immuta_procedures TO ROLE "IMMUTA_SYSTEM" COPY CURRENT GRANTS

• OWNERSHIP ON SCHEMA "IMMUTA".immuta_functions TO ROLE "IMMUTA_SYSTEM" COPY CURRENT GRANTS

• OWNERSHIP ON SCHEMA "IMMUTA".public TO ROLE "IMMUTA_SYSTEM" COPY CURRENT GRANTS

Optional features, like automatic object tagging, native query auditing, etc., require additional permissions to be granted to the Immuta system account, are listed in the section.

Policy enforcement

For a user to query Immuta-protected data, they must meet two qualifications:

1. They must be subscribed to the Immuta data source.

After a user has met these qualifications they can query Snowflake tables directly.

Comply with column length and precision requirements in a Snowflake masking policy

Consider these columns in a data source that have the following masking policies applied:

• Column A (VARCHAR(6)): Mask using hashing for everyone

• Column B (VARCHAR(5)): Mask using a constant REDACTED for everyone

• Column C (VARCHAR(6)): Mask by making null for everyone

• Column D (NUMBER(3, 0)): Mask by rounding to the nearest 10 for everyone

Querying this data source in Snowflake would return the following values:

Query performance

Integration health status

Registering data sources

Register Snowflake data sources using a dedicated Snowflake role. Avoid using individual user accounts for data source onboarding. Instead, create a service account (Snowflake user account TYPE=SERVICE) with SELECT access for onboarding data sources. No policies will apply to that account, ensuring that your integration works with the following use cases:

Snowflake bulk data source creation

Bulk data source creation is the more efficient process when loading more than 5000 data sources from Snowflake and allows for data sources to be registered in Immuta before running sensitive data discovery or applying policies.

Resource allocations

Based on performance tests that create 100,000 data sources, Immuta recommends a SaaS XL environment.

Limitations

• Performance gains are limited when enabling sensitive data discovery at the time of data source creation.

• External catalog integrations are not recognized during bulk data source creation. Users must manually trigger a catalog sync for tags to appear on the data source through the data source's health check.

Excepted roles/users

Excepted roles and users are assigned when the integration is installed, and no policies will apply to these users' queries, despite any Immuta policies enforced on the tables they are querying. Credentials used to register a data source in Immuta will be automatically added to this excepted list for that Snowflake table. Consequently, roles and users added to this list and used to register data sources in Immuta should be limited to service accounts.

Immuta excludes the listed roles and users from policies by wrapping all policies in a CASE statement that will check if a user is acting under one of the listed usernames or roles. If a user is, then the policy will not be acted on the queried table. If the user is not, then the policy will be executed like normal. Immuta does not distinguish between role and username, so if you have a role and user with the exact same name, both the user and any user acting under that role will have full access to the data sources and no policies will be enforced for them.

Data flow

2. Immuta creates a database inside the configured Snowflake warehouse that contains Immuta policy definitions and user entitlements.

6. The Immuta web service calls a stored procedure that modifies the user entitlements or policies.

9. A Snowflake user who is subscribed to the data source in Immuta queries the corresponding table directly in Snowflake and sees policy-enforced data.

Authentication methods

The Snowflake integration supports the following authentication methods to configure the integration and create data sources:

• Username and password: Users can authenticate with their Snowflake username and password.

Snowflake External OAuth

Workflow

1. An Immuta application administrator configures the Snowflake integration or creates a data source.

2. Immuta creates a custom token and sends it to the authorization server.

3. The authorization server confirms the information sent from Immuta and issues an access token to Immuta.

4. Immuta sends the access token it received from the authorization server to Snowflake.

5. Snowflake authenticates the token and grants access to the requested resources from Immuta.

6. The integration is connected and users can query data.

Supported Snowflake features

The Immuta Snowflake integration supports the following Snowflake features:

Supported Immuta features

The Snowflake integration supports the Immuta features outlined below. Click the links provided for more details.

Immuta project workspaces

Caveat

To use project workspaces with the Snowflake integration, the default role of the account used to create data sources in the project must be added to the "Excepted Roles/Users List." If the role is not added, you will not be able to query the equalized view using the project role in Snowflake.

Tag ingestion

You can enable Snowflake tag ingestion so that Immuta will ingest Snowflake object tags from your Snowflake instance into Immuta and add them to the appropriate data sources.

The Snowflake tags' key and value pairs will be reflected in Immuta as two levels: the key will be the top level and the value the second. As Snowflake tags are hierarchical, Snowflake tags applied to a database will also be applied to all of the schemas in that database, all of the tables within those schemas, and all of the columns within those tables. For example: If a database is tagged PII, all of the tables and columns in that database will also be tagged PII.

Caveats

Native query audit

Multiple Snowflake instances

Caveats

• There can only be one integration connection with Immuta per host.

• The host of the data source must match the host of the integration for the view to be created.

• Projects can only be configured to use one Snowflake host.

Limitations

• Once a Snowflake integration is disabled in Immuta, the user must remove the access that was granted in Snowflake. If that access is not revoked, users will be able to access the raw table in Snowflake.

• Migration must be done using the credentials and credential method (automatic or bootstrap) used to configure the integration.

• When configuring one Snowflake instance with multiple Immuta tenants, the user or system account that enables the integration on the app settings page must be unique for each Immuta tenant.

• You cannot add a masking policy to an external table column while creating the external table because a masking policy cannot be attached to a virtual column.

• Snowflake tables from imported databases are not supported. Instead, create a view of the table and register that view as a data source.

Custom WHERE clause limitations

Requirements for a custom WHERE policy

1. All column names must be fully qualified: Any column names that are unqualified (i.e., just the column name) will default to a column of the data source the policy is being applied to (if one matches the name).

2. The Immuta system account must have SELECT privileges on all tables/views referenced in a subquery: The Immuta system role name is specified by the user, and the role is created when the Snowflake instance is integrated.

Subquery limitations

Any subqueries that error in Snowflake will also error in Immuta.

1. Including one or more subqueries in the Immuta policy condition may cause errors in Snowflake. If an error occurs, it may happen during policy creation or at query-time. To avoid these errors, limit the number of subqueries, limit the number of JOIN operations, and simplify WHERE clause conditions.

2. For more information on the Snowflake subquery limitations see

This page details how to configure the using the legacy workflow. To configure the Snowflake integration and register data sources using , see this .

Permissions

Automatic setup permissions

When performing an automated installation, Immuta requires temporary, one-time use of credentials with the following permissions:

• CREATE DATABASE ON ACCOUNT WITH GRANT OPTION

• CREATE ROLE ON ACCOUNT WITH GRANT OPTION

• CREATE USER ON ACCOUNT WITH GRANT OPTION

• MANAGE GRANTS ON ACCOUNT WITH GRANT OPTION

• APPLY MASKING POLICY ON ACCOUNT WITH GRANT OPTION

• APPLY ROW ACCESS POLICY ON ACCOUNT WITH GRANT OPTION

These permissions will be used to create and configure a new IMMUTA database within the specified Snowflake instance. The credentials are not stored or saved by Immuta, and Immuta doesn’t retain access to them after initial setup is complete.

You can create a new account for Immuta to use that has these permissions, or you can grant temporary use of a pre-existing account. By default, the pre-existing account with appropriate permissions is ACCOUNTADMIN. If you create a new account, it can be deleted after initial setup is complete.

Alternatively, you can create the IMMUTA database within the specified Snowflake instance manually using the manual setup option.

Manual setup permissions

The specified role used to run the bootstrap needs to have the following privileges:

• CREATE DATABASE ON ACCOUNT WITH GRANT OPTION

• CREATE ROLE ON ACCOUNT WITH GRANT OPTION

• CREATE USER ON ACCOUNT WITH GRANT OPTION

• MANAGE GRANTS ON ACCOUNT WITH GRANT OPTION

• APPLY MASKING POLICY ON ACCOUNT WITH GRANT OPTION

• APPLY ROW ACCESS POLICY ON ACCOUNT WITH GRANT OPTION

It will create a user called IMMUTA_SYSTEM_ACCOUNT, and grant the following privileges to that user:

• APPLY MASKING POLICY ON ACCOUNT

• APPLY ROW ACCESS POLICY ON ACCOUNT

• Additional grants associated with the IMMUTA database

Configure the integration

2. Click the App Settings icon in the navigation panel.

3. Click the Integrations tab.

4. Click the +Add Native Integration button and select Snowflake from the dropdown menu.

5. Complete the Host, Port, and Default Warehouse fields.

7. Opt to check the Enable Impersonation box and customize the Impersonation Role to allow users to natively impersonate another user. You cannot edit this choice after you configure the integration.

8. 2. Enter how often, in hours, you want Immuta to ingest audit events from Snowflake as an integer between 1 and 24.

   3. Continue with your integration configuration.

Select your configuration method

You have two options for configuring your Snowflake environment:

Automatic setup

From the Select Authentication Method Dropdown, select one of the following authentication methods:

• Username and Password: Complete the Username, Password, and Role fields.

• Key Pair Authentication:

  1. Complete the Username field.

  2. When using a private key, enter the private key file password in the Additional Connection String Options. Use the following format: PRIV_KEY_FILE_PWD=<your_pw>

  3. Click Key Pair (Required), and upload a Snowflake key pair file.

  4. Complete the Role field.

Manual setup

It will create a user called IMMUTA_SYSTEM_ACCOUNT, and grant the following privileges to that user:

• APPLY MASKING POLICY ON ACCOUNT

• APPLY ROW ACCESS POLICY ON ACCOUNT

• Additional grants associated with the IMMUTA database

Run the script

1. Select Manual.

2. Use the Dropdown Menu to select your Authentication Method:

   • Username and password: Enter the Username and Password and set them in the bootstrap script for the Immuta system account credentials.

   • Key pair authentication: Upload the Key Pair file and when using a private key, enter the private key file password in the Additional Connection String Options. Use the following format: PRIV_KEY_FILE_PWD=<your_pw>

   • Snowflake External OAuth:

     2. Fill out the Token Endpoint. This is where the generated token is sent.

     3. Fill out the Client ID. This is the subject of the generated token.

     4. Select the method Immuta will use to obtain an access token:

        • Certificate

          1. Keep the Use Certificate checkbox enabled.

          2. Opt to fill out the Resource field with a URI of the resource where the requested token will be used.

          3. Enter the x509 Certificate Thumbprint. This identifies the corresponding key to the token and is often abbreviated as `x5t` or is called `sub` (Subject).

          4. Upload the PEM Certificate, which is the client certificate that is used to sign the authorization request.

        • Client secret

          1. Uncheck the Use Certificate checkbox.

          3. Enter the Client Secret (string). Immuta uses this secret to authenticate with the authorization server when it requests a token.

3. In the Setup section, click bootstrap script to download the script. Then, fill out the appropriate fields and run the bootstrap script in Snowflake.

Select available warehouses (optional)

If you enabled a Snowflake workspace, select Warehouses from the dropdown menu that will be available to project owners when creating native Snowflake workspaces. Select from a list of all the warehouses available to the privileged account entered above. Note that any warehouse accessible by the PUBLIC role does not need to be explicitly added.

Select excepted roles and users

Enter the Excepted Roles/User List. Each role or username (both case-sensitive) in this list should be separated by a comma.

Save the configuration

Click Save.

Opt to enable Snowflake tag ingestion

To allow Immuta to automatically import table and column tags from Snowflake, enable Snowflake tag ingestion in the external catalog section of the Immuta app settings page.

1. Navigate to the App Settings page.

2. Scroll to 2 External Catalogs, and click Add Catalog.

3. Enter a Display Name and select Snowflake from the dropdown menu.

4. Enter the Account.

5. Enter the Authentication information: Username, Password, Port, Default Warehouse, and Role.

6. Opt to enter the Proxy Host, Proxy Port, and Encrypted Key File Passphrase.

7. Opt to Upload Certificates.

8. Click the Test Connection button.

9. Click the Test Data Source Link.

10. Once both tests are successful, click Save.

Register data

1. Click the App Settings icon in the sidebar and scroll to the Global Integration Settings section.

2. Click the Enable Snowflake Low Row Access Policy Mode checkbox to enable the feature.

3. Confirm to allow Immuta to automatically disable impersonation for the Snowflake integration. If you do not confirm, you will not be able to enable Snowflake low row access policy mode.

4. Click Save.

Configure your Snowflake integration

If you already have a configured, you don't need to reconfigure your integration. Your Snowflake policies automatically refresh when you enable Snowflake low row access policy mode.

1. . Note that you will not be able to enable project workspaces or user impersonation with Snowflake low row access policy mode enabled.

2. Click Save and Confirm your changes.

The warehouse you select when configuring the Snowflake integration uses compute resources to set up the integration, register data sources, orchestrate policies, and run jobs like sensitive data discovery. Snowflake credit charges are based on the size of and amount of time the warehouse is active, not the number of queries run.

This document prescribes how and when to adjust the size and scale of clusters for your warehouse to manage workloads so that you can use Snowflake compute resources the most cost effectively.

In general, increase the size of and number of clusters for the warehouse to handle heavy workloads and multiple queries. Workloads are typically lighter after data sources are onboarded and policies are established in Immuta, so compute resources can be reduced after those workloads complete.

Integration and data source registration warehouse use

The Snowflake integration uses warehouse compute resources to sync policies created in Immuta to the Snowflake objects registered as data sources and, if enabled, to run and . Follow the guidelines below to adjust the warehouse size and scale according to your needs.

• Increase the of and of clusters for the warehouse during large policy syncs, updates, and changes.

• Enable to optimize resource use in Snowflake. In the Snowflake UI, the lowest auto suspend time setting is 5 minutes. However, through SQL query, you can set auto_suspend to 61 seconds (since the minimum uptime for a warehouse is 60 seconds). For example,

• Sensitive data discovery uses compute resources for each table registered if it is enabled. Consider when registering data sources if you have an or a tagging strategy in place.

• Register data before creating global policies. By default, Immuta on registered data (unless an existing global policy applies to it), which allows Immuta to only pull metadata instead of also applying policies when data sources are created. Registering data before policies are created reduces the workload and the Snowflake compute resources needed.

• Begin onboarding with a small dataset of tables, and then review and monitor query performance in the . Adjust the virtual warehouse accordingly to handle heavier loads.

• uses the compute warehouse that was employed during the initial ingestion to periodically monitor the schema for changes. If you expect a low number of new tables or minimal changes to the table structure, consider scaling down the warehouse size.

• Resize the warehouse after after data sources are registered and policies are established. For example,

For more details and guidance about warehouse sizing, see the .

Identifying bulk jobs and heavy workloads

Even after your integration is configured, data sources are registered, and policies are established, changes to those data sources or policies may initiate heavy workloads. Follow the guidelines below to adjust your warehouse size and scale according to your needs.

• Check how many credits queries have consumed:

Contact your Immuta representative to enable this feature in your Immuta tenant.

Configure the Snowflake integration

1. Navigate to the App Setting page and click the Integration tab.

2. Click +Add Native Integration and select Snowflake from the dropdown menu.

3. Complete the Host, Port, and Default Warehouse fields.

4. Enable Native Query Audit.

5. Enable Native Lineage and complete the following fields:

   • Ingest Batch Sizes: This setting configures the number of rows Immuta ingests per batch when streaming Access History data from your Snowflake instance.

   • Table Filter: This filter determines which tables Immuta will ingest lineage for. Enter a regular expression that excludes / from the beginning and end to filter tables. Without this filter, Immuta will attempt to ingest lineage for every table on your Snowflake instance.

   • Tag Filter: This filter determines which tags to propagate using lineage. Enter a regular expression that excludes / from the beginning and end to filter tags. Without this filter, Immuta will ingest lineage for every tag on your Snowflake instance.

6. Select Manual or Automatic Setup and

Trigger Snowflake lineage sync job

Prerequisite

.

Trigger the lineage job

The Snowflake lineage sync endpoint triggers the native lineage ingestion job that allows Immuta to propagate Snowflake tags added through lineage to Immuta data sources.

1. Copy the example and replace the Immuta URL and API key with your own.

2. Change the payload attribute values to your own, where

   • tableFilter (string): This regular expression determines which tables Immuta will ingest lineage for. Enter a regular expression that excludes / from the beginning and end to filter tables. Without this filter, Immuta will attempt to ingest lineage for every table on your Snowflake instance.

   • batchSize (integer): This parameter configures the number of rows Immuta ingests per batch when streaming Access History data from your Snowflake instance. Minimum 1.

   • lastTimestamp (string): Setting this parameter will only return lineage events later than the value provided. Use a format like 2022-06-29T09:47:06.012-07:00.

Next steps

Once the sync job is complete, you can complete the following steps:

Immuta is compatible with . Using both Immuta and Snowflake, organizations can share the policy-protected data of their Snowflake database with other Snowflake accounts with Immuta policies enforced in real time. This integration gives data consumers a live connection to the data and relieves data providers of the legal and technical burden of creating static data copies that leave their Snowflake environment.

Requirements:

• Snowflake Enterprise Edition or higher

• Immuta's

Configuration

This method requires that the data consumer account is registered as an Immuta user with the Snowflake user name equal to the consuming account.

At that point, the user that represents the account being shared with can have the appropriate attributes and groups assigned to them, relevant to the data policies that need to be enforced. Once that user has access to the share in the consuming account (not managed by Immuta), they can query the share with the data policies from the producer account enforced because Immuta is treating that account as if they are a single user in Immuta.

For a tutorial on this workflow, see the .

Benefits

Using Immuta with Snowflake Data Sharing allows the sharer to

• Only need limited knowledge of the context or goals of the existing policies in place: Because the sharer is not editing or creating policies to share their data, they only need a limited knowledge of how the policies work. Their main responsibility is making sure they properly represent the attributes of the data consumer (the account being shared to).

• Leave policies untouched.

The Snowflake low row access policy mode improves query performance in Immuta's Snowflake integration by decreasing the number of Immuta creates and by using table grants to manage user access.

Immuta manages access to Snowflake tables by administering and on those tables, allowing users to query them directly in Snowflake while policies are enforced.

Without Snowflake low row access policy mode enabled, row access policies are created and administered by Immuta in the following scenarios:

• are disabled and a subscription policy that does not automatically subscribe everyone to the data source is applied. Immuta administers Snowflake row access policies to filter out all the rows to restrict access to the entire table when the user doesn't have privileges to query it. However, if table grants are disabled and a subscription policy is applied that grants everyone access to the data source automatically, Immuta does not create a row access policy in Snowflake. See the for details about these policy types.

• is applied to a data source. A row access policy filters out all the rows of the table if users aren't acting under the purpose specified in the policy when they query the table.

• is applied to a data source. A row access policy filters out rows querying users don't have access to.

• is enabled. A row access policy is created for every Snowflake table registered in Immuta.

Reducing row access policies

Snowflake low row access policy mode is enabled by default to reduce the number of row access policies Immuta creates and improve query performance. Snowflake low row access policy mode requires

• .

• user impersonation to be disabled. User impersonation diminishes the performance of interactive queries because of the number of row access policies Immuta creates when it's enabled.

Requirements

Project-scoped purpose exceptions for Snowflake with low row access policy mode enabled

Masked joins for Snowflake with low row access policy mode enabled

Limitations and considerations

• Project workspaces are not compatible with this feature.

• Impersonation is not supported when the Snowflake low row access policy mode is enabled.

Snowflake table grants simplifies the management of privileges in Snowflake when using Immuta. Instead of having to manually grant users access to tables registered in Immuta, you allow Immuta to manage privileges on your Snowflake tables and views according to subscription policies. Then, users subscribed to a data source in Immuta can view and query the Snowflake table, while users who are not subscribed to the data source cannot view or query the Snowflake table.

Snowflake privileges

Enabling Snowflake table grants gives the following privileges to the Immuta Snowflake role:

• MANAGE GRANTS ON ACCOUNT allows the Immuta Snowflake role to grant and revoke SELECT privileges on Snowflake tables and views that have been added as data sources in Immuta.

• CREATE ROLE ON ACCOUNT allows for the creation of a Snowflake role for each user in Immuta, enabling fine-grained, attribute-based access controls to determine which tables are available to which individuals.

Table grants role

Since table privileges are granted to roles and not to users in Snowflake, Immuta's Snowflake table grants feature creates a new Snowflake role for each Immuta user. This design allows Immuta to manage table grants through fine-grained access controls that consider the individual attributes of users.

Each Snowflake user with an Immuta account will be granted a role that Immuta manages. The naming convention for this role is <IMMUTA>_USER_<username>, where

• <IMMUTA> is the prefix you specified when enabling the feature on the Immuta app settings page.

• <username> is the user's Immuta username.

Querying Snowflake tables managed by Immuta

Users are granted access to each Snowflake table or view automatically when they are subscribed to the corresponding data source in Immuta.

Users have two options for querying Snowflake tables that are managed by Immuta:

• that Immuta creates and manages. (For example, USE ROLE IMMUTA_USER_<username>. See the for details about the role and name conventions.) If the current active primary role is used to query tables, USAGE on a Snowflake warehouse must be granted to the Immuta-managed Snowflake role for each user.

• , which allows users to use the privileges from all roles that they have been granted, including IMMUTA_USER_<username>, in addition to the current active primary role. Users may also set a value for DEFAULT_SECONDARY_ROLES as an on a Snowflake user. To learn more about primary roles and secondary roles in Snowflake, see .

Applying GRANTs and REVOKEs at scale

Immuta uses an algorithm to determine the most optimal way to group users in a role hierarchy in order to optimize the number of GRANTs (or REVOKES) executed in Snowflake. This is done by determining the least amount of possible permutations of access across tables and users based on the policies in place; then, those become intermediate roles in the hierarchy that each user is added to, based on the intermediate roles they belong to.

As an example, take the below users and data sources they have access to. To do this naively by individually granting every user to the tables they have access to would result in 37 grants:

Conversely, using the Immuta algorithm, we can optimize the number of grants in the same scenario down to 29:

It’s important to consider a few things here:

1. If the permutations of access are small, there will be a huge optimization realized (very few intermediate roles). If every user has their own unique permutation of access, the optimization will be negligible (an intermediate role per user). It is most common that the number of permutations of access will be many multiples smaller than the actual user count, so there should be large optimizations. In other words, a much smaller number of intermediate roles and the number of total overall grants reduced, since the tables are granted to roles and roles to users.

2. This only happens once up front. After that, changes are incremental based on policy changes and user attribute changes (smaller updates), unless there’s a policy that makes a sweeping change across all users. The addition of new users who have access becomes much more straightforward also due to the fact above. User’s access will be granted via the intermediate role, and, therefore, a lot of the work is front loaded in the intermediate role creation.

Limitations

• If an Immuta tenant is connected to an external IAM and that external IAM has a username identical to another username in Immuta's built-in IAM, those users will have the same Snowflake role, leading both to see the same data.

• Sometimes the role generated can contain special characters such as @ because it's based on the user name configured from your identity manager. Because of this, it is recommended that any code references to the Immuta-generated role be enclosed with double quotes.

The how-to guides linked on this page illustrate how to integrate Snowflake with Immuta.

Requirement: Snowflake Enterprise Edition

Configure your Snowflake integration

These guides provide information on the recommended features to enable with Snowflake.

1. with the following features enabled:

   • (enabled by default)

   • (enabled by default)

   • (enabled by default)

2. Select None as your .

3. .

4. .

Register metadata

These guides provide instructions for organizing your Snowflake data to align with your governance structure.

1. .

These guides provide instructions for discovering, classifying, and tagging your data.

2. Validate the policy. You do not have to validate every policy you create in Immuta; instead, examine a few to validate the behavior you expect to see:

   1. Validate that the Immuta users impacted now have an Immuta role in Snowflake dedicated to them.

   2. Validate that when acting under the Immuta role those users have access to the table(s) in question.

   3. Validate that users without access in Immuta can still access the table with a different Snowflake role that has access.

   4. • they were not granted access by Immuta and

      • they have a role that provides them access, even if they are not currently acting under that role.

4. Validate that a user with a role that can access the table in question (whether it's an Immuta role or not) sees the impact of that data policy.

5. Once all Immuta policies are in place, remove or alter old roles.

The how-to guides linked on this page illustrate how to integrate Databricks Unity Catalog with Immuta.

Requirements:

• Unity Catalog metastore created and attached to a Databricks workspace. Immuta supports configuring a single metastore for each configured integration, and that metastore may be attached to multiple Databricks workspaces.

• Unity Catalog enabled on your Databricks cluster or SQL warehouse. All SQL warehouses have Unity Catalog enabled if your workspace is attached to a Unity Catalog metastore.

Configure your Databricks Unity Catalog integration

These guides provide information on the recommended features to enable with Databricks Unity Catalog.

1. Configure your Unity Catalog integration with the following feature enabled: Native query audit (enabled by default)

2. Select None as your default subscription policy.

3. Integrate an IAM with Immuta.

4. Map external user IDs from Unity Catalog to Immuta.

Register metadata

These guides provide instructions for organizing your Databricks Unity Catalog data to align with your governance structure.

1. Register Databricks securable objects in Immuta as data sources.

2. Recommended: Organize your data sources into domains and assign domain permissions to accountable teams.

Detect your user activity

These guides provide instructions for auditing and detecting your users' activity, or see the Detect use case for a comprehensive guide on the benefits of these features and other recommendations.

1. Set up audit export to S3 or ADLS Gen2 for your Databricks Unity Catalog audit logs.

2. View the Detect dashboards to see the activity of your users on Databricks Unity Catalog tables.

Discover your data

These guides provide instructions for discovering, classifying, and tagging your data.

1. Enable sensitive data discovery (SDD).

2. Register a subset of your tables to configure and validate SDD.

3. Configure SDD to discover entities of interest for your policy needs.

4. Validate that the SDD tags are applied correctly.

5. Register your remaining tables at the schema level with schema monitoring turned on.

6. Implement classification to categorize and tag sensitive data.

Secure your data

These guides provide instructions for configuring and securing your data with governance policies, or see the Secure use cases for a comprehensive guide on creating policies to fit your organization's use case.

1. Create a global subscription policy.

2. Create a global data policy.

3. Validate the policies. You do not have to validate every policy you create in Immuta; instead, examine a few to validate the behavior you expect to see.

4. Once all Immuta policies are in place, remove or alter old permissions and revoke access to the ungoverned tables.

Databricks Unity Catalog allows you to manage and access data in your Databricks account across all of your workspaces. With Immuta’s Databricks Unity Catalog integration, you can write your policies in Immuta and have them enforced automatically by Databricks across data in your Unity Catalog metastore.

This page details how to configure the integration. To configure the Databricks Unity Catalog integration and register data sources using the connections, see this how-to guide.

Permissions

Several different accounts are used to set up and maintain the Databricks Unity Catalog integration. The permissions required for each are outlined below.

• Immuta account (required): This user configures the integration on the app settings page in Immuta. To access the app settings page, this user needs the following permission:

  • APPLICATION_ADMIN Immuta permission

• Databricks service principal (required): This service principal is used continuously by Immuta to orchestrate Unity Catalog policies and maintain state between Immuta and Databricks. In the automatic setup option, Immuta also uses this service principal to create the Immuta-managed catalog. This service principal needs the following Databricks privileges:

  • CREATE CATALOG privilege on the Unity Catalog metastore. This is only required if you have Immuta automatically configure the integration in Databricks for you. If a separate user will run the Immuta script in Databricks to manually configure the integration, that Databricks user account needs this privilege instead.

  • OWNER permission on the Immuta catalog you configure.

  • OWNER privilege on one of the securables below so that Immuta can administer Unity Catalog row-level and column-level security controls.

    • on catalogs with schemas and tables registered as Immuta data sources. This permission could also be applied by granting OWNER on a catalog to a Databricks group that includes the Immuta service principal to allow for multiple owners.

    • on schemas with tables registered as Immuta data sources.

    • on all tables registered as Immuta data sources - if the OWNER permission cannot be applied at the catalog- or schema-level. In this case, each table registered as an Immuta data source must individually have the OWNER permission granted to the Immuta service principal.

  • USE CATALOG and USE SCHEMA on parent catalogs and schemas of tables registered as Immuta data sources so that the Immuta service principal can SELECT and MODIFY securables within the parent catalog and schema.

  • SELECT and MODIFY on all tables registered as Immuta data sources so that the Immuta service principal can grant and revoke access to tables and apply Unity Catalog row- and column-level security controls.

  • For native query audit (optional)

    • USE CATALOG on the system catalog

    • USE SCHEMA on the system.access schema

    • SELECT on the following system tables:

      • system.access.audit

      • system.access.table_lineage

      • system.access.column_lineage

• Databricks account (recommended): This user account can manually configure the integration in Databricks to create the Immuta-managed catalog. To do so, this account requires the following Databricks privileges:

  • CREATE CATALOG on the Unity Catalog metastore

  • ACCOUNT ADMIN on the Unity Catalog metastore for native query audit (optional)

Requirements

Before you configure the Databricks Unity Catalog integration, ensure that you have fulfilled the following requirements:

• Unity Catalog metastore created and attached to a Databricks workspace. Immuta supports configuring a single metastore for each configured integration, and that metastore may be attached to multiple Databricks workspaces.

• Unity Catalog enabled on your Databricks cluster or SQL warehouse. All SQL warehouses have Unity Catalog enabled if your workspace is attached to a Unity Catalog metastore. Immuta recommends linking a SQL warehouse to your Immuta tenant rather than a cluster for both performance and availability reasons.

Migrate data to Unity Catalog

1. Ensure that all Databricks clusters that have Immuta installed are stopped and the Immuta configuration is removed from the cluster. Immuta-specific cluster configuration is no longer needed with the Databricks Unity Catalog integration.

2. Move all data into Unity Catalog before configuring Immuta with Unity Catalog. Existing data sources will need to be re-created after they are moved to Unity Catalog and the Unity Catalog integration is configured. If you don't move all data before configuring the integration, metastore magic will protect your existing data sources throughout the migration process.

Create the Databricks service principal

In Databricks, create a service principal with the privileges listed above.

Opt to enable native query audit for Unity Catalog

1. Enable a system schema where the <SCHEMA_NAME> is access.

2. Enable verbose audit logs in Unity Catalog.

3. If you will configure the integration using the manual setup option, the Immuta script you will use includes the SQL statements for granting required privileges to the service principal, so you can skip this step and continue to the manual setup section. Otherwise, manually grant the Immuta service principal access to the Databricks Unity Catalog system tables. For Databricks Unity Catalog audit to work, the service principal must have the following access at minimum:

   • USE CATALOG on the system catalog

   • USE SCHEMA on the system.access schema

   • SELECT on the following system tables:

     • system.access.audit

     • system.access.table_lineage

     • system.access.column_lineage

Configure the Databricks Unity Catalog integration

You have two options for configuring your Databricks Unity Catalog integration:

• Automatic setup: Immuta creates the catalogs, schemas, tables, and functions using the integration's configured service principal.

• Manual setup: Run the Immuta script in Databricks yourself to create the catalog. You can also modify the script to customize your storage location for tables, schemas, or catalogs. The user running the script must have the Databricks privileges listed above.

Automatic setup

Required permissions: When performing an automatic setup, the Immuta service principal must have the permissions listed above.

1. Click the App Settings icon in the left sidebar.

2. Click the Integrations tab.

3. Click + Add Native Integration and select Databricks Unity Catalog from the dropdown menu.

4. Complete the following fields:

   • Server Hostname is the hostname of your Databricks workspace.

   • HTTP Path is the HTTP path of your Databricks cluster or SQL warehouse.

   • Immuta Catalog is the name of the catalog Immuta will create to store internal entitlements and other user data specific to Immuta. This catalog will only be readable for the Immuta service principal and should not be granted to other users. The catalog name may only contain letters, numbers, and underscores and cannot start with a number.

5. If using a proxy server with Databricks Unity Catalog, click the Enable Proxy Support checkbox and complete the Proxy Host and Proxy Port fields. The username and password fields are optional.

6. Opt to fill out the Exemption Group field with the name of a group in Databricks that will be excluded from having data policies applied and must not be changed from the default value. Create this account-level group for privileged users and service accounts that require an unmasked view of data before configuring the integration in Immuta.

7. Opt to scope the query audit ingestion by entering in Unity Catalog Workspace IDs. Enter a comma-separated list of the workspace IDs that you want Immuta to ingest audit records for. If left empty, Immuta will audit all tables and users in Unity Catalog.

8. Unity Catalog query audit is enabled by default; you can disable it by clicking the Enable Native Query Audit checkbox. Ensure you have enabled system tables in Unity Catalog and provided the required access to the Immuta service principal.

   1. Configure the audit frequency by scrolling to Integrations Settings and find the Unity Catalog Audit Sync Schedule section.

   2. Enter how often, in hours, you want Immuta to ingest audit events from Unity Catalog as an integer between 1 and 24.

   3. Continue with your integration configuration.

9. Select your authentication method from the dropdown:

   • Access Token: Enter a Databricks Personal Access Token. This is the access token for the Immuta service principal. This service principal must have the metastore privileges listed above for the metastore associated with the Databricks workspace. If this token is configured to expire, update this field regularly for the integration to continue to function.

   • OAuth machine-to-machine (M2M):

     • AWS Databricks:

       • Follow Databricks documentation to create a client secret for the Immuta service principal and assign this service principal the privileges listed above for the metastore associated with the Databricks workspace.

       • Fill out the Token Endpoint with the full URL of the identity provider. This is where the generated token is sent. The default value is https://<your workspace name>.cloud.databricks.com/oidc/v1/token.

       • Fill out the Client ID. This is a combination of letters, numbers, or symbols, used as a public identifier and is the client ID displayed in Databricks when creating the client secret for the service principal.

       • Enter the Scope (string). The scope limits the operations and roles allowed in Databricks by the access token. See the OAuth 2.0 documentation for details about scopes.

       • Enter the Client Secret you created above. Immuta uses this secret to authenticate with the authorization server when it requests a token.

     • Azure Databricks:

       • Follow Databricks documentation to create a service principal within Azure and then populate to your Databricks account and workspace.

       • Assign this service principal the privileges listed above for the metastore associated with the Databricks workspace.

       • Within Databricks, create an OAuth client secret for the service principal. This completes your Databricks-based service principal setup.

       • Within Immuta, fill out the Token Endpoint with the full URL of the identity provider. This is where the generated token is sent. The default value is https://<your workspace name>.azuredatabricks.net/oidc/v1/token.

       • Fill out the Client ID. This is a combination of letters, numbers, or symbols, used as a public identifier and is the client ID displayed in Databricks when creating the client secret for the service principal (note that Azure Databricks uses the Azure SP Client ID; it will be identical).

       • Enter the Scope (string). The scope limits the operations and roles allowed in Databricks by the access token. See the OAuth 2.0 documentation for details about scopes.

       • Enter the Client Secret you created above. Immuta uses this secret to authenticate with the authorization server when it requests a token.

10. Click Save.

Manual setup

Required permissions: When performing a manual setup, a service principal and a Databricks account must have the permissions listed above.

1. Click the App Settings icon in the left sidebar.

2. Click the Integrations tab.

3. Click + Add Native Integration and select Databricks Unity Catalog from the dropdown menu.

4. Complete the following fields:

   • Server Hostname is the hostname of your Databricks workspace.

   • HTTP Path is the HTTP path of your Databricks cluster or SQL warehouse.

   • Immuta Catalog is the name of the catalog Immuta will create to store internal entitlements and other user data specific to Immuta. This catalog will only be readable for the Immuta service principal and should not be granted to other users. The catalog name may only contain letters, numbers, and underscores and cannot start with a number.

5. If using a proxy server with Databricks Unity Catalog, click the Enable Proxy Support checkbox and complete the Proxy Host and Proxy Port fields. The username and password fields are optional.

6. Opt to fill out the Exemption Group field with the name of a group in Databricks that will be excluded from having data policies applied and must not be changed from the default value. Create this account-level group for privileged users and service accounts that require an unmasked view of data before configuring the integration in Immuta.

7. Opt to scope the query audit ingestion by entering in Unity Catalog Workspace IDs. Enter a comma-separated list of the workspace IDs that you want Immuta to ingest audit records for. If left empty, Immuta will audit all tables and users in Unity Catalog.

8. Unity Catalog query audit is enabled by default; you can disable it by clicking the Enable Native Query Audit checkbox. Ensure you have enabled system tables in Unity Catalog and provided the required access to the Immuta service principal.

   1. Configure the audit frequency by scrolling to Integrations Settings and find the Unity Catalog Audit Sync Schedule section.

   2. Enter how often, in hours, you want Immuta to ingest audit events from Unity Catalog as an integer between 1 and 24.

   3. Continue with your integration configuration.

9. Select your authentication method from the dropdown:

   • Access Token: Enter a Databricks Personal Access Token. This is the access token for the Immuta service principal. This service principal must have the metastore privileges listed above for the metastore associated with the Databricks workspace. If this token is configured to expire, update this field regularly for the integration to continue to function.

   • OAuth machine-to-machine (M2M):

     • AWS Databricks:

       • Follow Databricks documentation to create a client secret for the Immuta service principal and assign this service principal the privileges listed above for the metastore associated with the Databricks workspace.

       • Fill out the Token Endpoint with the full URL of the identity provider. This is where the generated token is sent. The default value is https://<your workspace name>.cloud.databricks.com/oidc/v1/token.

       • Fill out the Client ID. This is a combination of letters, numbers, or symbols, used as a public identifier and is the client ID displayed in Databricks when creating the client secret for the service principal.

       • Enter the Scope (string). The scope limits the operations and roles allowed in Databricks by the access token. See the OAuth 2.0 documentation for details about scopes.

       • Enter the Client Secret you created above. Immuta uses this secret to authenticate with the authorization server when it requests a token.

     • Azure Databricks:

       • Follow Databricks documentation to create a service principal within Azure and then populate to your Databricks account and workspace.

       • Assign this service principal the privileges listed above for the metastore associated with the Databricks workspace.

       • Within Databricks, create an OAuth client secret for the service principal. This completes your Databricks-based service principal setup.

       • Within Immuta, fill out the Token Endpoint with the full URL of the identity provider. This is where the generated token is sent. The default value is https://<your workspace name>.azuredatabricks.net/oidc/v1/token.

       • Fill out the Client ID. This is a combination of letters, numbers, or symbols, used as a public identifier and is the client ID displayed in Databricks when creating the client secret for the service principal (note that Azure Databricks uses the Azure SP Client ID; it will be identical).

       • Enter the Scope (string). The scope limits the operations and roles allowed in Databricks by the access token. See the OAuth 2.0 documentation for details about scopes.

       • Enter the Client Secret you created above. Immuta uses this secret to authenticate with the authorization server when it requests a token.

10. Select the Manual toggle and copy or download the script. You can modify the script to customize your storage location for tables, schemas, or catalogs.

11. Run the script in Databricks.

12. Click Save.

Map Databricks users to Immuta

If the usernames in Immuta do not match usernames in Databricks, map each Databricks username to each Immuta user account to ensure Immuta properly enforces policies using one of the methods linked below:

• Map the external IDs from an external identity manager

• Manually map the external IDs on the user's profile page

If the Databricks user doesn't exist in Databricks when you configure the integration, manually link their Immuta username to Databricks after they are created in Databricks. Otherwise, policies will not be enforced correctly for them in Databricks. Databricks user identities for Immuta users are automatically marked as invalid when the user is not found during policy application, preventing them from being affected by Databricks policy until their Immuta user identity is manually mapped to their Databricks identity.

Opt to enable Databricks Unity Catalog tag ingestion

Requirement: A Databricks Unity Catalog integration must be configured for tags to be ingested.

To allow Immuta to automatically import table and column tags from Databricks Unity Catalog, enable Databricks Unity Catalog tag ingestion in the external catalog section of the Immuta app settings page.

1. Navigate to the App Settings page.

2. Scroll to 2 External Catalogs, and click Add Catalog.

3. Enter a Display Name and select Databricks Unity Catalog from the dropdown menu.

4. Click Save and confirm your changes.

Register data

Register Databricks securables in Immuta.

This integration enforces policies on Databricks tables registered as data sources in Immuta, allowing users to query policy-enforced data on Databricks clusters (including job clusters). Immuta policies are applied to the plan that Spark builds for users' queries, all executed directly against Databricks tables.

The guides in this section outline how to integrate Databricks Spark with Immuta.

How-to guides

• Databricks Spark configuration: Configure the Databricks Spark integration.

• DBFS access: Access DBFS in Databricks for non-sensitive data.

• Limited enforcement in Databricks: Allow Immuta users to access tables that are not protected by Immuta.

• Hiding the Immuta database in Databricks: Hide the Immuta database from users in Databricks, since user queries do not need to reference it.

• Run spark-submit jobs on Databricks: Run R and Scala spark-submit jobs on your Databricks cluster.

• Project UDFs cache settings: Raise the caching on-cluster and lower the cache timeouts for the Immuta web service to allow use of project UDFs in Spark jobs.

• External metastores: Use an existing Hive external metastore instead of the built-in metastore.

Reference guides

• Databricks Spark integration reference guide: This guide describes the design and components of the integration.

• Configuration settings: These guides describe various integration settings that can be configured, including environment variables, cluster policies, and performance.

• Databricks change data feed: This guide describes Immuta's support of Databricks change data feed.

• Databricks libraries: The trusted libraries feature allows Databricks cluster administrators to avoid Immuta security manager errors when using third-party libraries. This guide describes the feature and its configuration.

• Delta Lake API: When using Delta Lake, the API does not go through the normal Spark execution path. This means that Immuta's Spark extensions do not provide protection for the API. To solve this issue and ensure that Immuta has control over what a user can access, the Delta Lake API is blocked. This reference guide outlines the Spark SQL options that can be substituted for the Delta Lake API.

• Spark direct file reads: Immuta allows direct file reads in Spark for file paths. This guide describes that process.

This guide details the manual installation method for enabling native access to Databricks with Immuta policies enforced. Before proceeding, ensure your Databricks workspace, instance, and permissions meet the guidelines outlined in the Installation Introduction.

Requirements

• If Databricks Unity Catalog is enabled in a Databricks workspace, you must use an Immuta cluster policy when you set up the Databricks Spark integration to create an Immuta-enabled cluster. See the Simplified Databricks Spark configuration guide for instructions.

• If Databricks Unity Catalog is not enabled in your Databricks workspace, you must disable Unity Catalog in your Immuta tenant before proceeding with your configuration of Databricks Spark:

  1. Navigate to the App Settings page and click Integration Settings.

  2. Uncheck the Enable Unity Catalog checkbox.

  3. Click Save.

1 - Download and Configure Immuta Artifacts

1. Navigate to the Immuta archives page. If you are prompted to log in and need basic authentication credentials, contact your Immuta support professional.

2. Navigate to the Databricks folder for the newest Immuta version. Ex: https://archives.immuta.com/hadoop/databricks/2024.2.1/.

3. Download the .jar file (Immuta plugin) as well as the other scripts listed below, which will load the plugin at cluster startup.

   Copy

   allowedCallingClasses.json
   immuta-benchmark-suite.dbc
   immuta-spark-hive-X.X.X_YYYYMMDD-hadoop-Z.Z.Z-public.jar
   immuta_cluster_init_script.sh
   obscuredCommands.yaml

   The immuta-benchmark-suite.dbc is a collection of notebooks packaged as a .dbc file. After you have added cluster policies to your cluster, you can import this file into Databricks to run performance tests and compare a regular Databricks cluster to one protected by Immuta. Detailed instructions are available in the first notebook, which will require an Immuta and non-Immuta cluster to generate test data and perform queries.

4. Specify the following properties as Spark environment variables or in the optional immuta_conf.xml file. If the same property is specified in both locations, the Spark environment variable takes precedence. The variable names are the config names in all upper case with _ instead of .. For example, to set the value of immuta.base.url via an environment variable, you would set the following in the Environment Variables section of cluster configuration: IMMUTA_BASE_URL=https://immuta.mycompany.com

   • immuta.system.api.key: Obtain this value from the Immuta Configuration UI under HDFS > System API Key. You will need to be a user with the APPLICATION_ADMIN role to complete this action.

   • immuta.base.url: The full URL for the target Immuta tenant Ex: https://immuta.mycompany.com.

   • immuta.user.mapping.iamid: If users authenticate to Immuta using an IAM different from Immuta's built-in IAM, you need to update the configuration file to reflect the ID of that IAM. The IAM ID is shown within the Immuta App Settings page within the Identity Management section. See Databricks to Immuta User Mapping for more details.

2 - Stage Immuta Artifacts

When configuring the Databricks cluster, a path will need to be provided to each of the artifacts downloaded/created in the previous step. To do this, those artifacts must be hosted somewhere that your Databricks instance can access. The following methods can be used for this step:

• Host files in AWS/S3 and provide access by the cluster

• Host files in Azure ADL Gen 1 or Gen 2 and provide access by the cluster

• Host files on an HTTPS server accessible by the cluster

• Host files in DBFS (Not recommended for production)

These artifacts will be downloaded to the required location within the clusters file-system by the init script downloaded in the previous step. In order for the init script to find these files, a URI will have to be provided through environment variables configured on the cluster. Each method's URI structure and setup is explained below.

AWS/S3

URI Structure: s3://[bucket]/[path]

1. Create an instance profile for clusters by following Databricks documentation.

2. Upload the configuration file, JSON file, and JAR file to an S3 bucket that the role from step 1 has access to.

Authenticating with Access Keys or Session Tokens (Optional)

If you wish to authenticate using access keys, add the following items to the cluster's environment variables:

IMMUTA_INIT_AWS_SECRET_ACCESS_KEY=<aws secret key>
IMMUTA_INIT_AWS_ACCESS_KEY_ID=<aws access key id>

If you've assumed a role and received a session token, that can be added here as well:

IMMUTA_INIT_AWS_SESSION_TOKEN=<aws session token>

Azure

ADL Gen 2

URI Structure: abfs(s)://[container]@[account].dfs.core.windows.net/[path]

Upload the configuration file, JSON file, and JAR file to an ADL gen 2 blob container.

Environment Variables:

If you want to authenticate using an account key, add the following to your cluster's environment variables:

IMMUTA_INIT_AZCOPY_CRED_TYPE=SharedKey
IMMUTA_INIT_ACCOUNT_NAME=<ADLg2 account name>
IMMUTA_INIT_ACCOUNT_KEY=<ADLg2 account key>

If you want to authenticate using an Azure SAS token, add the following to your cluster's environment variables:

IMMUTA_INIT_AZURE_SAS_TOKEN=<SAS token>

ADL Gen 1

URI Structure: adl://[account].azuredatalakestore.net/[path]

Upload the configuration file, JSON file, and JAR file to ADL gen 1.

Environment Variables:

If authenticating as a Microsoft Entra ID user,

IMMUTA_INIT_AZURE_AD_USER=<Microsoft Entra ID username>
IMMUTA_INIT_AZURE_PASSWORD=<Microsoft Entra ID password>

If authenticating using a service principal,

IMMUTA_INIT_AZURE_SERVICE_PRINCIPAL=<azure service principal>
IMMUTA_INIT_AZURE_PASSWORD=<azure service principal password>
IMMUTA_INIT_AZURE_TENANT=<tenant ID where principal was created>

HTTPS

URI Structure: http(s)://[host](:port)/[path]

Artifacts are available for download from Immuta using basic authentication. Your basic authentication credentials can be obtained from your Immuta support professional.

Environment Variables (Optional)

IMMUTA_INIT_HTTPS_USER=<basic auth username>
IMMUTA_INIT_HTTPS_PASSWORD=<basic auth password>

# Note: Credentials can also be included as part of the artifact URI. For example,
IMMUTA_INIT_JAR_URI=https://user:password@archives.immuta.com/path/to/file

DBFS

URI Structure: dbfs:/[path]

Upload the artifacts directly to DBFS using the Databricks CLI.

Since any user has access to everything in DBFS:

1. The artifacts can be stored anywhere in DBFS.

2. It's best to have a cluster-specific place for your artifacts in DBFS if you are testing to avoid overwriting or reusing someone else's artifacts accidentally.

3 - Protect Immuta Environment Variables with Databricks Secrets

It is important that non-administrator users on an Immuta-enabled Databricks cluster do not have access to view or modify Immuta configuration or the immuta-spark-hive.jar file, as this would potentially pose a security loophole around Immuta policy enforcement. Therefore, use Databricks secrets to apply environment variables to an Immuta-enabled cluster in a secure way.

Databricks secrets can be used in the Environment Variables configuration section for a cluster by referencing the secret path rather than the actual value of the environment variable. For example, if a user wanted to make the following value secret

MY_SECRET_ENV_VAR=super_secret_stuff

they could instead create a Databricks secret and reference it as the value of that variable. For instance, if the secret scope my_secrets was created, and the user added a secret with the key my_secret_env_var containing the desired sensitive environment variable, they would reference it in the Environment Variables section:

MY_SECRET_ENV_VAR={{secrets/my_secrets/my_secret_env_var}}

Then, at runtime, {{secrets/my_secrets/my_secret_env_var}} would be replaced with the actual value of the secret if the owner of the cluster has access to that secret.

4 - Create and Configure the Cluster

Cluster creation in an Immuta-enabled organization or Databricks workspace should be limited to administrative users to avoid allowing users to create non-Immuta enabled clusters.

1. Create a cluster in Databricks by following the Databricks documentation.

2. Select the Custom Access mode.

3. Opt to adjust the Autopilot Options and Worker Type settings. The default values provided here may be more than what is necessary for non-production or smaller use-cases. To reduce resource usage you can enable/disable autoscaling, limit the size and number of workers, and set the inactivity timeout to a lower value.

4. In the Advanced Options section, click the Instances tab.

   • IAM Role (AWS ONLY): Select the instance role you created for this cluster. (For access key authentication, you should instead use the environment variables listed in the AWS section.)

5. Click the Spark tab. In Spark Config field, add your configuration.

   • Cluster Configuration Requirements:

     Copy

     spark.executor.extraJavaOptions -Djava.security.manager=com.immuta.security.ImmutaSecurityManager /
         -Dimmuta.security.manager.classes.config=file:///databricks/immuta/allowedCallingClasses.json /
         -Dimmuta.spark.encryption.fpe.class=com.immuta.spark.encryption.ff1.ImmutaFF1Service
     spark.driver.extraJavaOptions -Djava.security.manager=com.immuta.security.ImmutaSecurityManager /
         -Dimmuta.security.manager.classes.config=file:///databricks/immuta/allowedCallingClasses.json /
         -Dimmuta.spark.encryption.fpe.class=com.immuta.spark.encryption.ff1.ImmutaFF1Service
     spark.databricks.repl.allowedLanguages python,sql
     spark.databricks.pyspark.enableProcessIsolation true
     spark.databricks.isv.product Immuta

6. In the Environment Variables section, add the environment variables necessary for your configuration. Remember that these variables should be protected with Databricks secrets as mentioned above.

   Copy

   # Specify the URI to the artifacts that were hosted in the previous steps
   # The URI must adhere to the supported types for each service mentioned above
   IMMUTA_INIT_JAR_URI=<Full URI to immuta-spark-hive.jar>
   IMMUTA_INIT_CONF_URI=<Full URI to Immuta configuration file>
   IMMUTA_INIT_ALLOWED_CALLING_CLASSES_URI=<full URI to allowedCallingClasses.json>
   IMMUTA_INIT_OBSCURED_COMMANDS_URI=<full URI to obscuredCommands.yaml>
   
   # (OPTIONAL)
   # Specify an additional configuration file to be added to the spark.sparkContext.hadoopConfiguration.
   # This file allows administrators to add sensitive configuration needed by the SparkSession that
   # should not viewable by users.
   # Further explanation of this variable as well as examples are provided below.
   IMMUTA_INIT_ADDITIONAL_CONF_URI=<full URI to additional configuration file>

7. Click the Init Scripts tab and set the following configurations:

   • Destination: Specify the service you used to host the Immuta artifacts.

   • File Path: Specify the full URI to the immuta_cluster_init_script.sh.

   • Add the new key/value to the configuration.

8. Click the Permissions tab and configure the following setting:

   • Who has access: Users or groups will need to have the permission Can Attach To to execute queries against Immuta configured data sources.

9. (Re)start the cluster.

Additional Hadoop Configuration File (Optional)

As mentioned in the "Environment Variables" section of the cluster configuration, there may be some cases where it is necessary to add sensitive configuration to SparkSession.sparkContext.hadoopConfiguration in order to read the data composing Immuta data sources.

As an example, when accessing external tables stored in Azure Data Lake Gen 2, Spark must have credentials to access the target containers/filesystems in ADLg2, but users must not have access to those credentials. In this case, an additional configuration file may be provided with a storage account key that the cluster may use to access ADLg2.

To use an additional Hadoop configuration file, you will need to set the IMMUTA_INIT_ADDITIONAL_CONF_URI environment variable referenced in the Create and configure the cluster section to be the full URI to this file.

The additional configuration file looks very similar to the Immuta Configuration file referenced above. Some example configuration files for accessing different storage layers are below.

Amazon S3

<configuration>
    <property>
        <name>fs.s3n.awsAccessKeyId</name>
        <value>[AWS access key ID]</value>
    </property>
    <property>
        <name>fs.s3n.awsSecretAccessKey</name>
        <value>[AWS secret key]</value>
    </property>
</configuration>

Azure Data Lake Gen 2

<configuration>
    <property>
        <name>fs.azure.account.key.[storage account name].dfs.core.windows.net</name>
        <value>[storage account key]</value>
    </property>
</configuration>

Azure Data Lake Gen 1

<configuration>
    <property>
        <name>fs.adl.oauth2.refresh.url</name>
        <value>https://login.microsoftonline.com/[directory ID]/oauth2/token</value>
    </property>
    <property>
        <name>fs.adl.oauth2.access.token.provider.type</name>
        <value>ClientCredential</value>
    </property>
    <property>
        <name>fs.adl.oauth2.credential</name>
        <value>[client secret from Azure]</value>
    </property>
    <property>
        <name>fs.adl.oauth2.client.id</name>
        <value>[client ID from Azure]</value>
    </property>
</configuration>

Azure Blob Storage

<configuration>
    <property>
        <name>fs.azure.account.key.[storage account name].blob.core.windows.net</name>
        <value>[storage account key]</value>
    </property>
</configuration>

5 - Register data

Register Databricks securables in Immuta.

6 - Query Immuta Data

When the Immuta enabled Databricks cluster has been successfully started, users will see a new database labeled "immuta". This database is the virtual layer provided to access data sources configured within the connected Immuta tenant.

Before users can query an Immuta data source, an administrator must give the user Can Attach To permissions on the cluster and GRANT the user access to the immuta database.

The following SQL query can be run as an administrator within a journal to give the user access to "Immuta":

%sql
GRANT SELECT,READ_METADATA ON DATABASE immuta TO `user@company.com`

Below are example queries that can be run to obtain data from an Immuta-configured data source. Because Immuta supports raw tables in Databricks, you do not have to use Immuta-qualified table names in your queries like the first example. Instead, you can run queries like the second example, which does not reference the immuta database.

%sql
select * from immuta.my_data_source limit 5;

%sql
select * from my_data_source limit 5;

Creating a Databricks Data Source

See the Databricks Data Source Creation guide for a detailed walkthrough.

Databricks to Immuta User Mapping

By default, the IAM used to map users between Databricks and Immuta is the BIM (Immuta's internal IAM). The Immuta Spark plugin will check the Databricks username against the username within the BIM to determine access. For a basic integration, this means the users email address in Databricks and the connected Immuta tenant must match.

It is possible within Immuta to have multiple users share the same username if they exist within different IAMs. In this case, the cluster can be configured to lookup users from a specified IAM. To do this, the value of immuta.user.mapping.iamid created and hosted in the previous steps must be updated to be the targeted IAM ID configured within the Immuta tenant. The IAM ID can be found on the App Settings page. Each Databricks cluster can only be mapped to one IAM.

This guide details the simplified installation method for enabling native access to Databricks with Immuta policies enforced.

Ensure your Databricks workspace, instance, and permissions meet the guidelines outlined in the Installation introduction before you begin.

Requirements

• If Databricks Unity Catalog is enabled in a Databricks workspace, you must use an Immuta cluster policy when you set up the Databricks Spark integration to create an Immuta-enabled cluster. See the configure cluster policies section below for guidance.

• If Databricks Unity Catalog is not enabled in your Databricks workspace, you must disable Unity Catalog in your Immuta tenant before proceeding with your configuration of Databricks Spark:

  1. Navigate to the App Settings page and click Integration Settings.

  2. Uncheck the Enable Unity Catalog checkbox.

  3. Click Save.

1 - Add the Integration on the App Settings Page

1. Log in to Immuta and click the App Settings icon in the left sidebar.

2. Scroll to the System API Key subsection under HDFS and click Generate Key.

3. Click Save and then Confirm.

4. Scroll to the Integration Settings section.

5. Click + Add Native Integration and select Databricks Integration from the dropdown menu.

6. Complete the Hostname field.

7. Enter a Unique ID for the integration. By default, your Immuta tenant URL populates this field. This ID is used to tie the set of cluster policies to your Immuta tenant and allows multiple Immuta tenants to access the same Databricks workspace without cluster policy conflicts.

8. Select your configured Immuta IAM from the dropdown menu.

9. Choose one of the following options for your data access model:

   • Protected until made available by policy: All tables are hidden until a user is permissioned through an Immuta policy. This is how most databases work and assumes least privileged access and also means you will have to register all tables with Immuta.

   • Available until protected by policy: All tables are open until explicitly registered and protected by Immuta. This makes a lot of sense if most of your tables are non-sensitive and you can pick and choose which to protect.

10. Select the Storage Access Type from the dropdown menu.

11. Opt to add any Additional Hadoop Configuration Files.

12. Click Add Native Integration.

2 - Configure Cluster Policies

Several cluster policies are available on the App Settings page when configuring this integration:

• Python & SQL

• Python & SQL & R

• Python & SQL & R with Library Support

• Scala

• Sparklyr

Click a link above to read more about each of these cluster policies before continuing with the tutorial.

1. Click Configure Cluster Policies.

2. Select one or more cluster policies in the matrix by clicking the Select button(s).

3. Opt to check the Enable Unity Catalog checkbox to generate cluster policies that will enable Unity Catalog on your cluster. This option is only available when Databricks runtime 11.3 is selected.

4. Opt to make changes to these cluster policies by clicking Additional Policy Changes and editing the text field.

5. Use one of the two Installation Types described below to apply the policies to your cluster:

   • Automatically push cluster policies: This option allows you to automatically push the cluster policies to the configured Databricks workspace. This will overwrite any cluster policy templates previously applied to this workspace.

     1. Select the Automatically Push Cluster Policies radio button.

     2. Enter your Admin Token. This token must be for a user who can create cluster policies in Databricks.

     3. Click Apply Policies.

   • Manually push cluster policies: Enabling this option will allow you to manually push the cluster policies to the configured Databricks workspace. There will be various files to download and manually push to the configured Databricks workspace.

     1. Select the Manually Push Cluster Policies radio button.

     2. Click Download Init Script.

     3. Follow the steps in the Instructions to upload the init script to DBFS section.

     4. Click Download Policies, and then manually add these Cluster Policies in Databricks.

6. Opt to click the Download the Benchmarking Suite to compare a regular Databricks cluster to one protected by Immuta. Detailed instructions are available in the first notebook, which will require an Immuta and non-Immuta cluster to generate test data and perform queries.

7. Click Close, and then click Save and Confirm.

3 - Add Policies to Your Cluster

1. Create a cluster in Databricks by following the Databricks documentation.

2. In the Policy dropdown, select the Cluster Policies you pushed or manually added from Immuta.

3. Select the Custom Access mode.

4. Opt to adjust Autopilot Options and Worker Type settings: The default values provided here may be more than what is necessary for non-production or smaller use-cases. To reduce resource usage you can enable/disable autoscaling, limit the size and number of workers, and set the inactivity timeout to a lower value.

5. Opt to configure the Instances tab in the Advanced Options section:

   • IAM Role (AWS ONLY): Select the instance role you created for this cluster. (For access key authentication, you should instead use the environment variables listed in the AWS section.)

6. Click Create Cluster.

4 - Register data

Register Databricks securables in Immuta.

5 - Query Immuta Data

When the Immuta-enabled Databricks cluster has been successfully started, Immuta will create an immuta database, which allows Immuta to track Immuta-managed data sources separately from remote Databricks tables so that policies and other security features can be applied. However, users can query sources with their original database or table name without referencing the immuta database. Additionally, when configuring a Databricks cluster you can hide immuta from any calls to SHOW DATABASES so that users aren't misled or confused by its presence. For more details, see the Hiding the immuta Database in Databricks page.

1. Before users can query an Immuta data source, an administrator must give the user Can Attach To permissions on the cluster.

2. See the Databricks Data Source Creation guide for a detailed walkthrough of creating Databricks data sources in Immuta.

Example Queries

Below are example queries that can be run to obtain data from an Immuta-configured data source. Because Immuta supports raw tables in Databricks, you do not have to use Immuta-qualified table names in your queries like the first example. Instead, you can run queries like the second example, which does not reference the immuta database.

%sql
select * from immuta.my_data_source limit 5;

%sql
select * from my_data_source limit 5;

While you're onboarding Snowflake data sources and designing policies, you don't want to disrupt your Snowflake users' existing workflows. Instead, you want to gradually onboard Immuta through a series of successive changes that will not impact your existing Snowflake users.

A phased onboarding approach to configuring the Snowflake integration ensures that your users will not be immediately affected by changes as you add data sources and configure policies.

Several features allow you to gradually onboard data sources and policies in Immuta:

• Subscription policy of “None” by default: By default, no policy is applied at registration time; instead of applying a restrictive policy immediately upon registration, the table is registered in Immuta and waits for a policy to be applied, if ever.

  There are several benefits to this design:

  • All existing roles maintain access to the data and registration of the table or view with Immuta has zero impact on your data platform.

  • It gives you time to configure tags on the Immuta registered tables and views, either manually or through automatic means, such as Immuta’s sensitive data detection (SDD), or an external catalog integration to include Snowflake tags.

  • It gives you time to assess and validate the sensitive data tags that were applied.

  • You can build only row and column controls with Immuta and let your existing roles manage table access instead of using Immuta subscription policies for table access.

• Snowflake table grants coupled with Snowflake low row access policy mode: With these features enabled, Immuta manages access to tables (subscription policies) through GRANTs. This works by assigning each user their own unique role created by Immuta and all table access is managed using that single role.

  Without these two features enabled, Immuta uses a Snowflake row access policy (RAP) to manage table access. A RAP only allows users to access rows in the table if they were explicitly granted access through an Immuta subscription policy; otherwise, the user sees no rows. This behavior means all existing Snowflake roles lose access to the table contents until explicitly granted access through Immuta subscription policies. Essentially, roles outside of Immuta don't control access anymore.

  By using table grants and the low row access policy mode, users and roles outside Immuta continue to work.

  There are two benefits to this approach:

  • All pre-existing Snowflake roles retain access to the data until you explicitly revoke access (outside Immuta).

  • It provides a way to test that Immuta GRANTs are working without impacting production workloads.

Requirements

The following configuration is required for phased Snowflake onboarding:

• Impersonation is disabled

• Project workspaces are disabled

If either of these capabilities is necessary for your use case, you cannot do phased Snowflake onboarding as described below.

Next

See the Getting started page for step-by-step guidance to implement phased Snowflake onboarding.

This integration allows you to manage and access data in your Databricks account across all of your workspaces. With Immuta’s Databricks Unity Catalog integration, you can write your policies in Immuta and have them enforced automatically by Databricks across data in your Unity Catalog metastore.

Getting started

This getting started guide outlines how to integrate Databricks Unity Catalog with Immuta.

How-to guides

• Databricks Unity Catalog configuration: Configure the Databricks Unity Catalog integration.

• Migrate to Databricks Unity Catalog: Migrate from the legacy Databricks Spark integrations to the Databricks Unity Catalog integration.

Reference guide

Databricks Unity Catalog integration reference guide: This guide describes the design and components of the integration.

When you enable Unity Catalog, Immuta automatically migrates your existing Databricks data sources in Immuta to reference the legacy hive_metastore catalog to account for Unity Catalog's three-level hierarchy. New data sources will reference the Unity Catalog metastore you create and attach to your Databricks workspace.

Because the hive_metastore catalog is not managed by Unity Catalog, existing data sources in the hive_metastore cannot have Unity Catalog access controls applied to them. Data sources in the Hive Metastore must be managed by the Databricks Spark integration.

To allow Immuta to administer Unity Catalog access controls on that data, move the data to Unity Catalog and re-register those tables in Immuta by completing the steps below. If you don't move all data before configuring the integration, metastore magic will protect your existing data sources throughout the migration process.

1. Ensure that all Databricks clusters that have Immuta installed are stopped and the Immuta configuration is removed from the cluster. Immuta-specific cluster configuration is no longer needed with the Databricks Unity Catalog integration.

2. Move all data into Unity Catalog before configuring Immuta with Unity Catalog. Existing data sources will need to be re-created after they are moved to Unity Catalog and the Unity Catalog integration is configured.

3. Enable Unity Catalog.

The immuta database on Immuta-enabled clusters allows Immuta to track Immuta-managed data sources separately from remote Databricks tables so that policies and other security features can be applied. However, Immuta supports raw tables in Databricks, so table-backed queries do not need to reference this database. When configuring a Databricks cluster, you can hide immuta from any calls to SHOW DATABASES so that users are not confused or misled by that database.

Hide the immuta Database

When configuring a Databricks cluster, hide immuta by using the following environment variable in the Spark cluster configuration:

IMMUTA_SPARK_SHOW_IMMUTA_DATABASE=false

This page outlines how to access DBFS in Databricks for non-sensitive data. Databricks Administrators should place the desired configuration in the Spark environment variables (recommended) or the immuta_conf.xml file (not recommended).

DBFS FUSE Mount

This feature (provided by Databricks) mounts DBFS to the local cluster filesystem at /dbfs. Although disabled when using process isolation, this feature can safely be enabled if raw, unfiltered data is not stored in DBFS and all users on the cluster are authorized to see each other’s files. When enabled, the entirety of DBFS essentially becomes a scratch path where users can read and write files in /dfbs/path/to/my/file as though they were local files.

For example,

%sh echo "I'm creating a new file in DBFS" > /dbfs/my/newfile.txt

In Python,

%python
with open("/dbfs/my/newfile.txt", "w") as f:
  f.write("I'm creating a new file in DBFS")

Note: This solution also works in R and Scala.

Enable DBFS FUSE Mount

To enable the DBFS FUSE mount, set this configuration: immuta.spark.databricks.dbfs.mount.enabled=true.

Scala DBUtils (and %fs magic) with Scratch Paths

Scratch paths will work when performing arbitrary remote filesystem operations with fs magic or Scala dbutils.fs functions. For example,

%fs put -f s3://my-bucket/my/scratch/path/mynewfile.txt "I'm creating a new file in S3"
%scala dbutils.fs.put("s3://my-bucket/my/scratch/path/mynewfile.txt", "I'm creating a new file in S3")

Configure Scala DBUtils (and %fs magic) with Scratch Paths

To support %fs magic and Scala DBUtils with scratch paths, configure

       <property>
           <name>immuta.spark.databricks.scratch.paths</name>
           <value>s3://my-bucket/my/scratch/path</value>
       </property>

Configure DBUtils in Python

To use dbutils in Python, set this configuration: immuta.spark.databricks.py4j.strict.enabled=false.

Example Workflow

This section illustrates the workflow for getting a file from a remote scratch path, editing it locally with Python, and writing it back to a remote scratch path.

%python
import os
import shutil

s3ScratchFile = "s3://some-bucket/path/to/scratch/file"
localScratchDir = os.environ.get("IMMUTA_LOCAL_SCRATCH_DIR")
localScratchFile = "{}/myfile.txt".format(localScratchDir)
localScratchFileCopy = "{}/myfile_copy.txt".format(localScratchDir)