This section illustrates how to install Immuta on Kubernetes using the Immuta Enterprise Helm chart.

Requirements

This reference guide provides an overview of the Immuta Enterprise Helm chart version requirements and infrastructure recommendations.

Install

The guides in this section illustrate how to install and deploy Immuta in your Kubernetes environment.

Upgrade

The guides in this section illustrate how to upgrade Immuta.

Guides

The guides in this section illustrate how to configure your Immuta Enterprise Helm chart for various scenarios, including optimizing your deployment for production environments.

Disaster recovery

This guide provides links to additional resources for disaster recovery strategies.

Troubleshooting

This page provides troubleshooting guidance and outlines frequently asked questions for the Immuta installation.

Conventions

This page introduces the core concepts and terminology essential for understanding the installation material.

Immuta gives everyone fast, governed access to data with the built-in controls, collaboration workflows, automated provisioning, and continuous monitoring you need to keep risk low and compliance high

Configure Immuta

Explore Immuta

In this integration, Immuta generates policy-enforced views in a schema in your configured Azure Synapse Analytics Dedicated SQL pool for tables registered as Immuta data sources.

Getting started

This guide outlines how to integrate Azure Synapse Analytics with Immuta.

How-to guide

Azure Synapse Analytics configuration: Configure the integration in Immuta.

Reference guides

• Azure Synapse Analytics integration reference guide: This guide describes the design and components of the integration.

• Azure Synapse Analytics pre-configuration details: This guide describes the prerequisites, supported features, and limitations of the integration.

If a Databricks cluster needs to be manually updated to reflect changes in the Immuta init script or cluster policies, you can remove and set up your integration again to get the updated policies and init script.

1. Log in to Immuta as an Application Admin.

2. Click the App Settings icon in the left sidebar and scroll to the Integration Settings section.

3. Your existing Databricks Spark integration should be listed here; expand it and note the configuration values. Now select Remove to remove your integration.

4. Click Add Integration and select Databricks Integration to add a new integration.

5. Enter your Databricks Spark integration settings again as configured previously.

6. Click Add Integration to add the integration, and then select Configure Cluster Policies to set up the updated cluster policies and init script.

7. Select the cluster policies you wish to use for your Immuta-enabled Databricks clusters.

8. Automatically push cluster policies and the init script (recommended) or manually update your cluster policies.

   • Automatically push cluster policies

     1. Select Automatically Push Cluster Policies and enter your privileged Databricks access token. This token must have privileges to write to cluster policies.

     2. Select Apply Policies to push the cluster policies and init script again.

     3. Click Save and Confirm to deploy your changes.

   • Manually update cluster policies

     1. Download the init script and the new cluster policies to your local computer.

     2. Click Save and Confirm to save your changes in Immuta.

     3. Log in to your Databricks workspace with your administrator account to set up cluster policies.

     4. Get the path you will upload the init script (`immuta_cluster_init_script_proxy.sh`) to by opening one of the cluster policy `.json` files and looking for the `defaultValue` of the field `init_scripts.0.dbfs.destination`. This should be a DBFS path in the form of `dbfs:/immuta-plugin/hostname/immuta_cluster_init_script_proxy.sh`.

     5. Click Data in the left pane to upload your init script to DBFS to the path you found above.

     6. To find your existing cluster policies you need to update, click Compute in the left pane and select the Cluster policies tab.

     7. Edit each of these cluster policies that were configured before and overwrite the contents of the JSON with the new cluster policy JSON you downloaded.

9. Restart any Databricks clusters using these updated policies for the changes to take effect.

Introduced in 2024.2, the Immuta Enterprise Helm chart (IEHC) is an entirely new Helm chart used to deploy Immuta. Unlike the previous Immuta Helm chart (IHC), the IEHC shares the same version as the Immuta product. Each version of the chart supports a singular version of Immuta. Upgrading the Immuta version now entails upgrading the underlying Helm chart. Failure to do so will lead to an unsupported configuration.

Upgrade guides

• : If you're upgrading from 2024.1.x or older, you must first migrate to the new Helm chart, as these versions were all installed using the legacy IHC. This migration process will include upgrading Immuta.

• : Upgrade from v2024.2 LTS using the Immuta Enterprise Helm chart.

This guide demonstrates how to configure a private container registry with the Immuta Enterprise Helm chart (IEHC).

Helm values

1. Examine the default Helm values in the chart; this will include all relevant values required to override the registry and images.

2. Edit the immuta-values.yaml to include the following Helm values. Update all with your own values.

All application state is stored in the PostgreSQL metadata database; therefore, recovering from a disaster event only entails restoring the aforementioned PostgreSQL database. Consult each cloud provider's point-in-time recovery (PITR) documentation for guidance:

For more details about point-in-time recovery, see the .

This integration enforces policies on Databricks securables registered in the legacy Hive metastore. Once these securables are registered as Immuta data sources, users can query policy-enforced data on Databricks clusters.

The guides in this section outline how to integrate Databricks Spark with Immuta.

This getting started guide outlines how to integrate Databricks with Immuta.

How-to guides

• : Manually update your cluster to reflect changes in the Immuta init script or cluster policies.

• : Register a Databricks library with Immuta as a trusted library to avoid Immuta Security Manager errors when using third-party libraries.

• : Raise the caching on-cluster and lower the cache timeouts for the Immuta web service to allow use of project UDFs in Spark jobs.

• : Run R and Scala spark-submit jobs on your Databricks cluster.

• : Access DBFS in Databricks for non-sensitive data.

• : Resolve errors in the Databricks Spark configuration.

Reference guides

• : This guide describes the design and components of the integration.

• : This guide provides an overview of the Immuta features that provide security for your users and Databricks clusters and that allow you to prove compliance and monitor for anomalies.

• : This guide provides an overview of registering Databricks securables and protecting them with Immuta policies.

• : This guide provides an overview of how Databricks users access data registered in Immuta.

This page outlines the configuration for setting up project UDFs, which allow users to set their current project in Immuta through Spark. For details about the specific functions available and how to use them, see the .

1. Lower the web service cache timeout in Immuta:

   1. Click the App Settings icon and scroll to the HDFS Cache Settings section.

   2. Lower the Cache TTL of HDFS user names (ms) to 0.

   3. Click Save.

2. Raise the cache timeout on your Databricks cluster: In the Spark environment variables section, set the IMMUTA_CURRENT_PROJECT_CACHE_TIMEOUT_SECONDS and IMMUTA_PROJECT_CACHE_TIMEOUT_SECONDS to high values (like 10000).

   Note: These caches will be invalidated on cluster when a user calls immuta.set_current_project, so they can effectively be cached permanently on cluster to avoid periodically reaching out to the web service.

1. Navigate to the App Settings page.

2. Scroll to the Global Integrations Settings section.

3. Opt to change the Role Prefix. Snowflake table grants creates a new Snowflake role for each Immuta user. To ensure these Snowflake role names do not collide with existing Snowflake roles, each Snowflake role created for Snowflake table grants requires a common prefix. When using multiple Immuta accounts within a single Snowflake account, the Snowflake table grants role prefix should be unique for each Immuta account. The prefix must adhere to and be less than 50 characters. Once the configuration is saved, the prefix cannot be modified; however, the Snowflake table grants feature can be disabled and re-enabled to change the prefix.

4. Finish configuring your integration by following one of these guidelines:

   • New Snowflake integration: Set up a new Snowflake integration by following the .

   • Existing Snowflake integration (automatic setup): You will be prompted to enter connection information for a Snowflake user. Immuta will execute the migration to Snowflake table grants using a connection established with this Snowflake user. The Snowflake user you provide here must have Snowflake privileges to run these .

   • Existing Snowflake integration (manual setup): Immuta will display a link to a migration script you must run in Snowflake and a link to a rollback script for use in the event of a failed migration. Important: Execute the migration script in Snowflake before clicking Save on the app settings page.

1. Click the App Settings icon in the sidebar and scroll to the Global Integration Settings section.

2. Click the Enable Snowflake Low Row Access Policy Mode checkbox to enable the feature.

3. Confirm to allow Immuta to automatically disable impersonation for the Snowflake integration. If you do not confirm, you will not be able to enable Snowflake low row access policy mode.

4. Click Save.

Configure your Snowflake integration

If you already have a configured, you don't need to reconfigure your integration. Your Snowflake policies automatically refresh when you enable Snowflake low row access policy mode.

1. . Note that you will not be able to enable project workspaces or user impersonation with Snowflake low row access policy mode enabled.

2. Click Save and Confirm your changes.

This section includes how-to guides for Immuta's legacy integrations.

Immuta offers both fine- and coarse-grained protection for Hive and Impala tables. However, additional protections are required to ensure that users cannot gain unauthorized access to data by connecting to Hive or Impala directly. Cloudera recommends using the Sentry service to secure access to Hive and Impala. As an alternative, this guide details steps that CDH cluster administrators can take to lock down Hive and Impala access without running the Sentry service.

Hadoop has the concept of a Group Mapping Service, which is a way to retrieve groups corresponding to a provided user/Kerberos principal. By default, Hadoop services (HDFS, Hive, Impala, etc.) retrieve a user's groups from the local OS by way of the ShellBasedUnixGroupsMapping class. This guide shows you how to enrich this data to include the user's current project in Immuta.

The guides in this section illustrate how to install and deploy Immuta in your Kubernetes environment.

Prerequisites

Checklist

• RBAC permissions have been granted to create resources in the cluster.

Quickstart

Get started quickly with these essential guides. For a more comprehensive understanding and advanced configurations, explore the full suite of guides.

1. Complete the guide that corresponds with your Kubernetes cluster's distribution.

   • Managed public cloud: This guide includes instructions for

     • Amazon Elastic Kubernetes Service (EKS)

     • Google Kubernetes Engine (GKE)

     • Microsoft Azure Kubernetes Service (AKS)

   • Red Hat OpenShift

2. Complete the Ingress configuration guide.

3. Complete the Production best practices guide.

The following guides offer practical guidance for handling common challenges and configurations.

Ingress configuration

Configure Ingress to complete your installation and access your Immuta application.

TLS configuration

Configure TLS termination for an Ingress resource.

Cosign verification

Verify artifacts hosted on the ocir.immuta.com OCI registry.

Production best practices

Follow these best practices when deploying Immuta in your production environment.

Rotating credentials

Update the credentials referenced in the Immuta Enterprise Helm chart.

External cache configuration

Configure an external key-value cache (such as Redis or Memcached) with the Immuta Enterprise Helm chart.

Enabling legacy query engine

Enable this legacy service for your deployment if you are using any of the legacy data platforms.

Private container registries

Configure pulling images from a private registry.

Air-gapped environments

Tips when installing Immuta without internet access.

This guide demonstrates how to verify signed artifacts (i.e., container images, Helm charts) hosted on ocir.immuta.com using Cosign from Sigstore.

Download public key

The provided key is used to sign the Helm chart and container images.

Identify container images

A DIGESTS.md markdown file comes bundled in the Helm chart and contains a comprehensive list of images and digests referenced. To view the file, follow these steps:

1. Download and extract the Helm chart into the working directory.

   Copy

   helm pull oci://ocir.immuta.com/stable/immuta-enterprise --destination . --untar --version 2025.1.6

2. Open file immuta-enterprise/DIGESTS.md

Verify signature

Verify an artifact's signature by referencing Immuta's public key.

cosign verify --key ./immuta-cosign.pub <image>

This guide demonstrates how to configure an external key-value cache (such as Redis or Memcached) with the Immuta Enterprise Helm chart (IEHC).

Prerequisite

The Production best practices guide must be completed before proceeding.

Redis

1. Edit secret immuta-secret that was created in the Immuta in production guide.

   Copy

   kubectl edit secret/immuta-secret

2. Add key-value IMMUTA_SERVER_CACHE_PROVIDER_OPTIONS_PASSWORD=<cache-password>.

Edit Helm values

Edit the immuta-values.yaml file to include the relevant Helm values listed below. Update all placeholder values with your own values.

Redis

cache:
  enabled: false

secure:
  extraConfig:
    server:
      cache:
        provider:
          constructor: catbox-redis
          options:
            host: <redis-fqdn>
            port: <port>
            # Setting options.tls to an empty dict enables TLS without configuring any other options.
            tls: {}

            # Dict representation of TLS config options json-object for package ioredis
            # https://github.com/redis/ioredis
            #
            # tls:
            #   ca:
            #   key:
            #   cert:

  extraEnvVars:
  - name: IMMUTA_SERVER_CACHE_PROVIDER_OPTIONS_PASSWORD
    valueFrom:
      secretKeyRef:
        key: IMMUTA_SERVER_CACHE_PROVIDER_OPTIONS_PASSWORD
        name: immuta-secret

Memcached

cache:
  enabled: false

secure:
  extraConfig:
    server:
      cache:
        provider:
          constructor: catbox-memcached
          options:
            host: <memcached-fqdn>
            port: <port>

Apply Helm values

Perform a Helm upgrade to apply the changes made to immuta-values.yaml.

helm upgrade <release-name> oci://ocir.immuta.com/stable/immuta-enterprise --values immuta-values.yaml --version 2025.1.6

The following conventions are used throughout the installation material.

Angle brackets ( < and > )

Phrases wrapped in angle brackets (i.e., <, >) are placeholders used to indicate values that must be substituted with user-provided values. Placeholders are typically written in either kebab case, or snake case; the following placeholders are equivalent:

• <the-quick-brown-fox>

• <the_quick_brown_fox>

Example

Input

computerScientists:
- Alan Turing
- Grace Hopper
- Donald Knuth
- Tim Berners-Lee
- John McCarthy
- <first-name> <last-name>

Output

computerScientists:
- Alan Turing
- Grace Hopper
- Donald Knuth
- Tim Berners-Lee
- John McCarthy
- Margaret Hamilton

Immuta integrates with your data platforms and external catalogs so you can register your data and effectively manage access controls on that data.

This section includes concept, reference, and how-to guides for configuring your data platform integration, registering data sources, and connecting your external catalog so that you can discover, monitor, and protect sensitive data.

Immuta integrations

This reference guide outlines the features, policies, and audit capabilities supported by each integration.

Amazon S3

This page includes how-to and reference content for Amazon S3 and how it integrates with Immuta.

Azure Synapse Analytics

This section includes how-to and reference guides for Azure Synapse Analytics and how it integrates with Immuta.

Databricks Spark

This section includes how-to and reference guides for Databricks Spark and how it integrates with Immuta.

Databricks Unity Catalog

This section includes how-to and reference guides for Databricks Unity Catalog and how it integrates with Immuta.

Google BigQuery

This page includes how-to and reference content for Google BigQuery and how it integrates with Immuta.

Redshift

This section includes how-to and reference guides for Redshift and how it integrates with Immuta.

Snowflake

This section includes how-to and reference guides for Snowflake and how it integrates with Immuta.

Starburst (Trino)

This section includes how-to and reference guides for Starburst (Trino) and how it integrates with Immuta.

Queries Immuta runs in remote platforms

This reference guide outlines the actions and features that trigger Immuta queries in your remote platform that may incur cost.

Connect your data

Immuta integrates with your data platforms so you can register your data and effectively manage access controls on that data. This section includes concept, reference, and how-to guides for registering and managing data sources and your connections.

The how-to guides linked on this page illustrate how to integrate Azure Synapse Analytics with Immuta. See the reference guide for information about the Azure Synapse Analytics integration.

Requirement: A running Dedicated SQL pool

This page describes the Azure Synapse integration, configuration options, and features. See the Azure Synapse integration page for a tutorial on enabling the integration and these features through the app settings page.

Feature Availability

Prerequisite

A running Dedicated SQL pool

Authentication Method

Immuta only supports the SQL authentication option for Azure Synapse Analytics to configure the integration and create data sources. The Microsoft Entra authentication option is unsupported. See the SQL Authentication in Azure Synapse Analytics documentation for details.

Tag Ingestion

Immuta cannot ingest tags from Synapse, but you can connect any of these supported external catalogs to work with your integration.

User Impersonation

Impersonation allows users to query data as another Immuta user in Synapse. To enable user impersonation, see the User Impersonation page.

Multiple Integrations

A user can configure multiple integrations of Synapse to a single Immuta tenant.

Limitations

• Immuta does not support the following masking types in this integration because of limitations with Dedicated SQL pools (linked below). Any column assigned one of these masking types will be masked to NULL:

  • Reversible Masking: Synapse UDFs currently only support SQL, but Immuta needs to execute code (such as JavaScript or Python) to support this masking feature. See the Synapse Documentation for details.

  • Format Preserving Masking: Synapse UDFs currently only support SQL, but Immuta needs to execute code (such as JavaScript or Python) to support this masking feature. See the Synapse Documentation for details.

  • Regex: The built in string replace function does not support full regex. See the Synapse Documentation for details.

• The delimiters configured when enabling the integration cannot be changed once they are set. To change the delimiters, the integration has to be disabled and re-enabled.

• If the generated view name is more than 128 characters, then the view name is shortened to 128 characters. This could cause collisions between view names if the shortened version is the same for two different data sources.

• For proper updates, the Dedicated SQL pools have to be running when changes are made to users or data sources in Immuta.

This page outlines how to enable access to DBFS in Databricks for non-sensitive data. Databricks administrators should place the desired configuration in the Spark environment variables.

DBFS FUSE mount

This Databricks feature mounts DBFS to the local cluster filesystem at /dbfs. Although disabled when using process isolation, this feature can safely be enabled if raw, unfiltered data is not stored in DBFS and all users on the cluster are authorized to see each other’s files. When enabled, the entirety of DBFS essentially becomes a scratch path where users can read and write files in /dfbs/path/to/my/file as though they were local files.

For example,

%sh echo "I'm creating a new file in DBFS" > /dbfs/my/newfile.txt

In Python,

%python
with open("/dbfs/my/newfile.txt", "w") as f:
  f.write("I'm creating a new file in DBFS")

Note: This solution also works in R and Scala.

Enable DBFS FUSE mount

To enable the DBFS FUSE mount, set this configuration in the Spark environment variables: IMMUTA_SPARK_DATABRICKS_DBFS_MOUNT_ENABLED=true.

Scala DBUtils (and %fs magic) with scratch paths

Scratch paths will work when performing arbitrary remote filesystem operations with fs magic or Scala dbutils.fs functions. For example,

%fs put -f s3://my-bucket/my/scratch/path/mynewfile.txt "I'm creating a new file in S3"
%scala dbutils.fs.put("s3://my-bucket/my/scratch/path/mynewfile.txt", "I'm creating a new file in S3")

Configure Scala DBUtils (and %fs magic) with scratch paths

To support %fs magic and Scala DBUtils with scratch paths, configure

<property>
   <name>immuta.spark.databricks.scratch.paths</name>
   <value>s3://my-bucket/my/scratch/path</value>
</property>

Configure DBUtils in Python

To use dbutils in Python, set this configuration: immuta.spark.databricks.py4j.strict.enabled=false.

Example workflow

This section illustrates the workflow for getting a file from a remote scratch path, editing it locally with Python, and writing it back to a remote scratch path.

%python
import os
import shutil

s3ScratchFile = "s3://some-bucket/path/to/scratch/file"
localScratchDir = os.environ.get("IMMUTA_LOCAL_SCRATCH_DIR")
localScratchFile = "{}/myfile.txt".format(localScratchDir)
localScratchFileCopy = "{}/myfile_copy.txt".format(localScratchDir)

1. Get the file from remote storage:

   Copy

   dbutils.fs.cp(s3ScratchFile, "file://{}".format(localScratchFile))

2. Make a copy if you want to explicitly edit localScratchFile, as it will be read-only and owned by root:

   Copy

   shutil.copy(localScratchFile, localScratchFileCopy)
   with open(localScratchFileCopy, "a") as f:
       f.write("Some appended file content")

3. Write the new file back to remote storage:

   Copy

   dbutils.fs.cp("file://{}".format(localScratchFileCopy), s3ScratchFile)

This page provides guidelines for troubleshooting issues with the Databricks Spark integration and resolving Py4J security and Databricks trusted library errors.

Debugging the integration

For easier debugging of the Databricks Spark integration, follow the recommendations below.

• Enable cluster init script logging:

  • In the cluster page in Databricks for the target cluster, navigate to Advanced Options -> Logging.

  • Change the Destination from NONE to DBFS and change the path to the desired output location. Note: The unique cluster ID will be added onto the end of the provided path.

• View the Spark UI on your target Databricks cluster: On the cluster page, click the Spark UI tab, which shows the Spark application UI for the cluster. If you encounter issues creating Databricks data sources in Immuta, you can also view the JDBC/ODBC Server portion of the Spark UI to see the result of queries that have been sent from Immuta to Databricks.

Using the validation and debugging notebook

The validation and debugging notebook is designed to be used by or under the guidance of an Immuta support professional. Reach out to your Immuta representative for assistance.

1. Import the notebook into a Databricks workspace by navigating to Home in your Databricks instance.

2. Click the arrow next to your name and select Import.

3. Once you have executed commands in the notebook and populated it with debugging information, export the notebook and its contents by opening the File menu, selecting Export, and then selecting DBC Archive.

Py4J security error

• Error Message: py4j.security.Py4JSecurityException: Constructor <> is not allowlisted

• Explanation: This error indicates you are being blocked by Py4J security rather than the Immuta Security Manager. Py4J security is strict and generally ends up blocking many ML libraries.

• Solution: Turn off Py4J security on the offending cluster by setting IMMUTA_SPARK_DATABRICKS_PY4J_STRICT_ENABLED=false in the environment variables section. Additionally, because there are limitations to the security mechanisms Immuta employs on-cluster when Py4J security is disabled, ensure that all users on the cluster have the same level of access to data, as users could theoretically see (policy-enforced) data that other users have queried.

Databricks trusted library errors

Check the driver logs for details. Some possible causes of failure include

• One of the Immuta-configured trusted library URIs does not point to a Databricks library. Check that you have configured the correct URI for the Databricks library.

• For trusted Maven artifacts, the URI must follow this format: maven:/group.id:artifact-id:version.

• Databricks failed to install a library. Any Databricks library installation errors will appear in the Databricks UI under the Libraries tab.

The Databricks Spark integration is one of two integrations Immuta offers for Databricks.

In this integration, Immuta installs an Immuta-maintained Spark plugin on your Databricks cluster. When a user queries data that has been registered in Immuta as a data source, the plugin injects policy logic into the plan Spark builds so that the results returned to the user only include data that specific user should see.

The reference guides in this section are written for Databricks administrators who are responsible for setting up the integration, securing Databricks clusters, and setting up users:

• Installation and compliance: This guide includes information about what Immuta creates in your Databricks environment and securing your Databricks clusters.

• Customizing the integration: Consult this guide for information about customizing the Databricks Spark integration settings.

• Setting up users: Consult this guide for information about connecting data users and setting up user impersonation.

• Spark environment variables: This guide provides a list of Spark environment variables used to configure the integration.

• Ephemeral overrides: This guide describes ephemeral overrides and how to configure them to reduce the risk that a user has overrides set to a cluster (or multiple clusters) that aren't currently up.

When the Databricks Spark plugin is running on a Databricks cluster, all Databricks users running jobs or queries are either a privileged user or a non-privileged user:

• Privileged users: Privileged users can effectively read from and write to any table or view in the cluster Metastore, or any file path accessible by the cluster, without restriction. Privileged users are either Databricks workspace admins or users specified in IMMUTA_SPARK_ACL_ALLOWLIST. Any user writing queries or jobs impersonating another user is a non-privileged user, even if they are impersonating a privileged user.

  Privileged users have effective authority to read from and write to any securable in the cluster metastore or file path, because in almost all cases Databricks clusters running with the Immuta Spark plug-in installed have disabled Hive metastore table access control. However, if Hive metastore table access control is enabled on the cluster, privileged users will have the authority granted to them that is specified by table access control.

• Non-privileged users: Non-privileged users are any users who are not privileged users, and all authorization for non-privileged users is determined by Immuta policies.

Whether a user is a privileged user or a non-privileged user, for a given query or job, is cached once first determined, based on IMMUTA_SPARK_ACL_PRIVILEGED_TIMEOUT_SECONDS environment variable. This caching can be disabled entirely by setting the value of that environment variable to 0.

Mapping Databricks users to Immuta

Usernames in Databricks must match the usernames in the connected Immuta tenant. By default, the Immuta Spark plugin checks the Databricks username against the username within Immuta's internal IAM to determine access. However, you can integrate your existing IAM with Immuta and use that instead of the default internal IAM. Ideally, you should use the same identity manager for Immuta that you use for Databricks. See the Immuta support matrix page for a list of supported identity providers and protocols.

It is possible within Immuta to have multiple users share the same username if they exist within different IAMs. In this case, the cluster can be configured to look up users from a specified IAM. To do this, the value of theIMMTUA_USER_MAPPING_IAMID Spark environment variable must be updated to be the targeted IAM ID configured within the Immuta tenant. The targeted IAM ID can be found on the App settings page. Each Databricks cluster can only be mapped to one IAM.

User impersonation

Databricks user impersonation allows a Databricks user to impersonate an Immuta user. With this feature,

• the Immuta user who is being impersonated does not have to have a Databricks account, but they must have an Immuta account.

• the Databricks user who is impersonating an Immuta user does not have to be associated with Immuta. For example, this could be a service account.

When acting under impersonation, the Databricks user loses their privileged access, so they can only access the tables the Immuta user has access to and only perform DDL commands when that user is acting under an allowed circumstance (such as workspaces, scratch paths, or non-Immuta reads/writes).

Use the IMMUTA_SPARK_DATABRICKS_ALLOWED_IMPERSONATION_USERS Spark environment variable to enable user impersonation.

In the context of the Databricks Spark integration, Immuta uses the term ephemeral to describe data sources where the associated compute resources can vary over time. This means that the compute bound to these data sources is not fixed and can change. All Databricks data sources in Immuta are ephemeral.

Ephemeral overrides are specific to each data source and user. They effectively bind cluster compute resources to a data source for a given user. Immuta uses these overrides to determine which cluster compute to use when connecting to Databricks for various maintenance operations.

The operations that use the ephemeral overrides include

• Visibility checks on the data source for a particular user. These checks assess how to apply row-level policies for specific users.

• Stats collection triggered by a specific user.

• Validating a custom WHERE clause policy against a data source. When owners or governors create custom WHERE clause policies, Immuta uses compute resources to validate the SQL in the policy. In this case, the ephemeral overrides for the user writing the policy are used to contact a cluster for SQL validation.

• High cardinality column detection. Certain advanced policy types (e.g., minimization) in Immuta require a high cardinality column, and that column is computed on data source creation. It can be recomputed on demand and, if so, will use the ephemeral overrides for the user requesting computation.

Triggering an ephemeral override request

An ephemeral override request can be triggered when a user queries the securable corresponding to a data source in a Databricks cluster with the Spark plug-in configured. The actual triggering of this request depends on the configuration settings.

Ephemeral overrides can also be set for a data source in the Immuta UI by navigating to the data source page, clicking on the data source actions button, and selecting Ephemeral overrides from the dropdown menu.

Ephemeral override requests made from a cluster for data sources and users where ephemeral overrides were set in the UI will not be successful.

If ephemeral overrides are never set (either through the user interface or the cluster configuration), the system will continue to use the connection details directly associated with the data source, which are set during data source registration.

Configuring overrides in Immuta-enabled clusters

Ephemeral overrides can be problematic in environments that have a dedicated cluster to handle maintenance activities, since ephemeral overrides can cause these operations to execute on a different cluster than the dedicated one.

To reduce the risk that a user has overrides set to a cluster (or multiple clusters) that aren't currently up, complete one of the following actions:

• Direct all clusters' HTTP paths for overrides to a cluster dedicated for metadata queries using the IMMUTA_EPHEMERAL_HOST_OVERRIDE_HTTPPATH Spark environment variable.

• Disable ephemeral overrides completely by setting the IMMTUA_EPHEMERAL_HOST_OVERRIDE Spark environment variable to false.

When using Delta Lake, the API does not go through the normal Spark execution path. This means that Immuta's Spark extensions do not provide protection for the API. To solve this issue and ensure that Immuta has control over what a user can access, the Delta Lake API is blocked.

Spark SQL can be used instead to give the same functionality with all of Immuta's data protections.

Requests

Below is a table of the Delta Lake API with the Spark SQL that may be used instead.

See here for a complete list of the Delta SQL Commands.

Merging tables in workspaces

When a table is created in a project workspace, you can merge a different Immuta data source from that workspace into that table you created.

1. Create a table in the project workspace.

2. Create a temporary view of the Immuta data source you want to merge into that table.

3. Use that temporary view as the data source you add to the project workspace.

4. Run the following command:

   Copy

   MERGE INTO delta_native.target_native as target
   USING immuta_temp_view_data_source as source
   ON target.dr_number = source.dr_number
   WHEN MATCHED THEN
   UPDATE SET target.date_reported = source.date_reported

The how-to guides linked on this page illustrate how to integrate Databricks Unity Catalog with Immuta. See the reference guide for information about the Databricks Unity Catalog integration.

Requirements:

• Unity Catalog metastore created and attached to a Databricks workspace. Immuta supports configuring a single metastore for each configured integration, and that metastore may be attached to multiple Databricks workspaces.

• Unity Catalog enabled on your Databricks cluster or SQL warehouse. All SQL warehouses have Unity Catalog enabled if your workspace is attached to a Unity Catalog metastore.

The how-to guides linked on this page illustrate how to integrate Snowflake with Immuta. See the reference guide for information about the Snowflake integration.

Requirements

• Snowflake enterprise edition

• Access to a Snowflake account that can create a Snowflake user

Immuta is compatible with Snowflake Secure Data Sharing. Using both Immuta and Snowflake, organizations can share the policy-protected data of their Snowflake database with other Snowflake accounts with Immuta policies enforced in real time.

Prerequisites:

• Snowflake integration enabled

• Snowflake tables registered in Immuta as data sources

Create Immuta Policies to Protect the Data

Required Permission: Immuta: GOVERNANCE

Build Immuta data policies to fit your organization's compliance requirements.

It's important to understand that subscription policies are not relevant to Snowflake data shares, because the act of sharing the data is the subscription policy. Data policies can be enforced on the consuming account from the producer account on a share following these instructions.

Register the Snowflake Data Consumer with Immuta

Required Permission: Immuta: USER_ADMIN

To register the Snowflake data consumer in Immuta,

1. Create a new Immuta user.

2. Update the Immuta user's Snowflake username to match the account ID for the data consumer. This value is the output on the data consumer side when SELECT CURRENT_ACCOUNT() is run in Snowflake.

3. Give the Immuta user the appropriate attributes and groups for your organization's policies.

4. Subscribe the Immuta user to the data sources.

Create the Snowflake Data Share

Required Permission: Snowflake ACCOUNTADMIN

To share the policy-protected data source,

1. Create a Snowflake Data Share of the Snowflake table that has been registered in Immuta.

2. Grant reference usage on the Immuta database to the share you created:

   Copy

   GRANT REFERENCE_USAGE ON DATABASE "<Immuta database of the provider account>" TO SHARE "<DATA_SHARE>";

   Replace the content in angle brackets above with the name of your Immuta database and Snowflake data share.

In this integration, Immuta generates policy-enforced views in your configured Redshift schema for tables registered as Immuta data sources.

Getting started

This guide outlines how to integrate Redshift with Immuta.

How-to guides

• Redshift integration configuration: Configure the integration in Immuta.

• Redshift Spectrum configuration: Configure Redshift Spectrum in Immuta.

Reference guides

• Redshift integration reference guide: This guide describes the design and components of the integration.

• Redshift pre-configuration details: This guide describes the prerequisites, supported features, and limitations of the integration.

Prerequisites

This upgrade step is necessary if you meet both of the following criteria:

• You have the Snowflake low row access policy mode enabled in private preview.

• You have user impersonation enabled.

If you do not meet this criteria, follow the instructions on the configuration guide.

Upgrade to Snowflake low row access policy mode

To upgrade to the generally available version of the feature, disable your Snowflake integration on the app settings page and then re-enable it.

1. In the Databricks Clusters UI, install your third-party library .jar or Maven artifact with Library Source Upload, DBFS, DBFS/S3, or Maven. Alternatively, use the Databricks libraries API.
2. In the Databricks Clusters UI, add the IMMUTA_SPARK_DATABRICKS_TRUSTED_LIB_URIS property as a Spark environment variable and set it to your artifact's URI. To specify more than one trusted library, comma delimit the URIs:

   Copy

   IMMUTA_SPARK_DATABRICKS_TRUSTED_LIB_URIS=maven:/my.group.id:my-package-id:1.2.3

IMMUTA_SPARK_DATABRICKS_TRUSTED_LIB_URIS=maven:/com.github.immuta.hadoop.immuta-spark-third-party-maven-lib-test:2020-11-17-144644

IMMUTA_SPARK_DATABRICKS_TRUSTED_LIB_URIS=dbfs:/immuta/bstabile/jars/immuta-spark-third-party-lib-test.jar

1. Restart the cluster.

2. Once the cluster is up, execute a command in a notebook. If the trusted library installation is successful, you should see driver log messages like this:

   Copy

   TrustedLibraryUtils: Successfully found all configured Immuta configured trusted libraries in Databricks.
   TrustedLibraryUtils: Wrote trusted libs file to [/databricks/immuta/immutaTrustedLibs.json]: true.
   TrustedLibraryUtils: Added trusted libs file with 1 entries to spark context.
   TrustedLibraryUtils: Trusted library installation complete.

This guide demonstrates how to update credentials referenced in the Immuta Enterprise Helm chart (IEHC).

Kubernetes secrets

Edit secrets

1. Validate that secret immuta-secret exists in the current namespace.

2. Edit secret immuta-secret in place.

3. Edit secret immuta-legacy-secret in place. Skip this step if the legacy query engine is disabled (the default).

4. Restart pods.

Legacy query engine

1. Validate that secret immuta-legacy-secret exists in the current namespace.

2. Get the query engine replica count, this value will be referenced in subsequent step(s).

3. Scale the replica count down to 1.

4. Get the query engine pod name, this value will be referenced in subsequent step(s).

5. Update the with a query engine superuser password.

6. Update the with a query engine replication password.

7. Update the with a query engine feature password.

8. Scale the replica count back up to the previous value by updating the .

Apply Helm values

1. Update credentials in the immuta-values.yaml file.

2. Perform a to apply the changes made to immuta-values.yaml. Update the with your own release name.

This guide demonstrates how to download and package the Immuta Enterprise Helm chart and its dependencies for consumption on a separate network with no internet access.

Prerequisite

Checklist

Skopeo

• Skopeo is authenticated with Immuta's registry on the networked machine.

• Skopeo is authenticated with the private registry on the air-gapped machine.

Helm

• Helm is authenticated with Immuta's registry on the networked machine.

Download artifacts

This section demonstrates how to download the Helm chart and container images to your local machine. These artifacts will be packaged and transferred to the air-gapped environment later.

1. Create a directory named offline-kit.

2. Download the Helm chart into directory offline-kit.

3. Extract file DIGESTS.md from the Helm chart archive.

4. Open file ./offline-kit/DIGESTS.md. This file includes the name and digest of every container image referenced by the Helm chart.

5. Download each image listed in file DIGESTS.md using . Each image will be saved to directory offline-kit with the filename<name>-<tag>.tar.

Transfer artifacts

This section demonstrates how to push the previously archived container images to a private registry that's accessible from within your air-gapped environment.

1. Transfer directory offline-kit (created in the previous section) onto a machine that's within your air-gapped environment.

2. Push each image to your private registry using .

Chart installation

Edit the immuta-values.yaml to reference the .

Frequently asked questions

How can I ensure the fully qualified domain name (FQDN) is resolvable from within the Kubernetes cluster?

1. Create a pod named debug-dns and spawn an interactive shell.

2. Install package bind-utils.

3. Perform DNS lookups on a given FQDN.

I'm unsure which Kubernetes namespace or Helm release is associated with my Immuta installation. How can I find this out?

I no longer have my immuta-values.yaml Helm values file. How do I recover this file?

I don't want to keep passing option --namespace every time I run a Helm command. How do I set a default?

PostgreSQL

How do I determine if the database is accepting connections?

1. Create a pod named debug-postgres and spawn an interactive shell.

2. Validate that the database is listening.

Redis

How can a TCP connection be established without using Redis CLI?

1. Create a pod named debug-redis and spawn an interactive shell.

2. Send a raw TCP message to the database using Netcat.

How do I establish a TCP connection?

1. Create a pod named debug-redis and spawn an interactive shell.

2. Establish a connection to the database using the Redis client. If a connection can be established with Netcat and the redis-cli command does not return, then Redis could be expecting a TLS connection. Pass option --tls.

Elasticsearch

How do I query the API using cURL?

1. Create a pod named debug-elasticsearch and spawn an interactive shell.

2. Install package curl.

3. Check the .

Helm

When installing the helm chart from ocir.immuta.com, I get error scheme "oci" not supported. What's going on?

The Immuta Enterprise Helm chart (IEHC) is distributed as an OCI artifact, and your current Helm version might not support it. Refer to the for further assistance.

1. Determine your Helm version.

2. If older than 3.8.0 you'll need to upgrade. Before this version OCI support wasn't enabled by default.

This page provides a tutorial for enabling the Azure Synapse Analytics integration on the Immuta app settings page. To configure this integration via the Immuta API, see the .

For an overview of the integration, see the documentation.

Requirement

A running Dedicated SQL pool is required.

Add an Azure Synapse Analytics integration

1. Click the App Settings icon in the left sidebar.

2. Click the Integrations tab.

3. Click the +Add Integration button and select Azure Synapse Analytics from the dropdown menu.

4. Complete the Host, Port, Immuta Database, and Immuta Schema fields.

5. Opt to check the Enable Impersonation box and customize the Impersonation Role name as needed. This will allow users to natively impersonate another user.

6. Opt to update the User Profile Delimiters. This will be necessary if any of the provided symbols are used in user profile information.

Select your configuration method

You have two options for configuring your Azure Synapse Analytic environment:

• : Grant Immuta one-time use of credentials to automatically configure your environment and the integration.

• : Run the Immuta script in your Azure Synapse Analytics environment yourself to configure the integration.

Automatic setup

Enter the username and password in the Privileged User Credentials section.

Manual setup

1. Select Manual.

2. Download, fill out the appropriate fields, and run the bootstrap master script and bootstrap script linked in the Setup section.

3. Enter the username and password in the Immuta System Account Credentials section. The username and password provided must be the credentials that were set in the bootstrap master script when you created the user.

Save the configuration

Click Save.

Register data

.

Edit an Azure Synapse Analytics integration

1. Click the App Settings icon in the left sidebar.

2. Navigate to the Integrations tab and click the down arrow next to the Azure Synapse Analytics Integration.

3. Edit the field you want to change. Note any field shadowed is not editable, and the integration must be disabled and re-installed to change it.

4. Enter Username and Password.

5. Click Save.

Remove an Azure Synapse Analytics integration

1. Click the App Settings icon in the left sidebar.

2. Navigate to the Integrations tab and click the down arrow next to the Azure Synapse Analytics Integration.

3. Click the checkbox to disable the integration.

4. Enter the username and password that were used to initially configure the integration.

5. Click Save.

This page describes the Azure Synapse Analytics integration, through which Immuta applies policies directly in Azure Synapse Analytics. For a tutorial on configuring Azure Synapse Analytics see the .

Overview

The Azure Synapse Analytics is a policy push integration that allows Immuta to apply policies directly in Azure Synapse Analytics Dedicated SQL pools without the need for users to go through a proxy. Instead, users can work within their existing Synapse Studio and have per-user policies dynamically applied at query time.

Architecture

This integration works on a per-Dedicated-SQL-pool basis: all of Immuta's policy definitions and user entitlements data need to be in the same pool as the target data sources because Dedicated SQL pools do not support cross-database joins. Immuta creates schemas inside the configured Dedicated SQL pool that contain policy-enforced views that users query.

When the integration is configured, the Application Admin specifies the

• Immuta database: This is the pre-existing database Immuta uses. Immuta will create views from the tables contained in this database, and all schemas and views created by Immuta will exist in this database, such as the schemas immuta_system, immuta_functions, and the immuta_procedures that contain the tables, views, UDFs, and stored procedures that support the integration.

• Immuta schema: The schema that Immuta manages. All views generated by Immuta for tables registered as data sources will be created in this schema.

• User profile delimiters: Since Azure Synapse Analytics dedicated SQL pools do not support array or hash objects, certain user access information is stored as delimited strings; the Application Admin can modify those delimiters to ensure they do not conflict with possible characters in strings.

For a tutorial on configuring the integration see the .

Data source naming convention

Synapse data sources are represented as views and are under one schema instead of a database, so their view names are a combination of their schema and table name, separated by an underscore.

For example, with a configuration that uses IMMUTA as the schema in the database dedicated_pool, the view name for the data source dedicated_pool.tpc.case would be dedicated_pool.IMMUTA.tpc_case.

You can see the view information on the data source details page under Connection Information.

Policy enforcement

This integration uses webhooks to keep views up-to-date with the corresponding Immuta data sources. When a data source or policy is created, updated, or disabled, a webhook is called that creates, modifies, or deletes the dynamic view in the Immuta schema. Note that only standard views are available because Azure Synapse Analytics Dedicated SQL pools do not support secure views.

Integration health status

The status of the integration is visible on the integrations tab of the Immuta application settings page. If errors occur in the integration, a banner will appear in the Immuta UI with guidance for remediating the error.

The definitions for each status and the state of configured data platform integrations is available in the . However, the UI consolidates these error statuses and provides detail in the error messages.

Data flow

1. An Immuta Application Administrator , registering their initial Synapse Dedicated SQL pool with Immuta.

2. Immuta creates Immuta schemas inside the configured Synapse Dedicated SQL pool.

3. A Data Owner in Immuta as data sources. A Data Owner, Data Governor, or Administrator or in Immuta.

4. Data source metadata, tags, user metadata, and policy definitions are stored in Immuta's Metadata Database.

5. The Immuta Web Service calls a stored procedure that modifies the user entitlements or policies and updates data source view definitions as necessary.

6. An Azure Synapse Analytics user who is subscribed to the data source in Immuta in Azure Synapse Analytics and sees policy-enforced data.

Immuta offers two integrations for Databricks:

• : This integration supports working with database objects registered in Unity Catalog.

• : This integration supports working with .

Which integration should you use?

To determine which integration you should use, evaluate the following elements:

• Cluster runtime

  • Databricks Runtime 11.3 and newer: See the list below to determine which integration is supported for your data's location.

• Location of data: Where is your data?

  • Legacy Hive metastore: Databricks recommends that you migrate all data from the legacy Hive metastore to Unity Catalog. However, when this migration is not possible, use the to protect securables registered in the Hive metastore.

  • Unity Catalog: To protect securables registered in the Unity Catalog metastore, use the .

  • Legacy Hive metastore and Unity Catalog: If you need to work with database objects registered in both the legacy Hive metastore and in Unity Catalog, allows you to use both integrations.

Metastore magic

Databricks metastore magic allows you to migrate your data from the Databricks legacy Hive metastore to the Unity Catalog metastore while protecting data and maintaining your current processes in a single Immuta instance.

Databricks metastore magic is for organizations who intend to use the , but must still protect tables in the Hive metastore until they can migrate all of their data to Unity Catalog.

Requirement

Unity Catalog support is enabled in Immuta.

Databricks metastores and Immuta policy enforcement

Databricks has two built-in metastores that contain metadata about your tables, views, and storage credentials:

• Legacy Hive metastore: Created at the workspace level. This metastore contains metadata of the registered securables in that workspace available to query.

• Unity Catalog metastore: Created at the account level and is attached to one or more Databricks workspaces. This metastore contains metadata of the registered securables available to query. All clusters on that workspace use the configured metastore and all workspaces that are configured to use a single metastore share those securables.

Databricks allows you to use the legacy Hive metastore and the Unity Catalog metastore simultaneously. However, Unity Catalog does not support controls on the Hive metastore, so you must attach a Unity Catalog metastore to your workspace and move existing databases and tables to the attached Unity Catalog metastore to use the governance capabilities of Unity Catalog.

Immuta's Databricks Spark integration and Unity Catalog integration enforce access controls on the Hive and Unity Catalog metastores, respectively. However, because these metastores have two distinct security models, users were discouraged from using both in a single Immuta instance before metastore magic; the Databricks Spark integration and Unity Catalog integration were unaware of each other, so using both concurrently caused undefined behavior.

Databricks metastore magic solution

Metastore magic reconciles the distinct security models of the legacy Hive metastore and the Unity Catalog metastore, allowing you to use multiple metastores (specifically, the Hive metastore or alongside Unity Catalog metastores) within a Databricks workspace and single Immuta instance and keep policies enforced on all your tables as you migrate them. The diagram below shows Immuta enforcing policies on registered tables across workspaces.

In clusters A and D, Immuta enforces policies on data sources in each workspace's Hive metastore and in the Unity Catalog metastore shared by those workspaces. In clusters B, C, and E (which don't have Unity Catalog enabled in Databricks), Immuta enforces policies on data sources in the Hive metastores for each workspace.

Enforce policies as you migrate

With metastore magic, the Databricks Spark integration enforces policies only on data in the Hive metastore, while the Unity Catalog integration enforces policies on tables in the Unity Catalog metastore.

To enforce plugin-based policies on Hive metastore tables and Unity Catalog native controls on Unity Catalog metastore tables, enable the and the Databricks Unity Catalog integration. Note that some Immuta policies are not supported in the Databricks Unity Catalog integration. See the for details.

Enforcing policies on Databricks SQL

Databricks SQL cannot run the Databricks Spark plugin to protect tables, so Hive metastore data sources will not be policy enforced in Databricks SQL.

To enforce policies on data sources in Databricks SQL, use to manually lock down Hive metastore data sources and the Databricks Unity Catalog integration to protect tables in the Unity Catalog metastore. Table access control is enabled by default on SQL warehouses, and any Databricks cluster without the Immuta plugin must have table access control enabled.

The how-to guides linked on this page illustrate how to integrate Databricks Spark with Immuta.

Requirements

• If Databricks Unity Catalog is enabled in a Databricks workspace, you must use an when you set up the Databricks Spark integration to create an Immuta-enabled cluster.

• If Databricks Unity Catalog is not enabled in your Databricks workspace, you must disable Unity Catalog in your Immuta tenant before proceeding with your configuration of Databricks Spark:

  1. Navigate to the App Settings page and click Integration Settings.

  2. Uncheck the Enable Unity Catalog checkbox.

  3. Click Save.

The how-to guides linked on this page illustrate how to integrate Redshift with Immuta. See the for information about the Redshift integration.

Requirement: Redshift cluster with an RA3 node is required for the multi-database integration. For other instance types, you may configure a single-database integration using one of the .

This integration allows you to manage and access data in your Databricks account across all of your workspaces. With Immuta’s Databricks Unity Catalog integration, you can write your policies in Immuta and have them enforced automatically by Databricks across data in your Unity Catalog metastore.

This getting started guide outlines how to integrate Databricks Unity Catalog with Immuta.

How-to guides

• : Configure the Databricks Unity Catalog integration.

• : Migrate from the legacy Databricks Spark integrations to the Databricks Unity Catalog integration.

Reference guide

: This guide describes the design and components of the integration.

Immuta manages access to Snowflake tables by administering Snowflake and on those tables, allowing users to query tables directly in Snowflake while dynamic policies are enforced.

This getting started guide outlines how to integrate your Snowflake account with Immuta.

How-to guides

• : Configure the Snowflake integration.

• : Migrate to using Snowflake table grants in your Snowflake integration.

• : Manage integration settings or delete your existing Snowflake integration.

• Integration settings:

  • : Enable Snowflake table grants and configure the Snowflake role prefix.

  • : Use Snowflake data sharing with table grants or project workspaces.

  • : Enable Snowflake low row access policy mode.

  • : Configure your Snowflake integration to automatically apply tags added to a Snowflake table to its descendant data source columns in Immuta.

Reference guides

• : This reference guide describes the design and features of the Snowflake integration.

• : Organizations can share the policy-protected data of their Snowflake database with other Snowflake accounts with Immuta policies enforced in real time. This guide describes the components of using Immuta with Snowflake data shares.

• : Snowflake column lineage specifies how data flows from source tables or columns to the target tables in write operations. When Snowflake lineage tag propagation is enabled in Immuta, Immuta automatically applies tags added to a Snowflake table to its descendant data source columns in Immuta so you can build policies using those tags to restrict access to sensitive data.

• : The Snowflake low row access policy mode improves query performance in Immuta's Snowflake integration. To do so, this mode decreases the number of Snowflake row access policies Immuta creates and uses table grants to manage user access. This guide describes the design and requirements of this mode.

• : Snowflake table grants simplifies the management of privileges in Snowflake when using Immuta. Instead of manually granting users access to tables registered in Immuta, you allow Immuta to manage privileges on your Snowflake tables and views according to subscription policies. This guide describes the components of Snowflake table grants and how they are used in Immuta's Snowflake integration.

• : Adjust the size and scale of clusters for your warehouse to manage workloads so that you can use Snowflake compute resources the most cost effectively.

Explanatory guide

: A phased onboarding approach to configuring the Snowflake integration ensures that your users will not be immediately affected by changes as you add data sources and policies. This guide describes the settings and requirements for implementing this phased approach.

Immuta is compatible with . Using both Immuta and Snowflake, organizations can share the policy-protected data of their Snowflake database with other Snowflake accounts with Immuta policies enforced in real time. This integration gives data consumers a live connection to the data and relieves data providers of the legal and technical burden of creating static data copies that leave their Snowflake environment.

Requirements:

• Snowflake Enterprise Edition or higher

• Immuta's

Configuration

This method requires that the data consumer account is registered as an Immuta user with the Snowflake user name equal to the consuming account.

At that point, the user that represents the account being shared with can have the appropriate attributes and groups assigned to them, relevant to the data policies that need to be enforced. Once that user has access to the share in the consuming account (not managed by Immuta), they can query the share with the data policies from the producer account enforced because Immuta is treating that account as if they are a single user in Immuta.

For a tutorial on this workflow, see the .

Benefits

Using Immuta with Snowflake Data Sharing allows the sharer to

• Only need limited knowledge of the context or goals of the existing policies in place: Because the sharer is not editing or creating policies to share their data, they only need a limited knowledge of how the policies work. Their main responsibility is making sure they properly represent the attributes of the data consumer (the account being shared to).

• Leave policies untouched.

The warehouse you select when configuring the Snowflake integration uses compute resources to set up the integration, register data sources, orchestrate policies, and run jobs like identification. Snowflake credit charges are based on the size of and amount of time the warehouse is active, not the number of queries run.

This document prescribes how and when to adjust the size and scale of clusters for your warehouse to manage workloads so that you can use Snowflake compute resources the most cost effectively.

In general, increase the size of and number of clusters for the warehouse to handle heavy workloads and multiple queries. Workloads are typically lighter after data sources are onboarded and policies are established in Immuta, so compute resources can be reduced after those workloads complete.

Integration and data source registration warehouse use

The Snowflake integration uses warehouse compute resources to sync policies created in Immuta to the Snowflake objects registered as data sources and, if enabled, to run and . Follow the guidelines below to adjust the warehouse size and scale according to your needs.

• Increase the of and of clusters for the warehouse during large policy syncs, updates, and changes.

• Enable to optimize resource use in Snowflake. In the Snowflake UI, the lowest auto suspend time setting is 5 minutes. However, through SQL query, you can set auto_suspend to 61 seconds (since the minimum uptime for a warehouse is 60 seconds). For example,

• Identification uses compute resources for each table it runs on. Consider when registering data sources if you have an or a tagging strategy in place.

• Register data before creating global policies. Immuta does not apply a subscription policy on registered data unless an existing global policy applies to it, which allows Immuta to only pull metadata instead of also applying policies when data sources are created. Registering data before policies are created reduces the workload and the Snowflake compute resources needed.

• Begin onboarding with a small dataset of tables, and then review and monitor query performance in the . Adjust the virtual warehouse accordingly to handle heavier loads.

• uses the compute warehouse that was employed during the initial ingestion to periodically monitor the schema for changes. If you expect a low number of new tables or minimal changes to the table structure, consider scaling down the warehouse size.

• Resize the warehouse after after data sources are registered and policies are established. For example,

For more details and guidance about warehouse sizing, see the .

Identifying bulk jobs and heavy workloads

Even after your integration is configured, data sources are registered, and policies are established, changes to those data sources or policies may initiate heavy workloads. Follow the guidelines below to adjust your warehouse size and scale according to your needs.

• Review your to identify query performance and bottlenecks.

• Check how many credits queries have consumed:

• After reviewing query performance and cost, implement to adjust your warehouse.

In this integration, Immuta policies are translated into Starburst rules and permissions and applied directly to tables within users’ existing catalogs.

This guide outlines how to integrate Starburst with Immuta.

How-to guides

• : Configure the integration in Immuta.

• : Configure how read and write access subscription policies translate to Starburst (Trino) privileges and apply to Starburst (Trino) data sources.

Reference guide

: This guide describes the design and components of the integration.

The how-to guides linked on this page illustrate how to integrate Starburst (Trino) with Immuta. See the for information about the Starburst (Trino) integration.

Once your data platform integration is configured, Immuta periodically runs queries in that data platform to orchestrate policies or implement various features. Depending on your configuration, data platform cost model, and data platform query load, there may be incremental cost incurred when various Immuta features are enabled. The actions and features that trigger Immuta queries in your remote platform are listed below.

• Configuring an integration or registering a connection: Immuta uses compute resources to set up the integration in the data platform. After the integration is configured, Immuta runs periodic validation queries to ensure the integration is still healthy. By default, this simple SELECT query is run once per hour to validate that the credentials, connection information, and network configuration are all functional.

• : Immuta uses compute resources to register data objects and data sources. If schema monitoring is enabled when registering a data source, Immuta uses the compute warehouse that was employed during the initial data source registration to periodically monitor the schema for changes. To adjust the schedule of the schema monitoring job to reduce cost, see the . Additionally, these actions will use compute resources:

  • Data object or data source disabled

  • Data object or data source enabled

  • Data object or data source deleted

• Policy applied to a data source: Immuta uses compute resources to orchestrate policies in the data platform. Consider registering data before creating global policies. Immuta does not apply a subscription policy on registered data unless an existing global policy applies to it, which allows Immuta to only pull metadata instead of also applying policies when data sources are created. Registering data before policies are created reduces the workload and the compute resources needed; Immuta will only perform a grant for the user who registered the data source. The following actions that trigger updates to policies will also use compute resources:

  • External user ID modifications

  • Group name changes

  • Policy modifications

  • Tags changing on data sources

  • User attributes changing

  • Users being added to or removed from groups

• Scheduled audit ingest or manually-triggered audit ingest (clicking the Load Audit Events button): Generally, the data platform cost from enabling query audits is directly related to warehouse uptime governed by the audit frequency and average query compute cost. During query audit retrieval, Immuta runs standard query operations (e.g., SELECT) against the system views and does not use other data transfer methods that incur additional data egress costs. For example, during query audit retrieval for Snowflake, Immuta will use the Snowflake warehouse that was configured during integration registration to query the Snowflake system views. If this warehouse is stopped, Immuta will start it.

• : To evaluate your data, Immuta generates a SQL query to execute in the remote technology. The query result contains the column name and the matching identifiers, and Immuta applies tags to the appropriate columns.

  This evaluating and tagging process occurs when identification runs, which happens from the following events:

  • A new data source is created.

  • is enabled and a new data source is detected.

  • is enabled and new columns are detected. Here, identification will only run on new columns, and no existing tags will be removed or changed.

  • A user to run from the data source health check menu, domain page, or through the API.

To edit or remove a Snowflake integration, you have two options:

• Automatic: Grant Immuta one-time use of credentials with the following privileges to automatically edit or remove the integration:

  • CREATE DATABASE ON ACCOUNT WITH GRANT OPTION

  • CREATE ROLE ON ACCOUNT WITH GRANT OPTION

  • CREATE USER ON ACCOUNT WITH GRANT OPTION

  • MANAGE GRANTS ON ACCOUNT WITH GRANT OPTION

• Manual: Run the Immuta script in your Snowflake environment as a user with the following privileges to edit or remove the integration:

  • CREATE DATABASE ON ACCOUNT WITH GRANT OPTION

  • CREATE ROLE ON ACCOUNT WITH GRANT OPTION

  • CREATE USER ON ACCOUNT WITH GRANT OPTION

  • MANAGE GRANTS ON ACCOUNT WITH GRANT OPTION

  • APPLY MASKING POLICY ON ACCOUNT WITH GRANT OPTION

  • APPLY ROW ACCESS POLICY ON ACCOUNT WITH GRANT OPTION

Edit a Snowflake integration

Select one of the following options for editing your integration:

• Automatic: Grant Immuta one-time use of credentials to automatically edit the integration.

• Manual: Run the Immuta script in your Snowflake environment yourself to edit the integration.

Automatic edit

1. Click the App Settings icon in the navigation menu.

2. Click the Integrations tab and click the down arrow next to the Snowflake integration.

3. Edit the field you want to change or check a checkbox of a feature you would like to enable. Note any field shadowed is not editable, and the integration must be disabled and re-installed to change it.

4. From the Select Authentication Method Dropdown, select either Username and Password or Key Pair Authentication:

   • Username and Password option: Complete the Username, Password, and Role fields.

   • Key Pair Authentication option:

     1. Complete the Username field.

     2. When using a private key, enter the private key file password in the Additional Connection String Options. Use the following format: PRIV_KEY_FILE_PWD=<your_pw>

     3. Click Key Pair (Required), and upload a Snowflake key pair file.

     4. Complete the Role field.

5. Click Save.

Manual edit

1. Click the App Settings icon in the navigation menu.

2. Click the Integrations tab and click the down arrow next to the Snowflake integration.

3. Edit the field you want to change or check a checkbox of a feature you would like to enable. Note any field shadowed is not editable, and the integration must be disabled and re-installed to change it.

4. Click edit script to download the script, and then run it in Snowflake.

5. Click Save.

Remove a Snowflake integration

Select one of the following options for deleting your integration:

• Automatic: Grant Immuta one-time use of credentials to automatically remove the integration and Immuta-managed resources from your Snowflake environment.

• Manual: Run the Immuta script in your Snowflake environment yourself to remove Immuta-managed resources and policies from Snowflake.

Automatic removal

1. Click the App Settings icon in the navigation menu.

2. Click the Integrations tab and click the down arrow next to the Snowflake integration.

3. Click the checkbox to disable the integration.

4. Enter the Username, Password, and Role that was entered when the integration was configured.

5. Click Save.

Manual removal

1. Click the App Settings icon in the navigation menu.

2. Click the Integrations tab and click the down arrow next to the Snowflake integration.

3. Click the checkbox to disable the integration.

4. Click cleanup script to download the script.

5. Click Save.

6. Run the cleanup script in Snowflake.

The Snowflake low row access policy mode improves query performance in Immuta's Snowflake integration by decreasing the number of Snowflake row access policies Immuta creates and by using table grants to manage user access.

Immuta manages access to Snowflake tables by administering Snowflake row access policies and column masking policies on those tables, allowing users to query them directly in Snowflake while policies are enforced.

Without Snowflake low row access policy mode enabled, row access policies are created and administered by Immuta in the following scenarios:

• Table grants are disabled and a subscription policy that does not automatically subscribe everyone to the data source is applied. Immuta administers Snowflake row access policies to filter out all the rows to restrict access to the entire table when the user doesn't have privileges to query it. However, if table grants are disabled and a subscription policy is applied that grants everyone access to the data source automatically, Immuta does not create a row access policy in Snowflake. See the subscription policies page for details about these policy types.

• Purpose-based policy is applied to a data source. A row access policy filters out all the rows of the table if users aren't acting under the purpose specified in the policy when they query the table.

• Row-level security policy is applied to a data source. A row access policy filters out rows querying users don't have access to.

• User impersonation is enabled. A row access policy is created for every Snowflake table registered in Immuta.

Reducing row access policies

Snowflake low row access policy mode is enabled by default to reduce the number of row access policies Immuta creates and improve query performance. Snowflake low row access policy mode requires

• table grants to be enabled.

• user impersonation to be disabled. User impersonation diminishes the performance of interactive queries because of the number of row access policies Immuta creates when it's enabled.

Requirements

• Snowflake integration enabled

• Snowflake table grants enabled

Project-scoped purpose exceptions for Snowflake with low row access policy mode enabled

Project-scoped purpose exceptions for Snowflake integrations allow you to apply purpose-based policies to Snowflake data sources in a project. As a result, users can only access that data when they are working within that specific project.

Masked joins for Snowflake with low row access policy mode enabled

This feature allows masked columns to be joined across data sources that belong to the same project. When data sources do not belong to a project, Immuta uses a unique salt per data source for hashing to prevent masked values from being joined. (See the Why use masked joins? guide for an explanation of that behavior.) However, once you add Snowflake data sources to a project and enable masked joins, Immuta uses a consistent salt across all the data sources in that project to allow the join.

For more information about masked joins and enabling them for your project, see the Masked joins section of documentation.

Limitations and considerations

• Project workspaces are not compatible with this feature.

• Impersonation is not supported when the Snowflake low row access policy mode is enabled.

While you're onboarding Snowflake data sources and designing policies, you don't want to disrupt your Snowflake users' existing workflows. Instead, you want to gradually onboard Immuta through a series of successive changes that will not impact your existing Snowflake users.

A phased onboarding approach to configuring the Snowflake integration ensures that your users will not be immediately affected by changes as you add data sources and configure policies.

Several features allow you to gradually onboard data sources and policies in Immuta:

• No subscription policies are applied at the time of data registration: No policy is applied at registration time unless an existing global policy applies to it; the table is registered in Immuta and waits for a policy to be applied, if ever.

  There are several benefits to this design:

  • All existing roles maintain access to the data and registration of the table or view with Immuta has zero impact on your data platform.

  • It gives you time to configure tags on the Immuta registered tables and views, either manually or through automatic means, such as Immuta’s identification, or an external catalog integration to include Snowflake tags.

  • It gives you time to assess and validate the sensitive data tags that were applied.

  • You can build only row and column controls with Immuta and let your existing roles manage table access instead of using Immuta subscription policies for table access.

• Snowflake table grants coupled with Snowflake low row access policy mode: With these features enabled, Immuta manages access to tables (subscription policies) through GRANTs. This works by assigning each user their own unique role created by Immuta and all table access is managed using that single role.

  Without these two features enabled, Immuta uses a Snowflake row access policy (RAP) to manage table access. A RAP only allows users to access rows in the table if they were explicitly granted access through an Immuta subscription policy; otherwise, the user sees no rows. This behavior means all existing Snowflake roles lose access to the table contents until explicitly granted access through Immuta subscription policies. Essentially, roles outside of Immuta don't control access anymore.

  By using table grants and the low row access policy mode, users and roles outside Immuta continue to work.

  There are two benefits to this approach:

  • All pre-existing Snowflake roles retain access to the data until you explicitly revoke access (outside Immuta).

  • It provides a way to test that Immuta GRANTs are working without impacting production workloads.

Requirements

The following configuration is required for phased Snowflake onboarding:

• Impersonation is disabled

• Project workspaces are disabled

If either of these capabilities is necessary for your use case, you cannot do phased Snowflake onboarding as described below.

Next

See the Getting started page for step-by-step guidance to implement phased Snowflake onboarding.

Contact your Immuta representative to enable this feature in your Immuta tenant.

Configure the Snowflake integration

1. Navigate to the App Setting page and click the Integration tab.

2. Click +Add Integration and select Snowflake from the dropdown menu.

3. Complete the Host, Port, and Default Warehouse fields.

4. Enable Query Audit.

5. Enable Lineage and complete the following fields:

   • Ingest Batch Sizes: This setting configures the number of rows Immuta ingests per batch when streaming Access History data from your Snowflake instance.

   • Table Filter: This filter determines which tables Immuta will ingest lineage for. Enter a regular expression that excludes / from the beginning and end to filter tables. Without this filter, Immuta will attempt to ingest lineage for every table on your Snowflake instance.

   • Tag Filter: This filter determines which tags to propagate using lineage. Enter a regular expression that excludes / from the beginning and end to filter tags. Without this filter, Immuta will ingest lineage for every tag on your Snowflake instance.

6. Select Manual or Automatic Setup and follow the steps in this guide to configure the Snowflake integration

Trigger Snowflake lineage sync job

Prerequisite

Authenticate with the Immuta API.

Trigger the lineage job

The Snowflake lineage sync endpoint triggers the lineage ingestion job that allows Immuta to propagate Snowflake tags added through lineage to Immuta data sources.

1. Copy the example and replace the Immuta URL and API key with your own.

2. Change the payload attribute values to your own, where

   • tableFilter (string): This regular expression determines which tables Immuta will ingest lineage for. Enter a regular expression that excludes / from the beginning and end to filter tables. Without this filter, Immuta will attempt to ingest lineage for every table on your Snowflake instance.

   • batchSize (integer): This parameter configures the number of rows Immuta ingests per batch when streaming Access History data from your Snowflake instance. Minimum 1.

   • lastTimestamp (string): Setting this parameter will only return lineage events later than the value provided. Use a format like 2022-06-29T09:47:06.012-07:00.

   Copy

   curl -X 'POST' \
       'https://www.organization.immuta.com/lineage/ingest/snowflake' \
       -H 'accept: application/json' \
       -H 'Content-Type: application/json' \
       -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f' \
       -d '{
       "tableFilter": "MY_DATABASE\\MY_SCHEMA\\..*",
       "batchSize": 1,
       "lastTimestamp": "2022-06-29T09:47:06.012-07:00"
       }'

Next steps

Once the sync job is complete, you can complete the following steps:

• Register Snowflake data sources

• Build policies

When you enable Unity Catalog, Immuta automatically migrates your existing Databricks data sources in Immuta to reference the legacy hive_metastore catalog to account for Unity Catalog's three-level hierarchy. New data sources will reference the Unity Catalog metastore you create and attach to your Databricks workspace.

Because the hive_metastore catalog is not managed by Unity Catalog, existing data sources in the hive_metastore cannot have Unity Catalog access controls applied to them. Data sources in the Hive Metastore must be managed by the Databricks Spark integration.

To allow Immuta to administer Unity Catalog access controls on that data, move the data to Unity Catalog and re-register those tables in Immuta by completing the steps below. If you don't move all data before configuring the integration, metastore magic will protect your existing data sources throughout the migration process.

1. Ensure that all Databricks clusters that have Immuta installed are stopped and the Immuta configuration is removed from the cluster. Immuta-specific cluster configuration is no longer needed with the Databricks Unity Catalog integration.

2. Move all data into Unity Catalog before configuring Immuta with Unity Catalog. Existing data sources will need to be re-created after they are moved to Unity Catalog and the Unity Catalog integration is configured.

3. Enable Unity Catalog.

Immuta comprises three core services: Secure, Discover, and Detect. These services rely on PostgreSQL and Elasticsearch to store their states, a caching layer, and Temporal for job execution. The illustration below shows the relationships among these services.

The Immuta Enterprise Helm chart (IEHC) does not include the deployment of PostgreSQL or Elasticsearch, so you must deploy them separately.

Although Immuta recommends using Elasticsearch because it supports several new Immuta features and services, you can deploy Immuta without Elasticsearch. The table below outlines the Immuta features supported with and without Elasticsearch and the dependencies you must deploy and manage yourself.

For information about legacy databases and services no longer enabled in the recommended deployment of Immuta, see the .

Version requirements

Kubernetes versions

• Kubernetes 1.29 - 1.32

Metadata database (PostgreSQL)

• PostgreSQL 15.0 or newer

• The pgcrypto and btree_gin extensions must be enabled

Elasticsearch

• Elasticsearch v7 API or newer

• AWS OpenSearch Service compatible with Elasticsearch v7 API or newer

  • AWS OpenSearch Serverless is not supported

OpenSearch user

The user provided during the install must have the following :

• cluster:monitor/health

• indices:data/write/bulk*

• indices:data/write/bulk

• indices:data/read/search

• indices:admin/exists

• indices:admin/create

• indices:admin/delete

• indices:admin/settings/update

• indices:admin/get

• indices:data/write/delete/byquery

• indices:data/write/index

• indices:admin/mapping/put

• indices:data/write/bulk

• indices:data/write/bulk*

Follow OpenSearch documentation to and add permissions, or see the .

Cache (Redis/Memcached)

• Redis 7.0 or newer

• Memcached 1.6 or newer

Temporal

• Temporal 1.24.2 or newer

Infrastructure recommendations

Legacy databases

Some legacy databases are no longer available when deploying Immuta using the recommended configuration of the IEHC. See the to enable support for these .

This guide demonstrates how to upgrade an existing Immuta deployment installed with the older Immuta Helm chart (IHC) to v2024.2 LTS using the Immuta Enterprise Helm chart (IEHC).

Prerequisites

Create a PostgreSQL database

1. The PostgreSQL instance has been provisioned and is actively running.

2. The PostgreSQL instance's hostname/FQDN is .

3. The PostgreSQL instance is .

For additional information, consult the Deployment requirements.

Validate the Helm release

1. Fetch the metadata for the Helm release associated with Immuta.

2. Review the output from the previous step and verify the following:

   • The Immuta version (appVersion) is

     • The last LTS (2022.5.x) or 2024.1 or newer

     • Less than 2024.2

   • The Immuta Helm chart (version) is greater than or equal to 4.13.5

   • The Immuta Helm chart name (chart) is immuta

3. If any of the criteria is not met, it's first necessary to perform a Helm upgrade using the IHC. Contact your Immuta representative for guidance.

Metadata database

The new IEHC no longer supports deploying a Metadata database (PostgreSQL) inside the Kubernetes cluster. Before transitioning to the new IEHC, it's first necessary to externalize the Metadata database.

Built-in

The following demonstrates how to take a database backup and import the data into each cloud provider's managed PostgreSQL service.

Create backup of old database

1. Get the metadata database pod name.

2. Spawn a shell inside the running metadata database pod.

3. Perform a database backup.

4. Type exit, and then press Enter to exit the shell prompt.

5. Copy file bometadata.dump from the pod to the host's working directory.

Setup new database

1. Create a pod named immuta-setup-db and spawn a shell.

2. Connect to the new PostgreSQL database as a superuser. Depending on the cloud provider, the default superuser name (postgres) might differ.

3. Create immuta, temporal, and temporal_visiblity databases and an immuta role.

4. Type \q, and then press Enter to exit the psql prompt.

Restore backup to new database

1. Create a pod named immuta-restore-db and spawn a shell.

2. Copy file bometadata.dump from the host's working directory to pod immuta-restore-db.

3. Spawn a shell inside pod immuta-restore-db.

4. Perform a database restore while authenticated as role immuta. Refer to the value substituted for <postgres-password> when prompted to enter a password.

5. Type exit, and then press Enter to exit the shell prompt.

6. Delete pod immuta-restore-db that was previously created.

External

No additional work is required. The existing database can be reused with the new IEHC.

Helm values

1. Rename the existing immuta-values.yaml Helm values file used by the IHC.

2. Follow the for your Kubernetes distribution of choice.

This guide demonstrates how to upgrade an existing 2024.2+ Immuta deployment installed with the Immuta Enterprise Helm chart (IEHC) to the latest Immuta release.

Requirements

Temporal

Starting in IEHC 2024.3, a Temporal server is included in the chart and requires two databases to store state. You can expand the existing PostgreSQL database in use for Immuta by creating Temporal databases like so:

1. Grant administrator privileges to the Postgres database role. Upon successfully completing this installation guide, you can optionally revoke this role grant:

2. Grant the Postgres user role to the current user. Upon successfully completing this installation guide, you can optionally revoke this role grant:

3. Create the new temporal databases and additional privileges for the Postgres user specified:

4. Connect to the new Temporal databases and run the following GRANT statements:

Enabling Temporal

To enable the Temporal deployment, set the following values. Include the tls settings if using a Cloud database that requires TLS:

Helm values

To improve the experience using the IEHC, two Helm value changes have been introduced. Before deploying the IEHC, you must perform the Helm value changes below.

PostgreSQL configuration

IEHC 2024.3.x and newer adds support for global and component-level PostgreSQL connection details. This means you only need to specify the PostgreSQL connection information once in the global scope and apply overrides (if necessary) at a component level.

If you installed IEHC 2024.2 LTS using our install guides, your immuta-values.yaml file probably looks something like this to configure your PostgreSQL connection for multiple components:

Now, with PostgreSQL configuration in the global scope, your immuta-values.yaml file can look like this to specify the PostgreSQL connection:

Feature flags

Feature flags have moved from environment variables in IEHC 2024.3.x and newer as well. You may now set feature flags globally, and then the IEHC will properly configure all applications for you. Migrate all feature flags from secure.extraEnvVars to global.featureFlags.

If you fail to migrate the values from secure.extraEnvVars to global.featureFlags , then Helm will display warnings similar to below:

Upgrade Immuta

After updating your immuta-values.yaml file to include any of the changes for the updates above, you can upgrade Immuta with the following command:

This guide demonstrates how to configure TLS termination for an .

Prerequisite

The must be completed before proceeding.

1. Edit immuta-values.yaml to include the following Helm values.

2. from a given public/private PEM formatted key pair.

3. Perform a to apply the changes made to immuta-values.yaml.

Refer to the for further assistance.

1. Edit immuta-values.yaml to include the following Helm values.

2. Perform a to apply the changes made to immuta-values.yaml.

Refer to the for further assistance.

1. Edit immuta-values.yaml to include the following Helm values.

2. Perform a to apply the changes made to immuta-values.yaml.

Refer to the for further assistance.

1. Edit immuta-values.yaml to include the following Helm values.

2. Perform a to apply the changes made to immuta-values.yaml.

Refer to the for further assistance.

1. Edit immuta-values.yaml to include the following Helm values.

2. from a given public/private PEM formatted key pair.

3. Perform a to apply the changes made to immuta-values.yaml.

Refer to the for further assistance.

The query engine is no longer installed by default. This guide demonstrates how to enable the query engine using the Immuta Enterprise Helm chart (IEHC).

If you are using any of the , you must enable the query engine.

Prerequisites

• The guide must be completed before proceeding.

• Validate that secret immuta-secret exists in the current namespace.

Create Kubernetes secret

1. Create a file named secret-data.env with the following content.

2. Create secret named immuta-legacy-secret from file secret-data.env

3. Delete file secret-data.env, as it's no longer needed.

Edit Helm values

1. Edit the immuta-values.yaml file to include the following Helm values.

2. Update all in the immuta-values.yaml file.

Apply Helm values

Perform a to apply the changes made to immuta-values.yaml.

This guide illustrates how to run R and Scala spark-submit jobs on Databricks, including prerequisites and caveats.

R spark-submit

Prerequisites

Before you can run spark-submit jobs on Databricks, complete the following steps.

1. Initialize the Spark session:

   1. Enter these settings into the R submit script to allow the R script to access Immuta data sources, scratch paths, and workspace tables: immuta.spark.acl.assume.not.privileged="true" and spark.hadoop.immuta.databricks.config.update.service.enabled="false".

   2. Once the script is written, upload the script to a location in dbfs/S3/ABFS to give the Databricks cluster access to it.

2. Because of how some user properties are populated in Databricks, load the SparkR library in a separate cell before attempting to use any SparkR functions.

Create the R spark submit Job

To create the R spark-submit job,

1. Go to the Databricks jobs page.

2. Create a new job, and select Configure spark-submit.

3. Set up the parameters:

   Note: The path dbfs:/path/to/script.R can be in S3 or ABFS (on Azure Databricks), assuming the cluster is configured with access to that path.

4. Edit the cluster configuration, and change the Databricks Runtime to be a .

5. Configure the section as you normally would for an Immuta cluster.

Scala spark-submit

Prerequisites

Before you can run spark-submit jobs on Databricks you must initialize the Spark session with the settings outlined below.

1. Configure the Spark session with immuta.spark.acl.assume.not.privileged="true" and spark.hadoop.immuta.databricks.config.update.service.enabled="false".

   Note: Stop your Spark session (spark.stop()) at the end of your job or the cluster will not terminate.

2. The spark submit job needs to be launched using a different classloader which will point at the designated user JARs directory. The following Scala template can be used to handle launching your submit code using a separate classloader:

Create the Scala spark-submit Job

To create the Scala spark-submit job,

1. Build and upload your JAR to dbfs/S3/ABFS where the cluster has access to it.

2. Select Configure spark-submit, and configure the parameters:

   Note: The fully-qualified class name of the class whose main function will be used as the entry point for your code in the --class parameter.

   Note: The path dbfs:/path/to/code.jar can be in S3 or ABFS (on Azure Databricks) assuming the cluster is configured with access to that path.

3. Edit the cluster configuration, and change the Databricks Runtime to a .

4. Include IMMUTA_INIT_ADDITIONAL_JARS_URI=dbfs:/path/to/code.jar in the "Environment Variables" (where dbfs:/path/to/code.jar is the path to your jar) so that the jar is uploaded to all the cluster nodes.

Caveats

• The user mapping works differently from notebooks because spark-submit clusters are not configured with access to the Databricks SCIM API. The cluster tags are read to get the cluster creator and match that user to an Immuta user.

• Privileged users (Databricks admins and allowlisted users) must be tied to an Immuta user and given access through Immuta to access data through spark-submit jobs because the setting immuta.spark.acl.assume.not.privileged="true" is used.

• There is an option of using the immuta.api.key setting with an Immuta API key generated on the Immuta profile page.

• Currently when an API key is generated it invalidates the previous key. This can cause issues if a user is using multiple clusters in parallel, since each cluster will generate a new API key for that Immuta user. To avoid these issues, manually generate the API key in Immuta and set the immuta.api.key on all the clusters or use a specified job user for the submit job.

Immuta offers several features to provide security for your users and Databricks clusters and to prove compliance and monitor for anomalies.

Authentication

Configuring the integration and registering data

Immuta supports the following authentication methods to configure the Databricks Spark integration and register data sources:

• OAuth machine-to-machine (M2M): Immuta uses the to integrate with , which allows Immuta to authenticate with Databricks using a client secret. Once Databricks verifies the Immuta service principal’s identity using the client secret, Immuta is granted a temporary OAuth token to perform token-based authentication in subsequent requests. When that token expires (after one hour), Immuta requests a new temporary token. See the for more details.

• Personal access token (PAT): This token gives Immuta temporary permission to push the cluster policies to the configured Databricks workspace and overwrite any cluster policy templates previously applied to the workspace when configuring the integration or to register securables as Immuta data sources.

User authentication

The built-in Immuta IAM can be used as a complete solution for authentication and fine-grained user entitlement. However, you can connect your existing identity management provider to Immuta to use that system for authentication and fine-grained user entitlement instead.

Each of the supported identity providers includes a specific set of configuration options that enable Immuta to communicate with the IAM system and map the users, permissions, groups, and attributes into Immuta.

See the guide for a list of supported providers and details.

See the for details and instructions on mapping Databricks user accounts to Immuta.

Cluster security

Data processing and encryption

See the for more information about transmission of policy decision data, encryption of data in transit and at rest, and encryption key management.

Protecting the Immuta configuration

Non-administrator users on an Immuta-enabled Databricks cluster must not have access to view or modify Immuta configuration, as this poses a security loophole around Immuta policy enforcement. allow you to securely apply environment variables to Immuta-enabled clusters.

Databricks secrets can be used in the environment variables configuration section for a cluster by referencing the secret path instead of the actual value of the environment variable.

See the for details and instructions on using Databricks secrets.

Scala cluster security

There are limitations to isolation among users in Scala jobs on a Databricks cluster. When data is broadcast, cached (spilled to disk), or otherwise saved to SPARK_LOCAL_DIR, it's impossible to distinguish between which user’s data is composed in each file/block. To address this vulnerability, Immuta suggests that you

• limit Scala clusters to Scala jobs only and

• require equalized projects, which will force all users to act under the same set of attributes, groups, and purposes with respect to their data access. This requirement guarantees that data being dropped into SPARK_LOCAL_DIR will have policies enforced and that those policies will be homogeneous for all users on the cluster. Since each user will have access to the same data, if they attempt to manually access other users' cached/spilled data, they will only see what they have access to via equalized permissions on the cluster. If project equalization is not turned on, users could dig through that directory and find data from another user with heightened access, which would result in a data leak.

See the for more details and configuration instructions.

Auditing and compliance

Immuta provides auditing features and governance reports so that data owners and governors can monitor users' access to data and detect anomalies in behavior.

You can view the information in these audit logs on or export the full audit logs to S3 and ADLS for long-term backup and processing with log data processors and tools. This capability fosters convenient integrations with log monitoring services and data pipelines.

See the for details about these capabilities and how they work with the Databricks Spark integration.

Databricks query audit

Immuta captures the code or query that triggers the Spark plan in Databricks, making audit records more useful in assessing what users are doing.

To audit what triggers the Spark plan, Immuta hooks into Databricks where notebook cells and JDBC queries execute and saves the cell or query text. Then, Immuta pulls this information into the audits of the resulting Spark jobs.

Immuta will audit queries that come from interactive notebooks, notebook jobs, and JDBC connections, but will not audit . Furthermore, Immuta only audits Spark jobs that are associated with Immuta tables. Consequently, Immuta will not audit a query in a notebook cell that does not trigger a Spark job, unless

See the page for examples of saved queries and the resulting audit records. To exclude query text from audit events, see the page.

Auditing all queries

Immuta supports auditing all queries run on a Databricks cluster, regardless of whether users touch Immuta-protected data or not.

See the for details and instructions.

Auditing queries run while impersonating another user

When a query is run by a user impersonating another user, the extra.impersonationUser field in the audit log payload is populated with the Databricks username of the user impersonating another user. The userId field will return the Immuta username of the user being impersonated:

See the for details about user impersonation.

Governance reports

Immuta governance reports allow users with the GOVERNANCE Immuta permission to use a natural language builder to instantly create reports that delineate user activity across Immuta. These reports can be based on various entity types, including users, groups, projects, data sources, purposes, policy types, or connection types.

See the page for a list of report types and guidance.

Once a Databricks securable is registered in Immut as a data source and you are subscribed to that data source, you must access that data through SQL:

See the sections below for more guidance on accessing data using , , and .

Delta Lake

When using Delta Lake, the API does not go through the normal Spark execution path. This means that Immuta's Spark extensions do not provide protection for the API. To solve this issue and ensure that Immuta has control over what a user can access, the Delta Lake API is blocked.

Spark SQL can be used instead to give the same functionality with all of Immuta's data protections. See the for a list of corresponding Spark SQL calls to use.

Spark direct file reads

In addition to supporting direct file reads through workspace and scratch paths, Immuta allows direct file reads in Spark for file paths. As a result, users who prefer to interact with their data using file paths or who have existing workflows revolving around file paths can continue to use these workflows without rewriting those queries for Immuta.

When reading from a path in Spark, the Immuta Databricks Spark plugin queries the Immuta Web Service to find Databricks data sources for the current user that are backed by data from the specified path. If found, the query plan maps to the Immuta data source and follows existing code paths for policy enforcement.

Users can read data from individual parquet files in a sub-directory and partitioned data from a sub-directory (or by using a where predicate). Expand the blocks below to view examples of reading data using these methods.

Limitations

• Direct file reads for Immuta data sources only apply to data sources created from tables, not data sources created from views or queries.

• If more than one data source has been created for a path, Immuta will use the first valid data source it finds. It is therefore not recommended to use this integration when more than one data source has been created for a path.

• In Databricks, multiple input paths are supported as long as they belong to the same data source.

• CSV-backed tables are not currently supported.

• Loading a delta partition from a sub-directory is not recommended by Spark and is not supported in Immuta. Instead, use a where predicate:

User impersonation

User impersonation allows Databricks users to query data as another Immuta user. To impersonate another user, see the .

This page illustrates how to configure the on the Immuta app settings page. To configure this integration via the Immuta API, see the .

For instructions on configuring Redshift Spectrum, see the guide.

Requirements

• A Redshift cluster with an RA3 node is required for the multi-database integration. You must use a Redshift RA3 instance type because Immuta requires cross-database views, which are only supported in Redshift RA3 instance types. For other instance types, you may configure a single-database integration using one of the .

• For automated installations, the credentials provided must be a Superuser or have the ability to create databases and users and modify grants.

• The must be set to false (default setting) for your Redshift cluster.

Add a Redshift integration

1. Click the App Settings icon in the left sidebar.

2. Click the Integrations tab.

3. Click the +Add Integration button and select Redshift from the dropdown menu.

4. Complete the Host and Port fields.

5. Enter an Immuta Database. This is a new database where all secure schemas and Immuta created views will be stored.

6. Opt to check the Enable Impersonation box and customize the Impersonation Role name as needed. This will allow users to natively impersonate another user.

Select your configuration method

You have two options for configuring your Redshift environment:

• : Grant Immuta one-time use of credentials to automatically configure your Redshift environment and the integration.

• : Run the Immuta script in your Redshift environment yourself to configure your environment and the integration.

Automatic setup

1. Select Automatic.

2. Enter an Initial Database from your Redshift integration for Immuta to use to connect.

3. Use the dropdown menu to select your Authentication Method.

   1. Username and Password: Enter the Username and Password of the privileged user.

   2. AWS Access Key: Enter the Database User, Access Key ID, and Secret Key. Opt to enter in the Session Token.

Manual setup

1. Select Manual and download both of the bootstrap scripts from the Setup section.

2. Run the bootstrap script (initial database) in the Redshift initial database.

3. Run the bootstrap script (Immuta database) in the new Immuta Database in Redshift.

4. Choose your authentication method, and enter the information of the newly created account.

Save the configuration

Click Save.

Register data

.

Edit a Redshift integration

1. Click the App Settings icon in the left sidebar.

2. Navigate to the Integrations tab and click the down arrow next to the Redshift Integration.

3. Edit the field you want to change. Note any field shadowed is not editable, and the integration must be disabled and re-installed to change it.

4. Enter Username and Password.

5. Click Save.

Remove a Redshift integration

1. Click the App Settings icon in the left sidebar.

2. Navigate to the Integrations tab and click the down arrow next to the Redshift Integration.

3. Click the checkbox to disable the integration.

4. Enter the username and password that were used to initially configure the integration.

5. Click Save.

Allow Immuta to create secure views of your external tables through one of these methods:

• that contains the external tables: Instead of creating an immuta database that manages all schemas and views created when Redshift data is registered in Immuta, the integration adds the Immuta-managed schemas and views to an existing database in Redshift

• and re-create all of your external tables in that database.

For an overview of the integration, see the documentation.

Requirements

• A Redshift cluster with an AWS row-level security patch applied. for guidance.

• that is .

• The must be set to false (default setting) for your Redshift cluster.

• The Redshift role used to run the Immuta bootstrap script must have the following privileges when configuring the integration to

  • Use an existing database:

    • ALL PRIVILEGES ON DATABASE for the database you configure the integration with, as you must manage grants on that database.

    • CREATE USER

    • GRANT TEMP ON DATABASE

  • Create a new database:

    • CREATE DATABASE

    • CREATE USER

    • GRANT TEMP ON DATABASE

    • REVOKE ALL PRIVILEGES ON DATABASE

• .

Use an existing database

1. Click the App Settings icon in the left sidebar.

2. Click the Integrations tab.

3. Click the +Add Integration button and select Redshift from the dropdown menu.

4. Complete the Host and Port fields.

5. Enter the name of the database you created the external schema in as the Immuta Database. This database will store all secure schemas and Immuta-created views.

6. Opt to check the Enable Impersonation box and customize the Impersonation Role name as needed. This will allow users to natively impersonate another user.

7. Select Manual and download both of the bootstrap scripts from the Setup section. The specified role used to run the bootstrap needs to have the following privileges:

   • ALL PRIVILEGES ON DATABASE for the database you configure the integration with, as you must manage grants on that database.

   • CREATE USER

   • GRANT TEMP ON DATABASE

8. Run the bootstrap script (Immuta database) in the Redshift database that contains the external schema.

9. Choose your authentication method, and enter the credentials from the bootstrap script for the Immuta_System_Account.

10. Click Save.

Register data

.

Create a new Immuta database

1. Click the App Settings icon in the left sidebar.

2. Click the Integrations tab.

3. Click the +Add Integration button and select Redshift from the dropdown menu.

4. Complete the Host and Port fields.

5. Enter an Immuta Database. This is a new database where all secure schemas and Immuta created views will be stored.

6. Opt to check the Enable Impersonation box and customize the Impersonation Role name as needed. This will allow users to natively impersonate another user.

7. Select Manual and download both of the bootstrap scripts from the Setup section. The specified role used to run the bootstrap needs to have the following privileges:

   • ALL PRIVILEGES ON DATABASE for the database you configure the integration with, as you must manage grants on that database.

   • CREATE DATABASE

   • CREATE USER

   • GRANT TEMP ON DATABASE

8. Run the bootstrap script (initial database) in the Redshift initial database.

9. Run the bootstrap script (Immuta database) in the new Immuta Database in Redshift.

10. Choose your authentication method, and enter the credentials from the bootstrap script for the Immuta_System_Account.

11. Click Save.

Then, add your external tables to the Immuta database.

Register data

.

This page describes the Redshift integration, configuration options, and features. For a tutorial to enable this integration, see the .

Feature Availability