This section illustrates how to install Immuta on Kubernetes using the Immuta Enterprise Helm chart.

Requirements

This reference guide provides an overview of the Immuta Enterprise Helm chart version requirements and infrastructure recommendations.

The guides in this section illustrate how to install and deploy Immuta in your Kubernetes environment.

The guides in this section illustrate how to upgrade Immuta.

The guides in this section illustrate how to configure your Immuta Enterprise Helm chart for various scenarios, including optimizing your deployment for production environments.

This guide provides links to additional resources for disaster recovery strategies.

This page provides troubleshooting guidance and outlines frequently asked questions for the Immuta installation.

This page introduces the core concepts and terminology essential for understanding the installation material.

Immuta helps you achieve the following outcomes in your data platform:

• Simplify Operations: Immuta’s dynamic access control and policy management require 93x fewer data policies to manage access control in your data platform according to the GigaOm study. It is simple and scalable, which improves change management and lowers the total cost of ownership of cloud data management.

• Improve data security: Immuta helps prove compliance with rules and regulations, even when securing hundreds of thousands of tables. An Immuta customer, Swedbank, migrated all critical analytics workloads to the cloud in less than 12 months, including over 100 terabytes from more than 2,500 sources.

• Unlock data’s value: Immuta helps organizations get access to more data 100x faster, which translates to improved productivity. An Immuta customer, enabled faster access to data, resulting in a 60x increase in data usage and greater productivity.

How does Immuta do it?

Immuta provides three modules to create a full data security platform suite.

Sensitive data discovery and classification:

Discover sensitive data from millions of fields without manual effort. With over 60 pre-built and domain-specific identifiers, you can tailor data classification to your unique business needs based on your desired confidence level.

Continuous data security monitoring:

Leverage timely insights into data access and user activity with anomaly indicators for faster analysis and proactive actions.

Data security and access control:

Immuta’s attribute-based access control (ABAC) delivers scalable data access without role explosion, and dynamic data masking ensures the right users can access the right data.

The guides in this section illustrate how to install and deploy Immuta in your Kubernetes environment.

Prerequisites

Checklist

• have been granted to create resources in the cluster.

Quickstart

Get started quickly with these essential guides. For a more comprehensive understanding and advanced configurations, .

1. Complete the guide that corresponds with your Kubernetes cluster's distribution.

   • : This guide includes instructions for

     • Amazon Elastic Kubernetes Service (EKS)

     • Google Kubernetes Engine (GKE)

To migrate from the private preview version of table grants (available before September 2022) to the GA version, complete the steps below.

1. Navigate to the App Settings page.

2. Click Integration Settings in the left panel, and scroll to the Global Integrations Settings section.

3. Uncheck the Snowflake Table Grants checkbox to disable the feature.

Introduced in 2024.2, the Immuta Enterprise Helm chart (IEHC) is an entirely new Helm chart used to deploy Immuta. Unlike the previous Immuta Helm chart (IHC), the IEHC shares the same version as the Immuta product. Each version of the chart supports a singular version of Immuta. Upgrading the Immuta version now entails upgrading the underlying Helm chart. Failure to do so will lead to an unsupported configuration.

Recommended

• : If you're upgrading from 2024.1.x or older, you must first migrate to the new Helm chart, as these versions were all installed using the legacy IHC. This migration process will include upgrading Immuta.

• : Upgrade from v2024.2 LTS to v2024.3 using the Immuta Enterprise Helm chart.

Legacy

• : Upgrade from v2024.2 LTS to v2024.3 using the legacy Immuta Helm chart.

1. Navigate to the App Settings page.

2. Scroll to the Global Integrations Settings section.

3. Opt to change the Role Prefix. Snowflake table grants creates a new Snowflake role for each Immuta user. To ensure these Snowflake role names do not collide with existing Snowflake roles, each Snowflake role created for Snowflake table grants requires a common prefix. When using multiple Immuta accounts within a single Snowflake account, the Snowflake table grants role prefix should be unique for each Immuta account. The prefix must adhere to Snowflake identifier requirements and be less than 50 characters. Once the configuration is saved, the prefix cannot be modified; however, the Snowflake table grants feature can be disabled and re-enabled to change the prefix.

4. Finish configuring your integration by following one of these guidelines:

   • New Snowflake integration: Set up a new Snowflake integration by following the .

   • Existing Snowflake integration (automatic setup): You will be prompted to enter connection information for a Snowflake user. Immuta will execute the migration to Snowflake table grants using a connection established with this Snowflake user. The Snowflake user you provide here must have Snowflake privileges to run these .

1. Click the App Settings icon in the sidebar and scroll to the Global Integration Settings section.

2. Click the Enable Snowflake Low Row Access Policy Mode checkbox to enable the feature.

3. Confirm to allow Immuta to automatically disable impersonation for the Snowflake integration. If you do not confirm, you will not be able to enable Snowflake low row access policy mode.

4. Click Save.

Configure your Snowflake integration

If you already have a configured, you don't need to reconfigure your integration. Your Snowflake policies automatically refresh when you enable Snowflake low row access policy mode.

1. . Note that you will not be able to enable project workspaces or user impersonation with Snowflake low row access policy mode enabled.

2. Click Save and Confirm your changes.

Immuta is compatible with Snowflake Secure Data Sharing. Using both Immuta and Snowflake, organizations can share the policy-protected data of their Snowflake database with other Snowflake accounts with Immuta policies enforced in real time.

Prerequisites:

• Snowflake integration enabled

• Snowflake tables registered in Immuta as data sources

Create Immuta Policies to Protect the Data

Required Permission: Immuta: GOVERNANCE

to fit your organization's compliance requirements.

It's important to understand that subscription policies are not relevant to Snowflake data shares, because the act of sharing the data is the subscription policy. Data policies can be enforced on the consuming account from the producer account on a share following these instructions.

Register the Snowflake Data Consumer with Immuta

Required Permission: Immuta: USER_ADMIN

To register the Snowflake data consumer in Immuta,

1. .

2. to match the account ID for the data consumer. This value is the output on the data consumer side when SELECT CURRENT_ACCOUNT() is run in Snowflake.

3. for your organization's policies.

4. .

Create the Snowflake Data Share

Required Permission: Snowflake ACCOUNTADMIN

To share the policy-protected data source,

1. of the Snowflake table that has been registered in Immuta.

2. Grant reference usage on the Immuta database to the share you created:

   Replace the content in angle brackets above with the name of your Immuta database and Snowflake data share.

Immuta is compatible with Snowflake Secure Data Sharing. Using both Immuta and Snowflake, organizations can share the policy-protected data of their Snowflake database with other Snowflake accounts with Immuta policies enforced in real time. This integration gives data consumers a live connection to the data and relieves data providers of the legal and technical burden of creating static data copies that leave their Snowflake environment.

Requirements:

• Snowflake Enterprise Edition or higher

• Immuta's table grants feature

Configuration

This method requires that the data consumer account is registered as an Immuta user with the Snowflake user name equal to the consuming account.

At that point, the user that represents the account being shared with can have the appropriate attributes and groups assigned to them, relevant to the data policies that need to be enforced. Once that user has access to the share in the consuming account (not managed by Immuta), they can query the share with the data policies from the producer account enforced because Immuta is treating that account as if they are a single user in Immuta.

For a tutorial on this workflow, see the .

Benefits

Using Immuta with Snowflake Data Sharing allows the sharer to

• Only need limited knowledge of the context or goals of the existing policies in place: Because the sharer is not editing or creating policies to share their data, they only need a limited knowledge of how the policies work. Their main responsibility is making sure they properly represent the attributes of the data consumer (the account being shared to).

• Leave policies untouched.

All application state is stored in the PostgreSQL metadata database; therefore, recovering from a disaster event only entails restoring the aforementioned PostgreSQL database. Consult each cloud provider's point-in-time recovery (PITR) documentation for guidance:

• Amazon RDS for PostgreSQL

For more details about point-in-time recovery, see the .

This guide demonstrates how to upgrade an existing Immuta deployment installed with the Immuta Helm chart (IHC) to v2024.3.

Prerequisites

Checklist

• The Immuta Helm chart (IHC) has been updated to the latest version.

Upgrade Immuta

1. Edit immuta-values.yaml to include the following Helm values.

1. Perform a to apply the changes made to immuta-values.yaml.

In this configuration, Immuta is able to rely on Databricks-native security controls, reducing overhead. The key security control here is the enablement of process isolation. This prevents users from obtaining unintentional access to the queries of other users. In other words, masked and filtered data is consistently made accessible to users in accordance with their assigned attributes. This Immuta cluster configuration relies on Py4J security being enabled.

Many Python ML classes (such as LogisticRegression, StringIndexer, and DecisionTreeClassifier) and dbutils.fs are unfortunately not supported with Py4J security enabled. Users will also be unable to use the Databricks Connect client library. Additionally, only Python and SQL are available as supported languages.

For full details on Databricks’ best practices in configuring clusters, read their .

The immuta database on Immuta-enabled clusters allows Immuta to track Immuta-managed data sources separately from remote Databricks tables so that policies and other security features can be applied. However, Immuta supports raw tables in Databricks, so table-backed queries do not need to reference this database. When configuring a Databricks cluster, you can hide immuta from any calls to SHOW DATABASES so that users are not confused or misled by that database.

Hide the immuta Database

When configuring a Databricks cluster, hide immuta by using the following environment variable in the :

Then, Immuta will not show this database when a SHOW DATABASES query is performed.

This integration allows you to manage and access data in your Databricks account across all of your workspaces. With Immuta’s Databricks Unity Catalog integration, you can write your policies in Immuta and have them enforced automatically by Databricks across data in your Unity Catalog metastore.

Getting started

This getting started guide outlines how to integrate Databricks Unity Catalog with Immuta.

How-to guides

• : Configure the Databricks Unity Catalog integration.

• : Migrate from the legacy Databricks Spark integrations to the Databricks Unity Catalog integration.

Reference guide

: This guide describes the design and components of the integration.

When you enable Unity Catalog, Immuta automatically migrates your existing Databricks data sources in Immuta to reference the legacy hive_metastore catalog to account for Unity Catalog's three-level hierarchy. New data sources will reference the Unity Catalog metastore you create and attach to your Databricks workspace.

Because the hive_metastore catalog is not managed by Unity Catalog, existing data sources in the hive_metastore cannot have Unity Catalog access controls applied to them. Data sources in the Hive Metastore must be managed by the Databricks Spark integration.

To allow Immuta to administer Unity Catalog access controls on that data, move the data to Unity Catalog and re-register those tables in Immuta by completing the steps below. If you don't move all data before configuring the integration, metastore magic will protect your existing data sources throughout the migration process.

1. Ensure that all Databricks clusters that have Immuta installed are stopped and the Immuta configuration is removed from the cluster. Immuta-specific cluster configuration is no longer needed with the Databricks Unity Catalog integration.

2. Move all data into Unity Catalog before configuring Immuta with Unity Catalog. Existing data sources will need to be re-created after they are moved to Unity Catalog and the Unity Catalog integration is configured.

3. .

In this integration, Immuta generates policy-enforced views in your configured Redshift schema for tables registered as Immuta data sources.

Getting started

This guide outlines how to integrate Redshift with Immuta.

How-to guides

• : Configure the integration in Immuta.

• : Configure Redshift Spectrum in Immuta.

Reference guide

: This guide describes the design and components of the integration.

Prerequisites

This upgrade step is necessary if you meet both of the following criteria:

• You have the Snowflake low row access policy mode enabled in private preview.

• You have user impersonation enabled.

If you do not meet this criteria, follow the instructions on the .

Upgrade to Snowflake low row access policy mode

To upgrade to the generally available version of the feature, on the app settings page and then re-enable it.

This guide demonstrates how to configure an external key-value cache (such as Redis or Memcached) with the Immuta Enterprise Helm chart (IEHC).

Immuta integrates with your data platforms and external catalogs so you can register your data and effectively manage access controls on that data.

This section includes concept, reference, and how-to guides for configuring your data platform integration, registering data sources, and connecting your external catalog so that you can discover, monitor, and protect sensitive data.

This reference guide outlines the features, policies, and audit capabilities supported by each integration.

This guide demonstrates how to verify signed artifacts (i.e., container images, Helm charts) hosted on ocir.immuta.com using from .

1 - Install the Library

1. In the Databricks Clusters UI, install your third-party library .jar or Maven artifact with Library Source Upload, DBFS, DBFS/S3, or Maven

This guide demonstrates how to configure a private container registry with the Immuta Enterprise Helm chart (IEHC).

In this configuration, you are able to rely on the Databricks-native security controls. The key security control here is the enablement of process isolation. This prevents users from obtaining unintentional access to the queries of other users. In other words, masked and filtered data is consistently made accessible to users in accordance with their assigned attributes.

Like the Python & SQL configuration, Py4j security is enabled for the Python & SQL & R configuration. However, because R has been added Immuta enables the SecurityManager, in addition to Py4j security, to provide more security guarantees. For example, by default all actions in R execute as the root user; among other things, this permits access to the entire filesystem (including sensitive configuration data), and, without iptable restrictions, a user may freely access the cluster’s cloud storage credentials. To address these security issues, Immuta’s initialization script wraps the R and Rscript binaries to launch each command as a temporary, non-privileged user with limited filesystem and network access and installs the Immuta SecurityManager, which prevents users from bypassing policies and protects against the above vulnerabilities from within the JVM.

The following conventions are used throughout the installation material.

Angle brackets ( < and > )

Phrases wrapped in angle brackets (i.e., <, >) are placeholders used to indicate values that must be substituted with user-provided values. Placeholders are typically written in either , or ; the following placeholders are equivalent:

It is most secure to leverage an equalized project when working in a Scala cluster; however, it is not required to limit Scala to equalized projects. This document outlines security recommendations for Scala clusters and discusses the security risks involved when equalized projects are not used.

Local or remote mode

Immuta supports the use of external metastores in , following the same configuration detailed in the .

Configure external Hive metastore

While you're onboarding Snowflake data sources and designing policies, you don't want to disrupt your Snowflake users' existing workflows. Instead, you want to gradually onboard Immuta through a series of successive changes that will not impact your existing Snowflake users.

A phased onboarding approach to configuring the Snowflake integration ensures that your users will not be immediately affected by changes as you add data sources and configure policies.

Several features allow you to gradually onboard data sources and policies in Immuta:

• : By default, no policy is applied at registration time; instead of applying a restrictive policy immediately upon registration, the table is registered in Immuta and waits for a policy to be applied, if ever.

  There are several benefits to this design:

CDF shows the row-level changes between versions of a Delta table. The changes displayed include row data and metadata that indicates whether the row was inserted, deleted, or updated.

Immuta does not support applying policies to the changed data, and the CDF cannot be read for data source tables if the user does not have access to the raw data in Databricks. However, the CDF can be read if the querying user is allowed to read the raw data and one of the following statements is true:

• the table is in the current workspace,

• the table is in a scratch path,

Where Scala language support is needed, this configuration can be used in the Custom .

According to Databricks’ cluster type support documentation, Scala clusters are intended for . However, nothing inherently prevents a Scala cluster from being configured for multiple users. Even with the Immuta SecurityManager enabled, there are limitations to user isolation within a Scala job.

For a secure configuration, it is recommended that clusters intended for Scala workloads are limited to Scala jobs only and are made homogeneous through the use of or externally via convention/cluster ACLs. (In homogeneous clusters, all users are at the same level of groups/authorizations; this is enforced externally, rather than directly by Immuta.)

For full details on Databricks’ best practices in configuring clusters, read their .

If a Databricks cluster needs to be manually updated to reflect changes in the Immuta init script or cluster policies, you can remove and set up your integration again to get the updated policies and init script.

1. Log in to Immuta as an Application Admin.

2. Click the App Settings icon in the left sidebar and scroll to the Integration Settings section.

3. Your existing Databricks Spark integration should be listed here; expand it and note the configuration values. Now select Remove

In this integration, Immuta policies are translated into Starburst rules and permissions and applied directly to tables within users’ existing catalogs.

This guide outlines how to integrate Starburst with Immuta.

This page describes how the Security Manager is disabled for Databricks clusters that do not allow R or Scala code to be executed. Databricks Administrators should place the desired configuration in the Spark environment variables (recommended) or immuta_conf.xml (not recommended).

Automatic Disabling of the Security Manager

The Immuta Security Manager is an essential element of the Databricks Spark deployment that ensures users can't perform unauthorized actions when using Scala and R, since those languages have features that allow users to circumvent policies without the Security Manager enabled. However, the Security Manager must inspect the call stack every time a permission check is triggered, which adds overhead to queries. To improve Immuta's query performance on Databricks, Immuta disables the Security Manager when Scala and R are not being used.

The cluster init script checks the cluster’s configuration and automatically removes the Security Manager configuration when

• spark.databricks.repl.allowedlanguages is a subset of {python, sql}

• IMMUTA_SPARK_DATABRICKS_PY4J_STRICT_ENABLED is true

When the cluster is configured this way, Immuta can rely on Databricks' process isolation and Py4J security to prevent user code from performing unauthorized actions.

Note: Immuta still expects the spark.driver.extraJavaOptions and spark.executor.extraJavaOptions to be set and pointing at the Security Manager.

Beyond disabling the Security Manager, Immuta will skip several startup tasks that are required to secure the cluster when Scala and R are configured, and fewer permission checks will occur on the Driver and Executors in the Databricks cluster, reducing overhead and improving performance.

Caveats

• There are still cases that require the Security Manager; in those instances, Immuta creates a fallback Security Manager to check the code path, so the IMMUTA_INIT_ALLOWED_CALLING_CLASSES_URI environment variable must always point to a valid calling class file.

• Databricks’ dbutils.fs is blocked by their PY4J security; therefore, it can’t be used to access scratch paths.

The following guides offer practical guidance for handling common challenges and configurations.

Ingress configuration

Configure Ingress to complete your installation and access your Immuta application.

Configure TLS termination for an Ingress resource.

Verify artifacts hosted on the ocir.immuta.com OCI registry.

Follow these best practices when deploying Immuta in your production environment.

Update the credentials referenced in the Immuta Enterprise Helm chart.

Configure an external key-value cache (such as Redis or Memcached) with the Immuta Enterprise Helm chart.

Enable these legacy services for your deployment if they are required for your business use case:

• If you are using any of the data platforms below, you must enable the query engine:

  • Amazon Redshift

  • Azure Synapse Analytics

  • Google BigQuery

Configure pulling images from a private registry.

Tips when installing Immuta without internet access.

This guide demonstrates how to update credentials referenced in the Immuta Enterprise Helm chart (IEHC).

Kubernetes secrets

Edit secrets

1. Validate that secret immuta-secret exists in the current namespace.

2. Edit secret immuta-secret in place.

3. Edit secret immuta-legacy-secret in place. Skip this step if the legacy query engine and fingerprint services are disabled (the default).

Legacy query engine

1. Validate that secret immuta-legacy-secret exists in the current namespace.

2. Get the query engine replica count, this value will be referenced in subsequent step(s).

3. Scale the replica count down to 1.

4. Get the query engine pod name, this value will be referenced in subsequent step(s).

Apply Helm values

1. Update credentials in the immuta-values.yaml file.

2. Perform a to apply the changes made to immuta-values.yaml. Update the with your own release name.

Immuta manages access to Snowflake tables by administering Snowflake row access policies and column masking policies on those tables, allowing users to query tables directly in Snowflake while dynamic policies are enforced.

Getting started

This getting started guide outlines how to integrate your Snowflake account with Immuta.

How-to guides

• : Configure the Snowflake integration.

• : Migrate to using Snowflake table grants in your Snowflake integration.

• : Manage integration settings or delete your existing Snowflake integration.

• Integration settings:

Reference guides

• : A phased onboarding approach to configuring the Snowflake integration ensures that your users will not be immediately affected by changes as you add data sources and policies. This guide describes the settings and requirements for implementing this phased approach.

• : This reference guide describes the design and features of the Snowflake integration.

• : Organizations can share the policy-protected data of their Snowflake database with other Snowflake accounts with Immuta policies enforced in real time. This guide describes the components of using Immuta with Snowflake data shares.

This page outlines how to access DBFS in Databricks for non-sensitive data. Databricks Administrators should place the desired configuration in the Spark environment variables (recommended) or the immuta_conf.xml file (not recommended).

DBFS FUSE Mount

This feature (provided by Databricks) mounts DBFS to the local cluster filesystem at /dbfs. Although disabled when using process isolation, this feature can safely be enabled if raw, unfiltered data is not stored in DBFS and all users on the cluster are authorized to see each other’s files. When enabled, the entirety of DBFS essentially becomes a scratch path where users can read and write files in /dfbs/path/to/my/file as though they were local files.

For example,

In Python,

Note: This solution also works in R and Scala.

Enable DBFS FUSE Mount

To enable the DBFS FUSE mount, set this configuration: immuta.spark.databricks.dbfs.mount.enabled=true.

Scala DBUtils (and %fs magic) with Scratch Paths

Scratch paths will work when performing arbitrary remote filesystem operations with fs magic or Scala dbutils.fs functions. For example,

Configure Scala DBUtils (and %fs magic) with Scratch Paths

To support %fs magic and Scala DBUtils with scratch paths, configure

Configure DBUtils in Python

To use dbutils in Python, set this configuration: immuta.spark.databricks.py4j.strict.enabled=false.

Example Workflow

This section illustrates the workflow for getting a file from a remote scratch path, editing it locally with Python, and writing it back to a remote scratch path.

1. Get the file from remote storage:

2. Make a copy if you want to explicitly edit localScratchFile, as it will be read-only and owned by root:

3. Write the new file back to remote storage:

In addition to supporting direct file reads through workspace and scratch paths, Immuta allows direct file reads in Spark for file paths. As a result, users who prefer to interact with their data using file paths or who have existing workflows revolving around file paths can continue to use these workflows without rewriting those queries for Immuta.

When reading from a path in Spark, the Immuta Databricks Spark plugin queries the Immuta Web Service to find Databricks data sources for the current user that are backed by data from the specified path. If found, the query plan maps to the Immuta data source and follows existing code paths for policy enforcement.

Read Data

Users can read data from individual parquet files in a sub-directory and partitioned data from a sub-directory (or by using a where predicate). Use the tabs below to view examples of reading data using these methods.

Limitations

• Direct file reads for Immuta data sources only apply to table-backed Immuta data sources, not data sources created from views or queries.

• If more than one data source has been created for a path, Immuta will use the first valid data source it finds. It is therefore not recommended to use this integration when more than one data source has been created for a path.

• In Databricks, multiple input paths are supported as long as they belong to the same data source.

• CSV-backed tables are not currently supported.

This integration enforces policies on Databricks tables registered as data sources in Immuta, allowing users to query policy-enforced data on Databricks clusters (including job clusters). Immuta policies are applied to the plan that Spark builds for users' queries, all executed directly against Databricks tables.

The guides in this section outline how to integrate Databricks Spark with Immuta.

How-to guides

• Databricks Spark configuration: Configure the Databricks Spark integration.

• : Access DBFS in Databricks for non-sensitive data.

• : Allow Immuta users to access tables that are not protected by Immuta.

• : Hide the Immuta database from users in Databricks, since user queries do not need to reference it.

• : Run R and Scala spark-submit jobs on your Databricks cluster.

• : Raise the caching on-cluster and lower the cache timeouts for the Immuta web service to allow use of project UDFs in Spark jobs.

• : Use an existing Hive external metastore instead of the built-in metastore.

Reference guides

• : This guide describes the design and components of the integration.

• Configuration settings: These guides describe various integration settings that can be configured, including , cluster policies, and .

• : This guide describes Immuta's support of Databricks change data feed.

• : The trusted libraries feature allows Databricks cluster administrators to avoid Immuta security manager errors when using third-party libraries. This guide describes the feature and its configuration.

This configuration does not rely on Databricks-native Py4j security to secure the cluster, while process isolation is still enabled to secure filesystem and network access from within Python processes. On an Immuta-enabled cluster, once Py4J security is disabled the Immuta SecurityManager is installed to prevent nefarious actions from Python in the JVM. Disabling Py4J security also allows for expanded Python library support, including many Python ML classes (such as LogisticRegression, StringIndexer, and DecisionTreeClassifier) and dbutils.fs.

By default, all actions in R will execute as the root user. Among other things, this permits access to the entire filesystem (including sensitive configuration data). And without iptable restrictions, a user may freely access the cluster’s cloud storage credentials. To properly support the use of the R language, Immuta’s initialization script wraps the R and Rscript binaries to launch each command as a temporary, non-privileged user. This user has limited filesystem and network access. The Immuta SecurityManager is also installed to prevent users from bypassing policies and protects against the above vulnerabilities from within the JVM.

The SecurityManager will incur a small increase in performance overhead; average latency will vary depending on whether the cluster is homogeneous or heterogeneous. (In homogeneous clusters, all users are at the same level of groups/authorizations; this is enforced externally, rather than directly by Immuta.)

When users install third-party Java/Scala libraries, they will be denied access to sensitive resources by default. However, cluster administrators can specify which of the installed Databricks libraries should be by Immuta.

A homogeneous cluster is recommended for configurations where Py4J security is disabled. If all users have the same level of authorization, there would not be any data leakage, even if a nefarious action was taken.

For full details on Databricks’ best practices in configuring clusters, read their .

In this integration, Immuta generates policy-enforced views in a schema in your configured Azure Synapse Analytics Dedicated SQL pool for tables registered as Immuta data sources.

Getting started

This guide outlines how to integrate Azure Synapse Analytics with Immuta.

How-to guide

: Configure the integration in Immuta.

Reference guide

: This guide describes the design and components of the integration.

This page outlines the configuration for setting up project UDFs, which allow users to set their current project in Immuta through Spark. For details about the specific functions available and how to use them, see the Use Project UDFs (Databricks) page.

Web Service and On-Cluster Caches

Immuta caches a mapping of user accounts and users' current projects in the Immuta Web Service and on-cluster. When users change their project with UDFs instead of the Immuta UI, Immuta invalidates all the caches on-cluster (so that everything changes immediately) and the cluster submits a request to change the project context to a web worker. Immediately after that request, another call is made to a web worker to refresh the current project.

To allow use of project UDFs in Spark jobs, raise the caching on-cluster and lower the cache timeouts for the Immuta Web Service. Otherwise, caching could cause dissonance among the requests and calls to multiple web workers when users try to change their project contexts.

Recommended Configuration

1 - Lower Web Service Cache Timeout

1. Click the App Settings icon in the left sidebar and scroll to the HDFS Cache Settings section.

2. Lower the Cache TTL of HDFS user names (ms) to 0.

3. Click Save.

2 - Raise Cache Timeout On-Cluster

In the Spark environment variables section, set the IMMUTA_CURRENT_PROJECT_CACHE_TIMEOUT_SECONDS and IMMUTA_PROJECT_CACHE_TIMEOUT_SECONDS to high values (like 10000).

Note: These caches will be invalidated on cluster when a user calls immuta.set_current_project, so they can effectively be cached permanently on cluster to avoid periodically reaching out to the web service.

Blocking UDFs

If your compliance requirements restrict users from changing projects within a session, you can block the use of Immuta's project UDFs on a Databricks Spark cluster. To do so, configure the immuta.spark.databricks.disabled.udfs option as described on the .

Frequently asked questions

How can I ensure the fully qualified domain name (FQDN) is resolvable from within the Kubernetes cluster?

To or a Snowflake integration, you have two options:

• Automatic: Grant Immuta one-time use of credentials with the following privileges to automatically edit or remove the integration:

  • CREATE DATABASE ON ACCOUNT WITH GRANT OPTION

The Snowflake low row access policy mode improves query performance in Immuta's Snowflake integration by decreasing the number of Immuta creates and by using table grants to manage user access.

Immuta manages access to Snowflake tables by administering and on those tables, allowing users to query them directly in Snowflake while policies are enforced.

Without Snowflake low row access policy mode enabled, row access policies are created and administered by Immuta in the following scenarios:

• are disabled and a subscription policy that does not automatically subscribe everyone to the data source is applied. Immuta administers Snowflake row access policies to filter out all the rows to restrict access to the entire table when the user doesn't have privileges to query it. However, if table grants are disabled and a subscription policy is applied that grants everyone access to the data source automatically, Immuta does not create a row access policy in Snowflake. See the for details about these policy types.

Contact your Immuta representative to enable this feature in your Immuta tenant.

Configure the Snowflake integration

The warehouse you select when configuring the Snowflake integration uses compute resources to set up the integration, register data sources, orchestrate policies, and run jobs like sensitive data discovery. Snowflake credit charges are based on the size of and amount of time the warehouse is active, not the number of queries run.

This document prescribes how and when to adjust the size and scale of clusters for your warehouse to manage workloads so that you can use Snowflake compute resources the most cost effectively.

In general, increase the size of and number of clusters for the warehouse to handle heavy workloads and multiple queries. Workloads are typically lighter after data sources are onboarded and policies are established in Immuta, so compute resources can be reduced after those workloads complete.

Integration and data source registration warehouse use

When using Delta Lake, the API does not go through the normal Spark execution path. This means that Immuta's Spark extensions do not provide protection for the API. To solve this issue and ensure that Immuta has control over what a user can access, the Delta Lake API is blocked.

Spark SQL can be used instead to give the same functionality with all of Immuta's data protections.

Requests

Below is a table of the Delta Lake API with the Spark SQL that may be used instead.

The how-to guides linked on this page illustrate how to integrate Starburst (Trino) with Immuta.

Configure your Starburst (Trino) integration

These guides provide information on the recommended features to enable with Starburst (Trino).

1. .

• Error Message: py4j.security.Py4JSecurityException: Constructor <> is not whitelisted

• Explanation: This error indicates you are being blocked by Py4j security rather than the Immuta Security Manager. is strict and generally ends up blocking many ML libraries.

• Solution: Turn off Py4j security on the offending cluster by setting IMMUTA_SPARK_DATABRICKS_PY4J_STRICT_ENABLED=false

The how-to guides linked on this page illustrate how to integrate Redshift with Immuta.

Requirement: Redshift cluster with an RA3 node is required for the multi-database integration. For other instance types, you may configure a single-database integration using one of the .

Configure your Redshift integration

These guides provide information on the recommended feature to enable with Redshift.

Snowflake column lineage specifies how data flows from source tables or columns to the target tables in write operations. When Snowflake lineage tag propagation is enabled in Immuta, Immuta automatically applies tags added to a Snowflake table to its descendant data source columns in Immuta so you can build policies using those tags to restrict access to sensitive data.

Snowflake Access History tracks user read and write operations. Snowflake column lineage extends this Access History to specify how data flows from source columns to the target columns in write operations, allowing data stewards to understand how sensitive data moves from ancestor tables to target tables so that they can

• trace data back to its source to validate the integrity of dashboards and reports,

• identify who performed write operations to meet compliance requirements,

• evaluate data quality and pinpoint points of failure, and

• tag sensitive data on source tables without having tag columns on their descendant tables.

However, tagging sensitive data doesn’t innately protect that data in Snowflake; users need Immuta to disseminate these lineage tags automatically to descendant tables registered in Immuta so data stewards can build policies using the semantic and business context captured by those tags to restrict access to sensitive data. When Snowflake lineage tag propagation is enabled, Immuta propagates tags applied to a data source to its descendant data source columns in Immuta, which keeps your data inventory in Immuta up-to-date and allows you to protect your data with policies without having to manually tag every new Snowflake data source you register in Immuta.

Data flow

1. An application administrator enables the feature on the Immuta app settings page.

2. Snowflake lineage metadata (column names and tags) for the Snowflake tables is stored in the metadata database.

3. A data owner creates a new data source (or adds a new column to a Snowflake table) that initiates a job that applies all tags for each column from its ancestor columns.

4. A data owner or governor adds a tag to a column in Immuta that has descendants, which initiates a job that propagates the tag to all descendants.

Snowflake access history view and Immuta lineage job

The Snowflake Account Usage ACCESS_HISTORY view contains column lineage information.

To appropriately propagate tags to descendant data sources, Immuta fetches Access History metadata to determine what column tags have been updated, stores this metadata in the Immuta metadata database, and then applies those tags to relevant descendant columns of tables registered in Immuta.

Consider the following example using the Customer, Customer 2, and Customer 3 tables that were all registered in Immuta as data sources.

• Customer: source table

• Customer 2: descendant of Customer

• Customer 3: descendant of Customer 2

If the Discovered.Electronic Mail Address tag is added to the Customer data source in Immuta, that tag will propagate through lineage to the Customer 2 and Customer 3 data sources.

Data source registration

After an application administrator has enabled Snowflake lineage tag propagation, data owners can register data in Immuta and have tags in Snowflake propagated from ancestor tables to descendant data sources. Whenever new tags are added to those tables in Immuta, those upstream tags will propagate to descendant data sources.

By default all tags are propagated, but these tags can be filtered on the app settings page or using the Immuta API.

Managing tags

Lineage tag propagation works with any tag added to the data dictionary. Tags can be manually added, synced from an external catalog, or discovered by SDD. Consider the following example using the Customer, Customer 2, and Customer 3 tables that were all registered in Immuta as data sources.

• Customer: source table

• Customer 2: descendant of Customer

• Customer 3: descendant of Customer 2

Immuta added the Discovered.Electronic Mail Address tag to the Customer data source, and that tag propagated through lineage to the Customer 2 and Customer 3 data sources.

Removing the tag from the Customer 2 table soft deletes it from the Customer 2 data source. When a tag is deleted, downstream lineage tags are removed, unless another parent data source still has that tag. The tag remains visible, but it will not be re-added if a future propagation event specifies the same tag again. Immuta prevents you from removing Snowflake object tags from data sources. You can only remove Immuta-managed tags. To remove Snowflake object tags from tables, you must remove them in Snowflake.

However the Discovered.Electronic Mail Address tag still applies to the Customer 3 data source because Customer still has the tag applied. The only way a tag will be removed from descendant data sources is if no other ancestor of the descendant still prescribes the tag.

If the Snowflake lineage tag propagation feature is disabled, tags will remain on Immuta data sources.

Sensitive data discovery

will still run on data sources and can be manually triggered. Tags applied through sensitive data discovery will propagate as tags added through lineage to descendant Immuta data sources.

Snowflake lineage audit

Immuta audit records include Snowflake lineage tag events when a tag is added or removed.

The example audit record below illustrates the SNOWFLAKE_TAGS.pii tag successfully propagating from the Customer table to Customer 2:

Limitations

• Without tableFilter set, Immuta will ingest lineage for every table on the Snowflake instance.

• Tag propagation based on lineage is not retroactive. For example, if you add a table, add tags to that table, and then run the lineage ingestion job, tags will not get propagated. However, if you add a table, run the lineage ingestion job, and then add tags to the table, the tags will get propagated.

• The lineage job needs to pull in lineage data before any tag is applied in Immuta. When Immuta gets new lineage information from Snowflake, Immuta does not update existing tags in Immuta.

This guide demonstrates how to download and package the Immuta Enterprise Helm chart and its dependencies for consumption on a separate network with no internet access.

Prerequisite

Checklist

Skopeo

• Skopeo is authenticated with Immuta's registry on the networked machine.

• Skopeo is authenticated with the private registry on the air-gapped machine.

Helm

• Helm is authenticated with Immuta's registry on the networked machine.

Download artifacts

This section demonstrates how to download the Helm chart and container images to your local machine. These artifacts will be packaged and transferred to the air-gapped environment later.

1. Create a directory named offline-kit.

2. Download the Helm chart into directory offline-kit.

3. Extract file DIGESTS.md from the Helm chart archive.

Transfer artifacts

This section demonstrates how to push the previously archived container images to a private registry that's accessible from within your air-gapped environment.

1. Transfer directory offline-kit (created in the previous section) onto a machine that's within your air-gapped environment.

2. Push each image to your private registry using .

Chart installation

Edit the immuta-values.yaml to reference the .

Snowflake table grants simplifies the management of privileges in Snowflake when using Immuta. Instead of having to manually grant users access to tables registered in Immuta, you allow Immuta to manage privileges on your Snowflake tables and views according to subscription policies. Then, users subscribed to a data source in Immuta can view and query the Snowflake table, while users who are not subscribed to the data source cannot view or query the Snowflake table.

Snowflake privileges

Enabling Snowflake table grants gives the following privileges to the Immuta Snowflake role:

• MANAGE GRANTS ON ACCOUNT allows the Immuta Snowflake role to grant and revoke SELECT privileges on Snowflake tables and views that have been added as data sources in Immuta.

• CREATE ROLE ON ACCOUNT allows for the creation of a Snowflake role for each user in Immuta, enabling fine-grained, attribute-based access controls to determine which tables are available to which individuals.

Table grants role

Since table privileges are granted to roles and not to users in Snowflake, Immuta's Snowflake table grants feature creates a new Snowflake role for each Immuta user. This design allows Immuta to manage table grants through fine-grained access controls that consider the individual attributes of users.

Each Snowflake user with an Immuta account will be granted a role that Immuta manages. The naming convention for this role is <IMMUTA>_USER_<username>, where

• <IMMUTA> is the prefix you specified when enabling the feature on the Immuta app settings page.

• <username> is the user's Immuta username.

Querying Snowflake tables managed by Immuta

Users are granted access to each Snowflake table or view automatically when they are subscribed to the corresponding data source in Immuta.

Users have two options for querying Snowflake tables that are managed by Immuta:

• that Immuta creates and manages. (For example, USE ROLE IMMUTA_USER_<username>. See the for details about the role and name conventions.) If the current active primary role is used to query tables, USAGE on a Snowflake warehouse must be granted to the Immuta-managed Snowflake role for each user.

• , which allows users to use the privileges from all roles that they have been granted, including IMMUTA_USER_<username>, in addition to the current active primary role. Users may also set a value for DEFAULT_SECONDARY_ROLES as an on a Snowflake user. To learn more about primary roles and secondary roles in Snowflake, see .

Applying GRANTs and REVOKEs at scale

Immuta uses an algorithm to determine the most optimal way to group users in a role hierarchy in order to optimize the number of GRANTs (or REVOKES) executed in Snowflake. This is done by determining the least amount of possible permutations of access across tables and users based on the policies in place; then, those become intermediate roles in the hierarchy that each user is added to, based on the intermediate roles they belong to.

As an example, take the below users and data sources they have access to. To do this naively by individually granting every user to the tables they have access to would result in 37 grants:

Conversely, using the Immuta algorithm, we can optimize the number of grants in the same scenario down to 29:

It’s important to consider a few things here:

1. If the permutations of access are small, there will be a huge optimization realized (very few intermediate roles). If every user has their own unique permutation of access, the optimization will be negligible (an intermediate role per user). It is most common that the number of permutations of access will be many multiples smaller than the actual user count, so there should be large optimizations. In other words, a much smaller number of intermediate roles and the number of total overall grants reduced, since the tables are granted to roles and roles to users.

2. This only happens once up front. After that, changes are incremental based on policy changes and user attribute changes (smaller updates), unless there’s a policy that makes a sweeping change across all users. The addition of new users who have access becomes much more straightforward also due to the fact above. User’s access will be granted via the intermediate role, and, therefore, a lot of the work is front loaded in the intermediate role creation.

Limitations

• are not supported when Snowflake table grants is enabled.

• If an Immuta tenant is connected to an external IAM and that external IAM has a username identical to another username in Immuta's built-in IAM, those users will have the same Snowflake role, leading both to see the same data.

• Sometimes the role generated can contain special characters such as @ because it's based on the user name configured from your identity manager. Because of this, it is recommended that any code references to the Immuta-generated role be enclosed with double quotes.

The how-to guides linked on this page illustrate how to integrate Databricks Unity Catalog with Immuta.

Requirements:

• Unity Catalog metastore created and attached to a Databricks workspace. Immuta supports configuring a single metastore for each configured integration, and that metastore may be attached to multiple Databricks workspaces.

• Unity Catalog enabled on your Databricks cluster or SQL warehouse. All SQL warehouses have Unity Catalog enabled if your workspace is attached to a Unity Catalog metastore.

Configure your Databricks Unity Catalog integration

These guides provide information on the recommended features to enable with Databricks Unity Catalog, or see the for a comprehensive guide on the benefits of these features and other recommendations.

1. with the following feature enabled: (enabled by default)

2. Select None as your .

3. .

4. .

Register metadata

These guides provide instructions for organizing your Databricks Unity Catalog data to align with your governance structure.

These guides provide step-by-step instructions for auditing and detecting your users' activity, or see the for a comprehensive guide on the benefits of these features and other recommendations.

1. or for your .

2. .

These guides provide instructions for discovering, classifying, and tagging your data.

1. .

2. to configure and validate SDD.

3. to discover entities of interest for your policy needs.

4. .

These guides provide instructions for configuring and securing your data with governance policies, or see the for a comprehensive guide on creating policies to fit your organization's use case.

1. .

2. .

3. Validate the policies. You do not have to validate every policy you create in Immuta; instead, examine a few to validate the behavior you expect to see.

4. Once all Immuta policies are in place, remove or alter old permissions and revoke access to the ungoverned tables.

The how-to guides linked on this page illustrate how to integrate Snowflake with Immuta.

Requirement: Snowflake Enterprise Edition

Configure your Snowflake integration

These guides provide information on the recommended features to enable with Snowflake, or see the Detect use case for a comprehensive guide on the benefits of these features and other recommendations.

1. with the following features enabled:

   • (enabled by default)

   • (enabled by default)

   • (enabled by default)

2. Select None as your .

3. .

4. .

Register metadata

These guides provide instructions for organizing your Snowflake data to align with your governance structure.

These guides provide step-by-step instructions for auditing and detecting your users' activity, or see the for a comprehensive guide on the benefits of these features and other recommendations.

1. or for your .

2. .

These guides provide step-by-step instructions for discovering, classifying, and tagging your data.

1. .

2. to configure and validate SDD.

3. to discover entities of interest for your policy needs.

4. .

These guides provide instructions for configuring and securing your data with governance policies, or see the for a comprehensive guide on creating policies to fit your organization's use case.

1. .

2. Validate the policy. You do not have to validate every policy you create in Immuta; instead, examine a few to validate the behavior you expect to see:

   1. Validate that the Immuta users impacted now have an Immuta role in Snowflake dedicated to them.

   2. Validate that when acting under the Immuta role those users have access to the table(s) in question.

This guide demonstrates how to upgrade an existing Immuta deployment installed with the older Immuta Helm chart (IHC) to v2024.2 LTS using the Immuta Enterprise Helm chart (IEHC).

Prerequisites

Create a PostgreSQL database

1. The PostgreSQL instance has been provisioned and is actively running.

2. The PostgreSQL instance's hostname/FQDN is .

3. The PostgreSQL instance is .

For additional information, consult the Deployment requirements.

Validate the Helm release

1. Fetch the metadata for the Helm release associated with Immuta.

2. Review the output from the previous step and verify the following:

   • The Immuta version (appVersion) is

Metadata database

The new IEHC no longer supports deploying a Metadata database (PostgreSQL) inside the Kubernetes cluster. Before transitioning to the new IEHC, it's first necessary to externalize the Metadata database.

Built-in

The following demonstrates how to take a database backup and import the data into each cloud provider's managed PostgreSQL service.

Create backup of old database

1. Get the metadata database pod name.

2. Spawn a shell inside the running metadata database pod.

3. Perform a database backup.

4. Type exit, and then press Enter to exit the shell prompt.

Setup new database

1. Create a pod named immuta-setup-db and spawn a shell.

2. Connect to the new PostgreSQL database as a superuser. Depending on the cloud provider, the default superuser name (postgres) might differ.

3. Create immuta, temporal, and temporal_visiblity

Restore backup to new database

1. Create a pod named immuta-restore-db and spawn a shell.

2. Copy file bometadata.dump from the host's working directory to pod immuta-restore-db.

3. Spawn a shell inside pod immuta-restore-db.

External

No additional work is required. The existing database can be reused with the new IEHC.

Helm values

1. Rename the existing immuta-values.yaml Helm values file used by the IHC.

2. Follow the for your Kubernetes distribution of choice.

Two cluster types can be configured with sparklyr: Single-User Clusters (recommended) and Multi-User Clusters (discouraged).

• Single-User Clusters: Credential Passthrough (required on Databricks) allows a single-user cluster to be created. This setting automatically configures the cluster to assume the role of the attached user when reading from storage. Because Immuta requires that raw data is readable by the cluster, the instance profile associated with the cluster should be used rather than a role assigned to the attached user.

• : Because Immuta cannot guarantee user isolation in a multi-user sparklyr cluster, it is not recommended to deploy a multi-user cluster. To force all users to act under the same set of attributes, groups, and purposes with respect to their data access and eliminate the risk of a data leak, all sparklyr multi-user clusters must be equalized either by convention (all users able to attach to the cluster have the same level of data access in Immuta) or by configuration (detailed below).

Single-User Cluster Configuration

1 - Enable sparklyr

In addition to the configuration for an Immuta cluster with R, add this environment variable to the Environment Variables section of the cluster:

This configuration makes changes to the iptables rules on the cluster to allow the sparklyr client to connect to the required ports on the JVM used by the sparklyr backend service.

2 - Set Up a sparklyr Connection in Databricks

1. Install and load libraries into a notebook. Databricks includes the stable version of sparklyr, so library(sparklyr) in an R notebook is sufficient, but you may opt to install the latest version of sparklyr from CRAN. Additionally, loading library(DBI) will allow you to execute SQL queries.

2. Set up a sparklyr connection:

3. Pass the connection object to execute queries:

3 - Configure a Single-User Cluster

Add the following items to the Spark Config section of the cluster:

The trustedFileSystems setting is required to allow Immuta’s wrapper FileSystem (used in conjunction with the ImmutaSecurityManager for data security purposes) to be used with credential passthrough. Additionally, the InstanceProfileCredentialsProvider must be configured to continue using the cluster’s instance profile for data access, rather than a role associated with the attached user.

Multi-User Cluster Configuration

The configurations in this section enable sparklyr, require project equalization, map sparklyr sessions to the correct Immuta user, and prevent users from accessing Immuta workspaces.

1. Add the following environment variables to the Environment Variables section of your cluster configuration:

2. Add the following items to the Spark Config section:

Limitations

Immuta’s integration with sparklyr does not currently support

• spark-submit jobs,

• UDFs, or

• Databricks Runtimes 5, 6, or 7.

Databricks non-admin users will only see sources to which they are subscribed in Immuta, and this can present problems if organizations have a data lake full of non-sensitive data and Immuta removes access to all of it. The Limited Enforcement Scope feature addresses this challenge by allowing Immuta users to access any tables that are not protected by Immuta (i.e., not registered as a data source or a table in a project workspace). Although this is similar to how privileged users in Databricks operate, non-privileged users cannot bypass Immuta controls.

This feature is composed of two configurations:

• Allowing non-Immuta reads: Immuta users with regular (unprivileged) Databricks roles may SELECT from tables that are not registered in Immuta.

• Allowing non-Immuta writes: Immuta users with regular (unprivileged) Databricks roles can run DDL commands and data-modifying commands against tables or spaces that are not registered in Immuta.

Additionally, Immuta supports auditing all queries run on a Databricks cluster, regardless of whether users touch Immuta-protected data or not. To configure Immuta to do so, navigate to the .

Enable Non-Immuta Reads

1. Enable non-Immuta Reads by setting this configuration in the Spark environment variables (recommended) or immuta_conf.xml (not recommended):

2. Opt to adjust the cache duration by changing the default value in the Spark environment variables (recommended) or immuta_conf.xml (not recommended). (Immuta caches whether a table has been exposed as an Immuta source to improve performance. The default caching duration is 1 hour.)

Enable Non-Immuta Writes

1. Enable non-Immuta Writes by setting this configuration in the Spark environment variables (recommended) or immuta_conf.xml (not recommended):

2. Opt to adjust the cache duration by changing the default value in the Spark environment variables (recommended) or immuta_conf.xml (not recommended). (Immuta caches whether a table has been exposed as an Immuta source to improve performance. The default caching duration is 1 hour.)

Enable Auditing of All Queries in Databricks

Enable support for auditing all queries run on a Databricks cluster (regardless of whether users touch Immuta-protected data or not) by setting this configuration in the Spark environment variables (recommended) or immuta_conf.xml (not recommended):

Default Configuration Values

The controls and default values associated with non-Immuta reads, non-Immuta writes, and audit functionality are outlined below.

In Immuta, a Databricks data source is considered ephemeral, meaning that the compute resources associated with that data source will not always be available.

Ephemeral data sources allow the use of ephemeral overrides, user-specific connection parameter overrides that are applied to Immuta metadata operations.

When a user runs a Spark job in Databricks, Immuta plugins automatically submit ephemeral overrides for that user to Immuta for all applicable data sources to use the current cluster as compute for all subsequent metadata operations for that user against the applicable data sources.

Example Query and Ephemeral Override Request

1. A user runs a query on cluster B.

2. The Immuta plugins on the cluster check if there is a source in the Metastore with a matching database, table name, and location for its underlying data. Note: If tables are dynamic or change over time, users can disable the comparison of the location of the underlying data by setting immuta.ephemeral.table.path.check.enabled to false; disabling allows users to avoid keeping the relevant data sources in Immuta up-to-date (which would require API calls and automation).

3. The Immuta plugins on the cluster detect that the user is subscribed to data sources 1, 2, and 3 and that data sources 1 and 3 are both present in the Metastore for cluster B, so the plugins submit ephemeral override requests for data sources 1 and 3 to override their connections with the HTTP path from cluster B.

If the user attempts to query data source 2 and they have not enabled JDBC sources, they will be presented with an error message telling them to do so:

com.immuta.spark.exceptions.ImmutaConfigurationException: This query plan will cause data to be pulled over JDBC. This spark context is not configured to allow this. To enable JDBC set immuta.enable.jdbc=true in the spark context hadoop configuration.

Immuta Operations that Use Ephemeral Overrides

Ephemeral overrides are enabled by default because Immuta must be aware of a cluster that is running to serve metadata queries. The operations that use the ephemeral overrides include

• Visibility checks on the data source for a particular user. These checks assess how to apply row-level policies for specific users.

• Stats collection triggered by a specific user.

• Validating a custom WHERE clause policy against a data source. When owners or governors create custom WHERE clause policies, Immuta uses compute resources to validate the SQL in the policy. In this case, the ephemeral overrides for the user writing the policy are used to contact a cluster for SQL validation.

However, ephemeral overrides can be problematic in environments that have a dedicated cluster to handle maintenance activities, since ephemeral overrides can cause these operations to execute on a different cluster than the dedicated one.

Configure Overrides in Immuta-Enabled Clusters

To reduce the risk that a user has overrides set to a cluster (or multiple clusters) that aren't currently up,

• direct all clusters' HTTP paths for overrides to a cluster dedicated for metadata queries or

• disable overrides completely.

Disable Ephemeral Overrides

To disable ephemeral overrides, set immuta.ephemeral.host.override in spark-defaults.conf to false.

This guide highlights best practices when deploying Immuta in a production environment.

Databricks metastore magic allows you to migrate your data from the Databricks legacy Hive metastore to the Unity Catalog metastore while protecting data and maintaining your current processes in a single Immuta instance.

Databricks metastore magic is for customers who intend to use the , but they would like to protect tables in the Hive metastore.

Requirement

Unity Catalog support is enabled in Immuta.

This guide illustrates how to run R and Scala spark-submit jobs on Databricks, including prerequisites and caveats.

This guide demonstrates how to configure TLS termination for an .

This page provides an overview of the Databricks Spark integration. For installation instructions, see the .

Overview

Databricks Spark is a plugin integration with Immuta. This integration allows you to protect access to tables and manage row-, column-, and cell-level controls without enabling table ACLs or credential passthrough. Policies are applied to the plan that Spark builds for a user's query and enforced live on-cluster.

This page describes the Redshift integration, configuration options, and features. For a tutorial to enable this integration, see the .

Feature Availability