The following guides offer practical guidance for handling common challenges and configurations.

Ingress configuration

Configure Ingress to complete your installation and access your Immuta application.

Configure TLS termination for an Ingress resource.

Verify artifacts hosted on the ocir.immuta.com OCI registry.

Follow these best practices when deploying Immuta in your production environment.

Update the credentials referenced in the Immuta Enterprise Helm chart.

Configure an external key-value cache (such as Redis or Memcached) with the Immuta Enterprise Helm chart.

Enable this legacy service for your deployment if you are using any of the .

Configure pulling images from a private registry.

Tips when installing Immuta without internet access.

Immuta gives everyone fast, governed access to data with the built-in controls, collaboration workflows, automated provisioning, and continuous monitoring you need to keep risk low and compliance high

Configure Immuta

Explore Immuta

Introduced in 2024.2, the Immuta Enterprise Helm chart (IEHC) is an entirely new Helm chart used to deploy Immuta. Unlike the previous Immuta Helm chart (IHC), the IEHC shares the same version as the Immuta product. Each version of the chart supports a singular version of Immuta. Upgrading the Immuta version now entails upgrading the underlying Helm chart. Failure to do so will lead to an unsupported configuration.

Upgrade guides

• : If you're upgrading from 2024.1.x or older, you must first migrate to the new Helm chart, as these versions were all installed using the legacy IHC. This migration process will include upgrading Immuta.

• : Upgrade from v2024.2 LTS using the Immuta Enterprise Helm chart.

In this integration, Immuta generates policy-enforced views in a schema in your configured Azure Synapse Analytics Dedicated SQL pool for tables registered as Immuta data sources.

Getting started

This guide outlines how to integrate Azure Synapse Analytics with Immuta.

How-to guide

: Configure the integration in Immuta.

Reference guides

• : This guide describes the design and components of the integration.

• : This guide describes the prerequisites, supported features, and limitations of the integration.

This section illustrates how to install Immuta on Kubernetes using the Immuta Enterprise Helm chart.

Requirements

This reference guide provides an overview of the Immuta Enterprise Helm chart version requirements and infrastructure recommendations.

The guides in this section illustrate how to install and deploy Immuta in your Kubernetes environment.

The guides in this section illustrate how to upgrade Immuta.

The guides in this section illustrate how to configure your Immuta Enterprise Helm chart for various scenarios, including optimizing your deployment for production environments.

This guide provides links to additional resources for disaster recovery strategies.

This page provides troubleshooting guidance and outlines frequently asked questions for the Immuta installation.

This page introduces the core concepts and terminology essential for understanding the installation material.

This guide demonstrates how to configure a private container registry with the Immuta Enterprise Helm chart (IEHC).

The guides in this section illustrate how to install and deploy Immuta in your Kubernetes environment.

Prerequisites

This guide demonstrates how to verify signed artifacts (i.e., container images, Helm charts) hosted on ocir.immuta.com using from .

If you're using AWS OpenSearch in your Immuta installation, use this how-to to set up the proper permissions needed for username and password authentication.

1. In the AWS console, and create a master user. This user will set up the permissions for the audit user.

2. In the OpenSearch console, . This user will be the audit user. You will enter the username and password for this user when installing Immuta.

3. for the audit user.

The following conventions are used throughout the installation material.

Angle brackets ( < and > )

Phrases wrapped in angle brackets (i.e., <, >) are placeholders used to indicate values that must be substituted with user-provided values. Placeholders are typically written in either kebab case, or snake case; the following placeholders are equivalent:

• <the-quick-brown-fox>

• <the_quick_brown_fox>

Example

Input

Output

All application state is stored in the PostgreSQL metadata database; therefore, recovering from a disaster event only entails restoring the aforementioned PostgreSQL database. Consult each cloud provider's point-in-time recovery (PITR) documentation for guidance:

• Amazon RDS for PostgreSQL

For more details about point-in-time recovery, see the .

This guide demonstrates how to configure an external key-value cache (such as Redis or Memcached) with the Immuta Enterprise Helm chart (IEHC).

This guide demonstrates how to update credentials referenced in the Immuta Enterprise Helm chart (IEHC).

Frequently asked questions

How can I ensure the fully qualified domain name (FQDN) is resolvable from within the Kubernetes cluster?

This guide demonstrates how to upgrade an existing Immuta deployment installed with the older Immuta Helm chart (IHC) to v2024.2 LTS using the Immuta Enterprise Helm chart (IEHC).

Prerequisites

Create a PostgreSQL database

1. The PostgreSQL instance has been provisioned and is actively running.

2. The PostgreSQL instance's hostname/FQDN is .

3. The PostgreSQL instance is .

For additional information, consult the Deployment requirements.

Validate the Helm release

1. Fetch the metadata for the Helm release associated with Immuta.

2. Review the output from the previous step and verify the following:

   • The Immuta version (appVersion) is

Metadata database

The new IEHC no longer supports deploying a Metadata database (PostgreSQL) inside the Kubernetes cluster. Before transitioning to the new IEHC, it's first necessary to externalize the Metadata database.

Built-in

The following demonstrates how to take a database backup and import the data into each cloud provider's managed PostgreSQL service.

Create backup of old database

1. Get the metadata database pod name.

2. Spawn a shell inside the running metadata database pod.

3. Perform a database backup.

4. Type exit, and then press Enter to exit the shell prompt.

Setup new database

1. Create a pod named immuta-setup-db and spawn a shell.

2. Connect to the new PostgreSQL database as a superuser. Depending on the cloud provider, the default superuser name (postgres) might differ.

3. Create immuta, temporal, and temporal_visiblity

Restore backup to new database

1. Create a pod named immuta-restore-db and spawn a shell.

2. Copy file bometadata.dump from the host's working directory to pod immuta-restore-db.

3. Spawn a shell inside pod immuta-restore-db.

External

No additional work is required. The existing database can be reused with the new IEHC.

Helm values

1. Rename the existing immuta-values.yaml Helm values file used by the IHC.

2. Follow the for your Kubernetes distribution of choice.

This guide demonstrates how to configure TLS termination for an Ingress resource.

Prerequisite

The must be completed before proceeding.

1. Edit immuta-values.yaml to include the following Helm values.

2. from a given public/private PEM formatted key pair.

3. Perform a to apply the changes made to immuta-values.yaml.

Refer to the for further assistance.

1. Edit immuta-values.yaml to include the following Helm values.

2. Perform a to apply the changes made to immuta-values.yaml.

Refer to the for further assistance.

1. Edit immuta-values.yaml to include the following Helm values.

2. Perform a to apply the changes made to immuta-values.yaml.

Refer to the for further assistance.

1. Edit immuta-values.yaml to include the following Helm values.

2. Perform a to apply the changes made to immuta-values.yaml.

Refer to the for further assistance.

1. Edit immuta-values.yaml to include the following Helm values.

2. from a given public/private PEM formatted key pair.

3. Perform a to apply the changes made to immuta-values.yaml.

Refer to the for further assistance.

This guide demonstrates how to configure Ingress. Ingress can be configured in numerous ways. Configurations for the most popular controllers are outlined below.

The Immuta web service listens on the following ports:

1. Edit the immuta-values.yaml file to include the following Helm values.

2. Perform a to apply the changes made to immuta-values.yaml.

Refer to the for further assistance.

1. Edit immuta-values.yaml to include the following Helm values.

2. Create a file named frontendconfig.yaml with the following content.

3. Apply the FrontendConfig CRD.

Refer to the for further assistance.

1. Edit immuta-values.yaml to include the following Helm values.

2. Perform a to apply the changes made to immuta-values.yaml.

Refer to the for further assistance.

1. Edit immuta-values.yaml to include the following Helm values.

2. Perform a to apply the changes made to immuta-values.yaml.

Refer to the for further assistance.

1. Edit immuta-values.yaml to include the following Helm values.

2. Create a file named middleware.yaml with the following content.

3. Apply the Middleware CRD.

Refer to the for further assistance.

1. Edit immuta-values.yaml to include the following Helm values. Because the Ingress resource will be managed by the OpenShift route you create and not the Immuta Enterprise Helm chart, ingress is set to false below.

2. Get the service name for Secure.

3. Create a file named route.yaml with the following content. Update all with your own values.

Refer to the for further assistance.

Immuta integrates with your data platforms and external catalogs so you can register your data and effectively manage access controls on that data.

This section includes concept, reference, and how-to guides for configuring your data platform integration, registering data sources, and connecting your external catalog so that you can discover, monitor, and protect sensitive data.

Integrations overview

This reference guide outlines the features, policies, and audit capabilities supported by each integration.

Integrations

The guides in these sections include information about how to connect your data platform to Immuta.

This reference guide outlines the actions and features that trigger Immuta queries in your remote platform that may incur cost.

Immuta integrates with your data platforms so you can register your data and effectively manage access controls on that data. This section includes concept, reference, and how-to guides for registering and managing data sources and your connections.

The Databricks Spark integration is one of two integrations Immuta offers for Databricks.

In this integration, Immuta installs an Immuta-maintained Spark plugin on your Databricks cluster. When a user queries data that has been registered in Immuta as a data source, the plugin injects policy logic into the plan Spark builds so that the results returned to the user only include data that specific user should see.

The reference guides in this section are written for Databricks administrators who are responsible for setting up the integration, securing Databricks clusters, and setting up users:

• Installation and compliance: This guide includes information about what Immuta creates in your Databricks environment and securing your Databricks clusters.

• : Consult this guide for information about customizing the Databricks Spark integration settings.

• : Consult this guide for information about connecting data users and setting up user impersonation.

• : This guide provides a list of Spark environment variables used to configure the integration.

• : This guide describes ephemeral overrides and how to configure them to reduce the risk that a user has overrides set to a cluster (or multiple clusters) that aren't currently up.

In this integration, Immuta generates policy-enforced views in your configured Redshift schema for tables registered as Immuta data sources.

Getting started

This guide outlines how to integrate Redshift with Immuta.

How-to guides

• : Configure the integration in Immuta.

• : Configure Redshift Spectrum in Immuta.

Reference guides

• : This guide describes the design and components of the integration.

• : This guide describes the prerequisites, supported features, and limitations of the integration.

This guide demonstrates how to upgrade an existing 2024.2+ Immuta deployment installed with the Immuta Enterprise Helm chart (IEHC) to the latest Immuta release.

Requirements

Temporal

The query engine is no longer installed by default. This guide demonstrates how to enable the query engine using the Immuta Enterprise Helm chart (IEHC).

If you are using any of the , you must enable the query engine.

If you're using AWS OpenSearch in your Immuta installation, use this how-to to set up the proper access and permissions needed for AWS role authentication.

Requirements

• An OpenSearch domain

This guide demonstrates how to download and package the Immuta Enterprise Helm chart and its dependencies for consumption on a separate network with no internet access.

Prerequisite

The how-to guides linked on this page illustrate how to integrate Databricks Spark with Immuta.

Requirements

• If Databricks Unity Catalog is enabled in a Databricks workspace, you must use an when you set up the Databricks Spark integration to create an Immuta-enabled cluster.

• If Databricks Unity Catalog is not enabled in your Databricks workspace, you must disable Unity Catalog in your Immuta tenant before proceeding with your configuration of Databricks Spark:

This page outlines how to enable access to DBFS in Databricks for non-sensitive data. Databricks administrators should place the desired configuration in the Spark environment variables.

DBFS FUSE mount

This Databricks feature mounts DBFS to the local cluster filesystem at /dbfs. Although disabled when using process isolation, this feature can safely be enabled if raw, unfiltered data is not stored in DBFS and all users on the cluster are authorized to see each other’s files. When enabled, the entirety of DBFS essentially becomes a scratch path where users can read and write files in /dfbs/path/to/my/file as though they were local files.

When the Databricks Spark plugin is running on a Databricks cluster, all Databricks users running jobs or queries are either a privileged user or a non-privileged user:

• Privileged users: Privileged users can effectively read from and write to any table or view in the cluster Metastore, or any file path accessible by the cluster, without restriction. Privileged users are either or users specified in . Any user writing queries or jobs impersonating another user is a non-privileged user, even if they are impersonating a privileged user.\

  Privileged users have effective authority to read from and write to any securable in the cluster metastore or file path, because in almost all cases Databricks clusters running with the Immuta Spark plug-in installed have disabled . However, if Hive metastore table access control is enabled on the cluster, privileged users will have the authority granted to them that is specified by table access control.

The how-to guides linked on this page illustrate how to integrate Azure Synapse Analytics with Immuta. See the for information about the Azure Synapse Analytics integration.

Requirement: A running Dedicated SQL pool

This integration allows you to manage and access data in your Databricks account across all of your workspaces. With Immuta’s Databricks Unity Catalog integration, you can write your policies in Immuta and have them enforced automatically by Databricks across data in your Unity Catalog metastore.

This getting started guide outlines how to integrate Databricks Unity Catalog with Immuta.

Immuta offers two integrations for Databricks:

• Databricks Unity Catalog integration: This integration supports working with database objects registered in Unity Catalog.

• Databricks Spark integration: This integration supports working with database objects registered in the legacy Hive metastore.

Which integration should you use?

To determine which integration you should use, evaluate the following elements:

• Cluster runtime

  • Databricks Runtime 11.3 and newer: See the list below to determine which integration is supported for your data's location.

• Location of data: Where is your data?

Metastore magic

Databricks metastore magic allows you to migrate your data from the Databricks legacy Hive metastore to the Unity Catalog metastore while protecting data and maintaining your current processes in a single Immuta instance.

Databricks metastore magic is for organizations who intend to use the , but must still protect tables in the Hive metastore until they can migrate all of their data to Unity Catalog.

Requirement

Unity Catalog support is enabled in Immuta.

Databricks metastores and Immuta policy enforcement

Databricks has two built-in metastores that contain metadata about your tables, views, and storage credentials:

• Legacy Hive metastore: Created at the workspace level. This metastore contains metadata of the registered securables in that workspace available to query.

• Unity Catalog metastore: Created at the account level and is attached to one or more Databricks workspaces. This metastore contains metadata of the registered securables available to query. All clusters on that workspace use the configured metastore and all workspaces that are configured to use a single metastore share those securables.

Databricks allows you to use the legacy Hive metastore and the Unity Catalog metastore simultaneously. However, Unity Catalog does not support controls on the Hive metastore, so you must attach a Unity Catalog metastore to your workspace and move existing databases and tables to the attached Unity Catalog metastore to use the governance capabilities of Unity Catalog.

Immuta's Databricks Spark integration and Unity Catalog integration enforce access controls on the Hive and Unity Catalog metastores, respectively. However, because these metastores have two distinct security models, users were discouraged from using both in a single Immuta instance before metastore magic; the Databricks Spark integration and Unity Catalog integration were unaware of each other, so using both concurrently caused undefined behavior.

Databricks metastore magic solution

Metastore magic reconciles the distinct security models of the legacy Hive metastore and the Unity Catalog metastore, allowing you to use multiple metastores (specifically, the Hive metastore or alongside Unity Catalog metastores) within a Databricks workspace and single Immuta instance and keep policies enforced on all your tables as you migrate them. The diagram below shows Immuta enforcing policies on registered tables across workspaces.

In clusters A and D, Immuta enforces policies on data sources in each workspace's Hive metastore and in the Unity Catalog metastore shared by those workspaces. In clusters B, C, and E (which don't have Unity Catalog enabled in Databricks), Immuta enforces policies on data sources in the Hive metastores for each workspace.

Enforce policies as you migrate

With metastore magic, the Databricks Spark integration enforces policies only on data in the Hive metastore, while the Unity Catalog integration enforces policies on tables in the Unity Catalog metastore.

To enforce plugin-based policies on Hive metastore tables and Unity Catalog native controls on Unity Catalog metastore tables, enable the and the Databricks Unity Catalog integration. Note that some Immuta policies are not supported in the Databricks Unity Catalog integration. See the for details.

Enforcing policies on Databricks SQL

Databricks SQL cannot run the Databricks Spark plugin to protect tables, so Hive metastore data sources will not be policy enforced in Databricks SQL.

To enforce policies on data sources in Databricks SQL, use to manually lock down Hive metastore data sources and the Databricks Unity Catalog integration to protect tables in the Unity Catalog metastore. Table access control is enabled by default on SQL warehouses, and any Databricks cluster without the Immuta plugin must have table access control enabled.

In the context of the Databricks Spark integration, Immuta uses the term ephemeral to describe data sources where the associated compute resources can vary over time. This means that the compute bound to these data sources is not fixed and can change. All Databricks data sources in Immuta are ephemeral.

Ephemeral overrides are specific to each data source and user. They effectively bind cluster compute resources to a data source for a given user. Immuta uses these overrides to determine which cluster compute to use when connecting to Databricks for various maintenance operations.

The operations that use the ephemeral overrides include

• Visibility checks on the data source for a particular user. These checks assess how to apply row-level policies for specific users.

• Stats collection triggered by a specific user.

• Validating a custom WHERE clause policy against a data source. When owners or governors create custom WHERE clause policies, Immuta uses compute resources to validate the SQL in the policy. In this case, the ephemeral overrides for the user writing the policy are used to contact a cluster for SQL validation.

• High cardinality column detection. Certain advanced policy types (e.g., minimization) in Immuta require a high cardinality column, and that column is computed on data source creation. It can be recomputed on demand and, if so, will use the ephemeral overrides for the user requesting computation.

Triggering an ephemeral override request

An ephemeral override request can be triggered when a user queries the securable corresponding to a data source in a Databricks cluster with the Spark plug-in configured. The actual triggering of this request depends on the .

Ephemeral overrides can also be set for a data source in the Immuta UI by navigating to the data source page, clicking on the data source actions button, and selecting Ephemeral overrides from the dropdown menu.

Ephemeral override requests made from a cluster for data sources and users where ephemeral overrides were set in the UI will not be successful.

If ephemeral overrides are never set (either through the user interface or the cluster configuration), the system will continue to use the connection details directly associated with the data source, which are set during .

Configuring overrides in Immuta-enabled clusters

Ephemeral overrides can be problematic in environments that have a dedicated cluster to handle maintenance activities, since ephemeral overrides can cause these operations to execute on a different cluster than the dedicated one.

To reduce the risk that a user has overrides set to a cluster (or multiple clusters) that aren't currently up, complete one of the following actions:

• Direct all clusters' HTTP paths for overrides to a cluster dedicated for metadata queries using the .

• Disable ephemeral overrides completely by setting the to false.

This integration enforces policies on Databricks securables registered in the legacy Hive metastore. Once these securables are registered as Immuta data sources, users can query policy-enforced data on Databricks clusters.

The guides in this section outline how to integrate Databricks Spark with Immuta.

Getting started

This getting started guide outlines how to integrate Databricks with Immuta.

How-to guides

• : Manually update your cluster to reflect changes in the Immuta init script or cluster policies.

• : Register a Databricks library with Immuta as a trusted library to avoid Immuta Security Manager errors when using third-party libraries.

• : Raise the caching on-cluster and lower the cache timeouts for the Immuta web service to allow use of project UDFs in Spark jobs.

Reference guides

• : This guide describes the design and components of the integration.

• : This guide provides an overview of the Immuta features that provide security for your users and Databricks clusters and that allow you to prove compliance and monitor for anomalies.

• : This guide provides an overview of registering Databricks securables and protecting them with Immuta policies.

• : This guide provides an overview of how Databricks users access data registered in Immuta.

In this integration, Immuta policies are translated into Starburst rules and permissions and applied directly to tables within users’ existing catalogs.

Getting started

This guide outlines how to integrate Starburst with Immuta.

How-to guides

• : Configure the integration in Immuta.

• : Configure how read and write access subscription policies translate to Starburst (Trino) privileges and apply to Starburst (Trino) data sources.

Reference guide

: This guide describes the design and components of the integration.

To migrate from the private preview version of table grants (available before September 2022) to the GA version, complete the steps below.

1. Navigate to the App Settings page.

2. Click Integration Settings in the left panel, and scroll to the Global Integrations Settings section.

3. Uncheck the Snowflake Table Grants checkbox to disable the feature.

4. Click Save. Wait for about 1 minute per 1000 users. This gives time for Immuta to drop all the previously created user roles.

5. Use the to re-enable the feature.

Immuta comprises three core services: Secure, Discover, and Detect. These services rely on PostgreSQL and Elasticsearch to store their states, a caching layer, and Temporal for job execution. The illustration below shows the relationships among these services.

The Immuta Enterprise Helm chart (IEHC) does not include the deployment of PostgreSQL or Elasticsearch, so you must deploy them separately.

Although Immuta recommends using Elasticsearch because it supports all audit, you can deploy Immuta without Elasticsearch. The table below outlines the Immuta features supported with and without Elasticsearch and the dependencies you must deploy and manage yourself.

This guide highlights best practices when deploying Immuta in a production environment.

This page provides a tutorial for enabling the Azure Synapse Analytics integration on the Immuta app settings page. To configure this integration via the Immuta API, see the .

For an overview of the integration, see the documentation.

Requirement

A running Dedicated SQL pool is required.

To or a Snowflake integration, you have two options:

• Automatic: Grant Immuta one-time use of credentials with the following privileges to automatically edit or remove the integration:

  • CREATE DATABASE ON ACCOUNT WITH GRANT OPTION

The how-to guides linked on this page illustrate how to integrate Snowflake with Immuta. See the for information about the Snowflake integration.

Requirements

• Snowflake enterprise edition

• Access to a Snowflake account that can create a Snowflake user

The Snowflake low row access policy mode improves query performance in Immuta's Snowflake integration by decreasing the number of Immuta creates and by using table grants to manage user access.

Immuta manages access to Snowflake tables by administering and on those tables, allowing users to query them directly in Snowflake while policies are enforced.

Without Snowflake low row access policy mode enabled, row access policies are created and administered by Immuta in the following scenarios:

• are disabled and a subscription policy that does not automatically subscribe everyone to the data source is applied. Immuta administers Snowflake row access policies to filter out all the rows to restrict access to the entire table when the user doesn't have privileges to query it. However, if table grants are disabled and a subscription policy is applied that grants everyone access to the data source automatically, Immuta does not create a row access policy in Snowflake. See the for details about these policy types.

The warehouse you select when configuring the Snowflake integration uses compute resources to set up the integration, register data sources, orchestrate policies, and run jobs like identification. Snowflake credit charges are based on the size of and amount of time the warehouse is active, not the number of queries run.

This document prescribes how and when to adjust the size and scale of clusters for your warehouse to manage workloads so that you can use Snowflake compute resources the most cost effectively.

In general, increase the size of and number of clusters for the warehouse to handle heavy workloads and multiple queries. Workloads are typically lighter after data sources are onboarded and policies are established in Immuta, so compute resources can be reduced after those workloads complete.

Integration and data source registration warehouse use

This page outlines the configuration for setting up project UDFs, which allow users to set their current project in Immuta through Spark. For details about the specific functions available and how to use them, see the .

1. Lower the web service cache timeout in Immuta:

Immuta offers several features to provide security for your users and Databricks clusters and to prove compliance and monitor for anomalies.

Authentication

Configuring the integration and registering data

Immuta supports the following authentication methods to configure the Databricks Spark integration and register data sources:

• OAuth machine-to-machine (M2M): Immuta uses the to integrate with , which allows Immuta to authenticate with Databricks using a client secret. Once Databricks verifies the Immuta service principal’s identity using the client secret, Immuta is granted a temporary OAuth token to perform token-based authentication in subsequent requests. When that token expires (after one hour), Immuta requests a new temporary token. See the for more details.\

• Personal access token (PAT): This token gives Immuta temporary permission to push the cluster policies to the configured Databricks workspace and overwrite any cluster policy templates previously applied to the workspace when configuring the integration or to register securables as Immuta data sources.

User authentication

The built-in Immuta IAM can be used as a complete solution for authentication and fine-grained user entitlement. However, you can connect your existing identity management provider to Immuta to use that system for authentication and fine-grained user entitlement instead.

Each of the supported identity providers includes a specific set of configuration options that enable Immuta to communicate with the IAM system and map the users, permissions, groups, and attributes into Immuta.

See the guide for a list of supported providers and details.

See the for details and instructions on mapping Databricks user accounts to Immuta.

Cluster security

Data processing and encryption

See the for more information about transmission of policy decision data, encryption of data in transit and at rest, and encryption key management.

Protecting the Immuta configuration

Non-administrator users on an Immuta-enabled Databricks cluster must not have access to view or modify Immuta configuration, as this poses a security loophole around Immuta policy enforcement. allow you to securely apply environment variables to Immuta-enabled clusters.

Databricks secrets can be used in the environment variables configuration section for a cluster by referencing the secret path instead of the actual value of the environment variable.

See the for details and instructions on using Databricks secrets.

Scala cluster security

There are limitations to isolation among users in Scala jobs on a Databricks cluster. When data is broadcast, cached (spilled to disk), or otherwise saved to SPARK_LOCAL_DIR, it's impossible to distinguish between which user’s data is composed in each file/block. To address this vulnerability, Immuta suggests that you

• limit Scala clusters to Scala jobs only and

• require equalized projects, which will force all users to act under the same set of attributes, groups, and purposes with respect to their data access. This requirement guarantees that data being dropped into SPARK_LOCAL_DIR will have policies enforced and that those policies will be homogeneous for all users on the cluster. Since each user will have access to the same data, if they attempt to manually access other users' cached/spilled data, they will only see what they have access to via equalized permissions on the cluster. If project equalization is not turned on, users could dig through that directory and find data from another user with heightened access, which would result in a data leak.

See the for more details and configuration instructions.

Auditing and compliance

Immuta provides auditing features and governance reports so that data owners and governors can monitor users' access to data and detect anomalies in behavior.

You can view the information in these audit logs on or export the full audit logs to S3 and ADLS for long-term backup and processing with log data processors and tools. This capability fosters convenient integrations with log monitoring services and data pipelines.

See the for details about these capabilities and how they work with the Databricks Spark integration.

Databricks query audit

Immuta captures the code or query that triggers the Spark plan in Databricks, making audit records more useful in assessing what users are doing.

To audit what triggers the Spark plan, Immuta hooks into Databricks where notebook cells and JDBC queries execute and saves the cell or query text. Then, Immuta pulls this information into the audits of the resulting Spark jobs.

Immuta will audit queries that come from interactive notebooks, notebook jobs, and JDBC connections, but will not audit . Furthermore, Immuta only audits Spark jobs that are associated with Immuta tables. Consequently, Immuta will not audit a query in a notebook cell that does not trigger a Spark job, unless

See the page for examples of saved queries and the resulting audit records. To exclude query text from audit events, see the page.

Auditing all queries

Immuta supports auditing all queries run on a Databricks cluster, regardless of whether users touch Immuta-protected data or not.

See the for details and instructions.

Auditing queries run while impersonating another user

When a query is run by a user impersonating another user, the extra.impersonationUser field in the audit log payload is populated with the Databricks username of the user impersonating another user. The userId field will return the Immuta username of the user being impersonated:

See the for details about user impersonation.

Governance reports

Immuta governance reports allow users with the GOVERNANCE Immuta permission to use a natural language builder to instantly create reports that delineate user activity across Immuta. These reports can be based on various entity types, including users, groups, projects, data sources, purposes, policy types, or connection types.

See the page for a list of report types and guidance.

Requirements

• APPLICATION_ADMIN Immuta permission

• The Snowflake user registering the connection and running the script must have the following privileges:

  • CREATE DATABASE ON ACCOUNT WITH GRANT OPTION

Prerequisites

No Snowflake integration configured in Immuta. If your Snowflake integration is already configured on the app settings page, follow the .

Set up the Immuta system account

Complete the following actions in Snowflake:

1. . Immuta will use this system account continuously to orchestrate Snowflake policies and maintain state between Immuta and Snowflake.

2. with a minimum of the following privileges:

   • USAGE on all databases and schemas with registered data sources.

Register a connection

To register a Snowflake connection, follow the instructions below.

1. Click Data and select the Connections tab in the navigation menu.

2. Click the + Add Connection button.

3. Select the Snowflake data platform tile.

4. Enter the connection information:

Once a Databricks securable is registered in Immut as a data source and you are subscribed to that data source, you must access that data through SQL:

df = spark.sql("select * from immuta.table")

import org.apache.spark.sql.SparkSession

val spark = SparkSession
  .builder()
  .appName("Spark SQL basic example")
  .config("spark.some.config.option", "some-value")
  .getOrCreate()
val sqlDF = spark.sql("SELECT * FROM immuta.table")

%sql
select * from immuta.table

library(SparkR)
df <- SparkR::sql("SELECT * from immuta.table")

See the sections below for more guidance on accessing data using Delta Lake, direct file reads in Spark for file paths, and user impersonation.

Delta Lake

When using Delta Lake, the API does not go through the normal Spark execution path. This means that Immuta's Spark extensions do not provide protection for the API. To solve this issue and ensure that Immuta has control over what a user can access, the Delta Lake API is blocked.

Spark SQL can be used instead to give the same functionality with all of Immuta's data protections. See the for a list of corresponding Spark SQL calls to use.

Spark direct file reads

In addition to supporting direct file reads through workspace and scratch paths, Immuta allows direct file reads in Spark for file paths. As a result, users who prefer to interact with their data using file paths or who have existing workflows revolving around file paths can continue to use these workflows without rewriting those queries for Immuta.

When reading from a path in Spark, the Immuta Databricks Spark plugin queries the Immuta Web Service to find Databricks data sources for the current user that are backed by data from the specified path. If found, the query plan maps to the Immuta data source and follows existing code paths for policy enforcement.

Users can read data from individual parquet files in a sub-directory and partitioned data from a sub-directory (or by using a where predicate). Expand the blocks below to view examples of reading data using these methods.

Limitations

• Direct file reads for Immuta data sources only apply to data sources created from tables, not data sources created from views or queries.

• If more than one data source has been created for a path, Immuta will use the first valid data source it finds. It is therefore not recommended to use this integration when more than one data source has been created for a path.

• In Databricks, multiple input paths are supported as long as they belong to the same data source.

• CSV-backed tables are not currently supported.

User impersonation

User impersonation allows Databricks users to query data as another Immuta user. To impersonate another user, see the .

This page illustrates how to configure the on the Immuta app settings page. To configure this integration via the Immuta API, see the .

For instructions on configuring Redshift Spectrum, see the guide.

Requirements

• A Redshift cluster with an RA3 node is required for the multi-database integration. You must use a Redshift RA3 instance type because Immuta requires cross-database views, which are only supported in Redshift RA3 instance types. For other instance types, you may configure a single-database integration using one of the