Immuta integrates with your data platforms so you can register your data and effectively manage access controls on that data.

This section includes concept, reference, and how-to guides for configuring your data platform integration and registering data sources so that you can discover, monitor, and protect sensitive data.

This reference guide outlines the features, policies, and audit capabilities of each data platform Immuta supports.

This page includes how-to and reference content for Amazon S3 and how it integrates with Immuta.

This section includes how-to and reference guides for AWS Lake Formation and how it connects with Immuta.

This section includes how-to and reference guides for Azure Synapse Analytics and how it integrates with Immuta.

This page includes how-to and reference content for Google BigQuery and how it integrates with Immuta.

This section includes how-to and reference guides for Redshift and how it integrates with Immuta.

This section includes how-to and reference guides for Snowflake and how it integrates with Immuta.

This section includes how-to and reference guides for Starburst (Trino) and how it integrates with Immuta.

This reference guide outlines the actions and features that trigger Immuta queries in your remote platform that may incur cost.

Immuta integrates with your data platforms so you can register your data and effectively manage access controls on that data. This section includes concept, reference, and how-to guides for registering and managing data sources.

Immuta does not require users to learn a new API or language to access protected data. Instead, Immuta integrates with existing tools and ongoing work while remaining invisible to downstream consumers.

The following data platforms integrate with Immuta:

• Databricks:

Feature support

The table below outlines the features supported by each of Immuta's integrations.

Data policy support

• On Databricks data sources, joins will not be allowed on data protected with replace with NULL or constant policies.

• Databricks Unity Catalog ARRAY, MAP, or STRUCT type columns only support masking with NULL.

Audit support for platform queries

The table below outlines what information is included in the query audit logs for each integration where query audit is supported.

Legend:

In the Lake Formation connection, Immuta orchestrates Lake Formation access controls on data registered in the Glue Data Catalog. Then, Immuta users who have been granted access to the Glue Data Catalog table or view can query it using one of these analytic engines:

• Amazon Athena

• Amazon EMR Spark

• Amazon Redshift Spectrum

This getting started guide outlines how to connect AWS Lake Formation with Immuta.

How-to guide

Reference guides

Getting started

Requirements

• No location is registered in your S3 Access Grants instance before configuring the integration in Immuta

Permissions

• APPLICATION_ADMIN Immuta permission to configure the integration

• CREATE_S3_DATASOURCE Immuta permission to register S3 prefixes

• The AWS account credentials or optional AWS IAM role you provide Immuta to configure the integration must

  • • accessgrantslocation resource:

      • s3:CreateAccessGrant

      • s3:DeleteAccessGrantsLocation

      • s3:GetAccessGrantsLocation

      • s3:UpdateAccessGrantsLocation

    • accessgrantsinstance resource:

      • s3:CreateAccessGrantsInstance

      • s3:CreateAccessGrantsLocation

      • s3:DeleteAccessGrantsInstance

      • s3:GetAccessGrantsInstance

      • s3:GetAccessGrantsInstanceForPrefix

      • s3:GetAccessGrantsInstanceResourcePolicy

      • s3:ListAccessGrants

      • s3:ListAccessGrantsLocations

    • accessgrant resource:

      • s3:DeleteAccessGrant

      • s3:GetAccessGrant

    • bucket resource: s3:ListBucket

    • role resource:

      • iam:GetRole

      • iam:PassRole

    • all resources: s3:ListAccessGrantsInstances

Set up S3 Access Grants instance

2. • sts:AssumeRole

   • sts:SetSourceIdentity

{
  "Version": "2012-10-17",
    "Statement": [
    {
      "Sid": "Stmt1234567891011",
      "Effect": "Allow",
      "Principal": {
        "Service":"access-grants.s3.amazonaws.com"
      },
      "Action": [
        "sts:AssumeRole", 
        "sts:SetSourceIdentity"
      ]
    }
  ]
}           

• s3:GetObject

• s3:GetObjectVersion

• s3:GetObjectAcl

• s3:GetObjectVersionAcl

• s3:ListMultipartUploadParts

• s3:PutObject

• s3:PutObjectAcl

• s3:PutObjectVersionAcl

• s3:DeleteObject

• s3:DeleteObjectVersion

• s3:AbortMultipartUpload

• s3:ListBucket

• s3:ListAllMyBuckets

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ObjectLevelReadPermissions",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:GetObjectAcl",
                "s3:GetObjectVersionAcl",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                <bucket arn>
            ]
        },
        {
            "Sid": "ObjectLevelWritePermissions",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:PutObjectVersionAcl",
                "s3:DeleteObject",
                "s3:DeleteObjectVersion",
                "s3:AbortMultipartUpload"
            ],
            "Resource": [
                <bucket arn>
            ]
        },
        {
            "Sid": "BucketLevelReadPermissions",
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "s3:ListBucket"
            ],
            "Resource": [
                <bucket arn>
            ]
        }
    ]
}

If you use server-side encryption with AWS Key Management Service (AWS KMS) keys to encrypt your data, the following permissions are required for the IAM role in the policy. If you do not use this feature, do not include these permissions in your IAM policy:

• kms:Decrypt

• kms:GenerateDataKey

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "RolePermissions",
            "Effect": "Allow",
            "Action": [
                "iam:GetRole",
                "iam:PassRole"
            ],
            "Resource": "<role_arn>"
        },
        {
            "Sid": "AccessGrants",
            "Effect": "Allow",
            "Action": [
                "s3:CreateAccessGrant",
                "s3:DeleteAccessGrantsLocation",
                "s3:GetAccessGrantsLocation",
                "s3:CreateAccessGrantsLocation",
                "s3:GetAccessGrantsInstance",
                "s3:GetAccessGrantsInstanceForPrefix",
                "s3:GetAccessGrantsInstanceResourcePolicy",
                "s3:ListAccessGrants",
                "s3:ListAccessGrantsLocations",
                "s3:ListAccessGrantsInstances",
                "s3:DeleteAccessGrant",
                "s3:GetAccessGrant"
            ],
            "Resource": [
                "<access_grants_instance_arn>"
            ]
        }
    ]
}

{
  "Sid": "sso",
  "Effect": "Allow",
  "Action": [
    "sso:DescribeInstance",
    "sso:DescribeApplication",
    "sso-directory:DescribeUsers"
  ],
  "Resource": [
    "<iam_identity_center_instance_arn>",
    "<iam_identity_center_application_arn_for_s3_access_grants>",
    "arn:aws:identitystore:::user/*",
    "arn:aws:identitystore::<aws_account>:identitystore/<identity_store_id>"
  ]
}, {
  "Sid": "idc",
  "Effect": "Allow",
  "Action": [
    "identitystore:DescribeUser",
    "identitystore:DescribeGroup"
  ],
  "Resource": [
    "<iam_identity_center_instance_arn>",
    "<iam_identity_center_application_arn_for_s3_access_grants>",
    "arn:aws:identitystore:::user/*",
    "arn:aws:identitystore::<aws_account>:identitystore/<identity_store_id>"
  ]
}

Configure the integration in Immuta

1. In Immuta, click the App Settings icon in the navigation menu and click the Integrations tab.

2. Click + Add Integration.

3. Select Amazon S3 from the dropdown menu and click Continue Configuration.

4. Complete the connection details fields, where

   • Friendly Name is a name for the integration that is unique across all Amazon S3 integrations configured in Immuta.

   • AWS Account ID is the ID of your AWS account.

   • AWS Region is the AWS region to use.

   • S3 Access Grants Location IAM Role ARN is the role the S3 Access Grants service assumes to vend credentials to the grantee. When a grantee accesses S3 data, the Access Grants service attaches session policies and assumes this role in order to vend credentials scoped to a prefix or bucket to the grantee. This role needs full access to all paths under the S3 location prefix.

   • S3 Access Grants S3 Location Scope is the base S3 location that Immuta will use for this connection when registering S3 prefixes. This path must be unique across all S3 integrations configured in Immuta. During data source registration, this prefix is prepended to the data source prefixes to build the final path used to grant or revoke access to that data in S3. For example, a location prefix of s3://research-data would be prepended to the data source prefix /demographics to generate a final path of s3://research-data/demographics.

5. Select your authentication method:

   • Access using AWS IAM role: Provide an AWS IAM Role that Immuta will assume when interacting with the AWS API. This option allows you to provide Immuta with an IAM role from your AWS account that is granted a trust relationship with Immuta's IAM role for providing S3 access grants operations. Immuta will assume this IAM role from Immuta's AWS account in order to perform any operations in your AWS account. Before proceeding, contact your Immuta representative for the AWS account to add to your trust policy. Then, complete the steps below.

     • Enter the role ARN in the AWS IAM Role field. Immuta will assume this role when interacting with AWS.
   • Access using access key and secret access key: Provide your AWS Access Key ID and AWS Secret Access Key.

6. Click Verify Credentials.

7. Click Next to review and confirm your connection information, and then click Complete Setup.

Editing an integration

You can edit the following settings for an existing Amazon S3 integration on the app settings page:

• friendly name

• authentication type and values (access key, secret, and role)

Register S3 data

Protect data

Requirements: USER_ADMIN Immuta permission and either the GOVERNANCE or CREATE_S3_DATASOURCE Immuta permission

2. Map AWS IAM principals to each Immuta user to ensure Immuta properly enforces policies:

   1. Click Identities in the navigation menu and select Users.

   2. Navigate to the user's page and click the more actions icon next to their username.

   3. Select Change S3 User or AWS IAM Role from the dropdown menu.

   4. • Unset (fallback to Immuta username): When selecting this option, the S3 username is assumed to be the same as the Immuta username.

   5. Click Save.

Access data

Requirement: User must be subscribed to the data source in Immuta

S3 integration overview

With this integration, users can avoid

• hand-writing AWS IAM policies

• managing AWS IAM role limits

• manually tracking what user or role has access to what files in AWS S3 and verifying those are consistent with intent

S3 Access Grants components

To enforce controls on S3 data, Immuta interacts with several S3 Access Grants components:

• Access Grants instance: An Access Grants instance is a logical container for individual grants that specify who can access what level of data in S3 in your AWS account and region. AWS supports one Access Grants instance per region per AWS account.

• Location: A location specifies what data the Access Grants instance can grant access to. For example, registering a location with a scope of s3:// allows Access Grants to manage access to all S3 buckets in that AWS account and region, whereas setting the bucket s3://research-data as the scope limits Access Grants to managing access to that single bucket for that location. When you configure the S3 integration in Immuta, you specify a location's scope and IAM assumed role, and Immuta registers the location in your Access Grants instance and associates it with the provided IAM role for you. Each S3 integration you configure in Immuta is associated with one location, and Immuta manages all grants in that location. Therefore, grants cannot be manually created by users in an Access Grants instance location that Immuta has registered and manages. During data source registration, this location scope is prepended to the data source prefixes to build the final path used to grant or revoke access to that data in S3. For example, a location scope of s3://research-data would be prepended to the data source prefix /demographics to generate a final path of s3://research-data/demographics.

• Individual grants: Individual permission grants in S3 Access Grants specify the identity that can access the data, the access level, and the location of the S3 data. Immuta creates a grant for each user subscribed to a prefix, bucket, or object by interacting with the Access Grants API. Each grant has its own ID and gives the user or role principle access to the data.

• IAM assumed role: This is an IAM role you create in S3 that has full access to all prefixes, buckets, and objects in the Access Grants location registered by Immuta. This IAM role is used to vend temporary credentials to users or applications. When a grantee requests temporary credentials, the S3 Access Grants service assumes this role to vend credentials scoped to the prefix, bucket, or object specified in the grant to the grantee. The grantee then uses these credentials to access S3 data. When configuring the integration in Immuta, you specify this role, and then Immuta associates this role with the registered location in the Access Grants instance.

The diagram below illustrates how these S3 Access Grants components interact.

How does the integration work?

After an administrator creates an Access Grants instance and an assumed IAM role in their AWS account, an application administrator configures the Amazon S3 integration in Immuta. During configuration, the administrator provides the following connection information so that Immuta can create and register a location in that Access Grants instance:

• AWS account ID and region

• ARN for the existing Access Grants instance

• ARN for the assumed IAM role

In the example below, an application administrator registers the following location prefix and IAM role for their Access Grants instance in AWS account 123456:

• Location path: s3://. This path allows a single Amazon S3 integration to manage all objects in S3 in that AWS account and region. Data owners can scope down access further when registering specific S3 prefixes and applying policies.

• Location IAM role: The arn:aws:iam::123456:role/access-grants-role IAM role will be used to vend temporary credentials to users and applications.

Immuta registers this location and associated IAM role in the user's Access Grants instance:

After the S3 integration is configured, a data owner can register S3 prefixes and buckets that are in the configured Access Grants location path to enforce access controls on resources. Immuta stores the connection information for the prefix so that the metadata can be used to create and enforce subscription policies on S3 data.

A data owner or governor can apply a subscription policy to a registered prefix, bucket, or object to control who can access objects beginning with that prefix or in that bucket after it is registered in Immuta. Once a subscription policy is created and Immuta users are subscribed to the prefix, bucket, or object, Immuta calls the Access Grants API to create a grant for each subscribed user, specifying the following parameters in the payload so that Access Grants can create and store a grant for each user:

• Access Grants location

• READ access

• User or role principle

• Registered prefix, bucket, or object

In the example below, a data owner registers the s3://research-data/* bucket, and Immuta stores the connection information in the Immuta metadata database. Once the user, Taylor, is subscribed to s3://research-data/*, Immuta calls the Access Grants API to create a grant for that user to allow them to read and write S3 data in that bucket:

Integration health status

The status of the integration is visible on the integrations tab of the Immuta application settings page. If errors occur in the integration, a banner will appear in the Immuta UI with guidance for remediating the error.

Accessing S3 data

In the example below, Taylor requests temporary credentials from S3 Access Grants. Access Grants looks up the grant ID (1) for that user, assumes the arn:aws:iam::123456:role/access-grants-role IAM role for the location, and vends temporary credentials to Taylor, who then uses the credentials to access the research-data bucket in S3:

Note that when accessing data through S3 Access Grants, the user or application interacts directly with the Access Grants API to request temporary credentials; Immuta does not act in this process at all. See the diagram below for an illustration of the process for accessing data through S3 Access Grants.

Policy enforcement

Immuta's S3 integration allows data owners and governors to apply object-level access controls on data in S3 through subscription policies. When a user is subscribed to a registered prefix, bucket, or object, Immuta calls the Access Grants API to create an individual grant that narrows the scope of access within the location to that registered prefix, bucket, or object. See the diagram below for a visualization of this process.

When a user's entitlements change or a subscription policy is added to, updated, or deleted from a prefix, Immuta performs one of the following processes for each user subscribed to the registered prefix:

• User added to the prefix: Immuta specifies a permission (READ or READWRITE) for each user and uses the Access Grants API to create an individual grant for each user.

• User updated: Immuta deletes the current grant ID and creates a new one using the Access Grants API.

• User deleted: Immuta deletes the grant ID using the Access Grants API.

• Read access policies manage who can get objects from S3.

• Write access policies manage who can modify data in S3.

Data policies, which provide more granular controls by redacting or masking values in a table, are not supported for S3.

Prefix registration

Each prefix added in the data registration workflow is created as a single Immuta data source, and a subscription policy added to a data source applies to any objects in that bucket or beginning with that prefix:

Therefore, data owners should register prefixes or buckets at the lowest level of access control they need for that data. Using the example above, if the data owner needed to allow different users to access s3://yellow-bucket/research-data/* than those who should access s3://yellow-bucket/analyst-data/*, the data owner must register the research-data/* and analyst-data/* prefixes separately and then apply a subscription policy to those prefixes:

Deleting registered prefixes

When an S3 data source is deleted, Immuta deletes all the grants associated with that prefix, bucket, or object in that location.

User provisioning

However, if you manage access in AWS through IAM roles instead of users, user provisioning in Immuta must be done using IAM role principals. This means that if users share IAM roles, you could end up in a situation where you over-provision access to everyone in the IAM role.

See the guidelines below for the best practices to avoid this behavior if you currently use IAM roles to manage access.

Mapping IAM principals in Immuta

Immuta supports mapping an Immuta user to AWS in one of the following ways:

Existing S3 integrations

The Amazon S3 integration will not interfere with existing legacy S3 integrations, and multiple S3 integrations can exist in a single Immuta tenant.

Supported AWS services

Limitations

• During private preview, Immuta supports up to 500 prefixes (data sources) and up to 20 Immuta users that are mapped to S3 identities principals. This is a preview limitation that will be removed in a future phase of the integration.

• The following Immuta features are not currently supported by the integration in private preview:

  • Audit

  • Data policies

  • Schema monitoring

  • Tag ingestion

Requirements

• No AWS Lake Formation connections configured in the same Immuta instance for the same Glue Data Catalog.

Permissions

• APPLICATION_ADMIN Immuta permission to register the connection

• Create LF-Tag AWS permission

• • • glue:GetDatabase

    • glue:GetTables

    • glue:GetDatabases

    • glue:GetTable

  • • lakeformation:ListPermissions

    • lakeformation:BatchGrantPermissions

    • lakeformation:BatchRevokePermissions

    • lakeformation:CreateLFTag

    • lakeformation:UpdateLFTag

    • lakeformation:DeleteLFTag

    • lakeformation:AddLFTagsToResource

    • lakeformation:RemoveLFTagsFromResource

Set up the Immuta service principal

The Immuta service principal is the AWS IAM role that Immuta will assume to perform operations in your AWS account. This role must have all the necessary permissions in AWS Glue and AWS Lake Formation to allow Immuta to register data sources and apply policies.

2. Add the following IAM permissions to the service principal from the admin account. These permissions will allow the service principal to register data sources and apply policies on Immuta's behalf.
3. Grant the service principal permissions on any tables that will be registered in Immuta. There are two ways to give the service principal these permissions: either make a new LF-Tag that gives the appropriate permissions and apply it to all databases or tables that Immuta will manage, or make the role a superuser in Lake Formation.

Register an AWS Lake Formation connection

1. Click Data and select Connections in the navigation menu.

2. Click the + Add Connection button.

3. Select the AWS Lake Formation tile.

4. Enter the host connection information:

   1. Display Name: This is the name of your new connection. This name will be used in the API (connectionKey), in data source names from the host, and on the connections page.

   2. AWS Glue Catalog ARN: The Amazon resource name of the Glue Data Catalog that contains the data you want to register and protect.

   3. AWS Account ID: The ID of the AWS account associated with the Glue Data Catalog.

   4. AWS Region: The region of the AWS account associated with the Glue Data Catalog.

5. Click Next.

6. Select an authentication method from the dropdown menu.

   2. AWS IAM Role (recommended): Immuta will assume this IAM role from Immuta's AWS account in order to perform any operations in your AWS account. Before proceeding, contact your Immuta representative for the AWS account to add to your trust policy. Then, complete the steps below.

      1. Enter the role ARN in the AWS IAM Role field. Immuta will assume this role when interacting with AWS.
7. Ensure that you have the correct permissions and click Validate Connection.

8. If the connection is successful, click Next. If there are any errors, check the connection details and credentials to ensure they are correct and try again.

9. Ensure all the details are correct in the summary and click Complete Setup.

Map users

Requirement: USER_ADMIN Immuta permission

Map AWS IAM principals to each Immuta user to ensure Immuta properly enforces policies.

1. Click People and select Users in the navigation menu.

2. Click the user's name to navigate to their page and scroll to the External User Mapping section.

3. Click Edit in the AWS User row.

4. • AWS Identity Center user IDs: You must use the numeric User ID value found in AWS IAM Identity Center, not the user's email address.

   • Unset (fallback to Immuta username): When selecting this option, the AWS username is assumed to be the same as the Immuta username.

5. Click Save.

Once data is registered through the AWS Lake Formation connection, you will access your data in one of these AWS analytic engines as you normally would:

• Amazon Athena

• Amazon EMR Spark

• Amazon Redshift Spectrum

When you submit a query, the analytic engine requests metadata from Glue Data Catalog, which then queries Lake Formation to determine what data you are allowed to see. Then, the analytic engine requests temporary access from Lake Formation, retrieves the data from S3, and filters the data to returns policy-enforced data to you.

The diagram below illustrates how the analytic engine interacts with Glue Data Catalog and Lake Formation to access data.

In the AWS Lake Formation connection, Immuta orchestrates Lake Formation access controls on data registered in the Glue Data Catalog. Then, Immuta users who have been granted access to the Glue Data Catalog table or view can query it using one of these analytic engines:

• Amazon Athena

• Amazon EMR Spark

• Amazon Redshift Spectrum

The sequence diagram below outlines the events that occur when an Immuta user who is subscribed to a data source submits a query in their AWS analytic engine.

Registering a connection

Once the Lake Formation connection is registered, you can author policies in Immuta to orchestrate Lake Formation access controls.

Protecting data

After Glue Data Catalog views and tables are registered in Immuta, you can author subscription policies in Immuta to orchestrate Lake Formation access controls. Once a subscription policy is applied, users can be subscribed to data sources in the following ways:

Consider the following example that illustrates how Immuta enforces a subscription policy that only allows users in the analysts group to access to yellow-table. When this policy is authored and applied to the data source, Immuta generates a Lake Formation (LF) tag that is applied to the Glue Data Catalog yellow-table and permissions on that tag are granted to all AWS users (registered in Immuta) that are part of the analysts group.

In the image above, the user in the analysts group accesses yellow-table , while the user who is a part of the research group is denied access.

Immuta gives everyone fast, governed access to data with the built-in controls, collaboration workflows, automated provisioning, and continuous monitoring you need to keep risk low and compliance high.

Configure Immuta

Explore Immuta

Immuta offers several features to provide security for your users and to prove compliance and monitor for anomalies.

Security

Data processing and encryption

Authentication

Registering the connection

The Lake Formation connection supports the following authentication methods to register a connection:

• Access using AWS IAM role (recommended): Immuta will assume this role when interacting with the AWS API. This option allows you to provide Immuta with an IAM role from your AWS account that is granted a trust relationship with Immuta's IAM role. Immuta will assume this IAM role from Immuta's AWS account in order to perform any operations in your AWS account.

• Access using access key and secret access key: These credentials are used temporarily by Immuta to register the connection.

Identity providers for user authentication

The built-in Immuta IAM can be used as a complete solution for authentication and user entitlement. However, you can connect your existing identity management provider to Immuta to use that system for authentication and user entitlement instead.

Each of the supported identity providers includes a specific set of configuration options that enable Immuta to communicate with the IAM system and map the users, permissions, groups, and attributes into Immuta.

Auditing and compliance

Immuta provides governance reports so that data owners and governors can monitor users' access to data and detect anomalies in behavior.

Immuta governance reports allow users with the GOVERNANCE Immuta permission to use a natural language builder to instantly create reports that delineate user activity across Immuta. These reports can be based on various entity types, including users, groups, projects, data sources, purposes, policy types, or connection types.

Requirement: A running Dedicated SQL pool

Requirement

A running Dedicated SQL pool is required.

Add an Azure Synapse Analytics integration

1. Click the App Settings icon in the navigation menu.

2. Click the Integrations tab.

3. Click the +Add Integration button and select Azure Synapse Analytics from the dropdown menu.

4. Complete the Host, Port, Immuta Database, and Immuta Schema fields.

5. Opt to check the Enable Impersonation box and customize the Impersonation Role name as needed. This will allow users to natively impersonate another user.

6. Opt to update the User Profile Delimiters. This will be necessary if any of the provided symbols are used in user profile information.

Select your configuration method

You have two options for configuring your Azure Synapse Analytic environment:

Automatic setup

Enter the username and password in the Privileged User Credentials section.

Manual setup

1. Select Manual.

2. Download, fill out the appropriate fields, and run the bootstrap master script and bootstrap script linked in the Setup section.

3. Enter the username and password in the Immuta System Account Credentials section. The username and password provided must be the credentials that were set in the bootstrap master script when you created the user.

Save the configuration

Click Save.

Register data

Edit an Azure Synapse Analytics integration

1. Click the App Settings icon in the navigation menu.

2. Navigate to the Integrations tab and click the down arrow next to the Azure Synapse Analytics Integration.

3. Edit the field you want to change. Note any field shadowed is not editable, and the integration must be disabled and re-installed to change it.

4. Enter Username and Password.

5. Click Save.

Remove an Azure Synapse Analytics integration

1. Click the App Settings icon in the navigation menu.

2. Navigate to the Integrations tab and click the down arrow next to the Azure Synapse Analytics Integration.

3. Click the checkbox to disable the integration.

4. Enter the username and password that were used to initially configure the integration.

5. Click Save.

If a Databricks cluster needs to be manually updated to reflect changes in the Immuta init script or cluster policies, you can remove and set up your integration again to get the updated policies and init script.

1. Log in to Immuta as an Application Admin.

2. Click the App Settings icon in the navigation menu and scroll to the Integration Settings section.

3. Your existing Databricks Spark integration should be listed here; expand it and note the configuration values. Now select Remove to remove your integration.

4. Click Add Integration and select Databricks Integration to add a new integration.

5. Enter your Databricks Spark integration settings again as configured previously.

6. Click Add Integration to add the integration, and then select Configure Cluster Policies to set up the updated cluster policies and init script.

7. Select the cluster policies you wish to use for your Immuta-enabled Databricks clusters.

8. Automatically push cluster policies and the init script (recommended) or manually update your cluster policies.

   • Automatically push cluster policies

     1. Select Automatically Push Cluster Policies and enter your privileged Databricks access token. This token must have privileges to write to cluster policies.

     2. Select Apply Policies to push the cluster policies and init script again.

     3. Click Save and Confirm to deploy your changes.

   • Manually update cluster policies

     1. Download the init script and the new cluster policies to your local computer.

     2. Click Save and Confirm to save your changes in Immuta.

     3. Log in to your Databricks workspace with your administrator account to set up cluster policies.

     4. Get the path you will upload the init script (immuta_cluster_init_script_proxy.sh) to by opening one of the cluster policy .json files and looking for the defaultValue of the field init_scripts.0.dbfs.destination. This should be a DBFS path in the form of dbfs:/immuta-plugin/hostname/immuta_cluster_init_script_proxy.sh.

     5. Click Data in the left pane to upload your init script to DBFS to the path you found above.

     6. To find your existing cluster policies you need to update, click Compute in the left pane and select the Cluster policies tab.

     7. Edit each of these cluster policies that were configured before and overwrite the contents of the JSON with the new cluster policy JSON you downloaded.

9. Restart any Databricks clusters using these updated policies for the changes to take effect.

The how-to guides linked on this page illustrate how to integrate Databricks Spark with Immuta.

Requirements

• If Databricks Unity Catalog is not enabled in your Databricks workspace, you must disable Unity Catalog in your Immuta tenant before proceeding with your configuration of Databricks Spark:

  1. Navigate to the App Settings page and click Integration Settings.

  2. Uncheck the Enable Unity Catalog checkbox.

  3. Click Save.

Immuta offers two integrations for Databricks:

Which integration should you use?

To determine which integration you should use, evaluate the following elements:

• Cluster runtime

  • Databricks Runtime 11.3 and newer: See the list below to determine which integration is supported for your data's location.

• Location of data: Where is your data?

Metastore magic

Databricks metastore magic allows you to migrate your data from the Databricks legacy Hive metastore to the Unity Catalog metastore while protecting data and maintaining your current processes in a single Immuta instance.

Requirement

Unity Catalog support is enabled in Immuta.

Databricks metastores and Immuta policy enforcement

Databricks has two built-in metastores that contain metadata about your tables, views, and storage credentials:

• Legacy Hive metastore: Created at the workspace level. This metastore contains metadata of the registered securables in that workspace available to query.

• Unity Catalog metastore: Created at the account level and is attached to one or more Databricks workspaces. This metastore contains metadata of the registered securables available to query. All clusters on that workspace use the configured metastore and all workspaces that are configured to use a single metastore share those securables.

Databricks allows you to use the legacy Hive metastore and the Unity Catalog metastore simultaneously. However, Unity Catalog does not support controls on the Hive metastore, so you must attach a Unity Catalog metastore to your workspace and move existing databases and tables to the attached Unity Catalog metastore to use the governance capabilities of Unity Catalog.

Immuta's Databricks Spark integration and Unity Catalog integration enforce access controls on the Hive and Unity Catalog metastores, respectively. However, because these metastores have two distinct security models, users were discouraged from using both in a single Immuta instance before metastore magic; the Databricks Spark integration and Unity Catalog integration were unaware of each other, so using both concurrently caused undefined behavior.

Databricks metastore magic solution

In clusters A and D, Immuta enforces policies on data sources in each workspace's Hive metastore and in the Unity Catalog metastore shared by those workspaces. In clusters B, C, and E (which don't have Unity Catalog enabled in Databricks), Immuta enforces policies on data sources in the Hive metastores for each workspace.

Enforce policies as you migrate

With metastore magic, the Databricks Spark integration enforces policies only on data in the Hive metastore, while the Unity Catalog integration enforces policies on tables in the Unity Catalog metastore. The table below illustrates this policy enforcement.

Enforcing policies on Databricks SQL

Databricks SQL cannot run the Databricks Spark plugin to protect tables, so Hive metastore data sources will not be policy enforced in Databricks SQL.

Feature Availability

Prerequisite

• A running Dedicated SQL pool

Authentication Method

The Azure Synapse Analytics integration supports the username and password authentication method to configure the integration and create data sources.

Tag Ingestion

User Impersonation

Multiple Integrations

Limitations

• Immuta does not support the following masking types in this integration because of limitations with Dedicated SQL pools (linked below). Any column assigned one of these masking types will be masked to NULL:
• The delimiters configured when enabling the integration cannot be changed once they are set. To change the delimiters, the integration has to be disabled and re-enabled.

• If the generated view name is more than 128 characters, then the view name is shortened to 128 characters. This could cause collisions between view names if the shortened version is the same for two different data sources.

• For proper updates, the Dedicated SQL pools have to be running when changes are made to users or data sources in Immuta.

In this integration, Immuta generates policy-enforced views in a schema in your configured Azure Synapse Analytics Dedicated SQL pool for tables registered as Immuta data sources.

This guide outlines how to integrate Azure Synapse Analytics with Immuta.

How-to guide

Reference guide

This guide illustrates how to run R and Scala spark-submit jobs on Databricks, including prerequisites and caveats.

R spark-submit

Prerequisites

Before you can run spark-submit jobs on Databricks, complete the following steps.

1. Initialize the Spark session:

   1. Enter these settings into the R submit script to allow the R script to access Immuta data sources, scratch paths, and workspace tables: immuta.spark.acl.assume.not.privileged="true" and spark.hadoop.immuta.databricks.config.update.service.enabled="false".

   2. Once the script is written, upload the script to a location in dbfs/S3/ABFS to give the Databricks cluster access to it.

2. Because of how some user properties are populated in Databricks, load the SparkR library in a separate cell before attempting to use any SparkR functions.

Create the R spark submit Job

To create the R spark-submit job,

1. Go to the Databricks jobs page.

2. Create a new job, and select Configure spark-submit.

3. Set up the parameters:

   Note: The path dbfs:/path/to/script.R can be in S3 or ABFS (on Azure Databricks), assuming the cluster is configured with access to that path.

Scala spark-submit

Prerequisites

Before you can run spark-submit jobs on Databricks you must initialize the Spark session with the settings outlined below.

1. Configure the Spark session with immuta.spark.acl.assume.not.privileged="true" and spark.hadoop.immuta.databricks.config.update.service.enabled="false".

   Note: Stop your Spark session (spark.stop()) at the end of your job or the cluster will not terminate.

2. The spark submit job needs to be launched using a different classloader which will point at the designated user JARs directory. The following Scala template can be used to handle launching your submit code using a separate classloader:

Create the Scala spark-submit Job

To create the Scala spark-submit job,

1. Build and upload your JAR to dbfs/S3/ABFS where the cluster has access to it.

2. Select Configure spark-submit, and configure the parameters:

   Note: The fully-qualified class name of the class whose main function will be used as the entry point for your code in the --class parameter.

   Note: The path dbfs:/path/to/code.jar can be in S3 or ABFS (on Azure Databricks) assuming the cluster is configured with access to that path.

4. Include IMMUTA_INIT_ADDITIONAL_JARS_URI=dbfs:/path/to/code.jar in the "Environment Variables" (where dbfs:/path/to/code.jar is the path to your jar) so that the jar is uploaded to all the cluster nodes.

Caveats

• The user mapping works differently from notebooks because spark-submit clusters are not configured with access to the Databricks SCIM API. The cluster tags are read to get the cluster creator and match that user to an Immuta user.

• Privileged users (Databricks admins and allowlisted users) must be tied to an Immuta user and given access through Immuta to access data through spark-submit jobs because the setting immuta.spark.acl.assume.not.privileged="true" is used.

• There is an option of using the immuta.api.key setting with an Immuta API key generated on the Immuta profile page.

• Currently when an API key is generated it invalidates the previous key. This can cause issues if a user is using multiple clusters in parallel, since each cluster will generate a new API key for that Immuta user. To avoid these issues, manually generate the API key in Immuta and set the immuta.api.key on all the clusters or use a specified job user for the submit job.

1. In the Databricks Clusters UI, install your third-party library .jar or Maven artifact with Library Source Upload, DBFS, DBFS/S3, or Maven. Alternatively, use the Databricks libraries API.

2. In the Databricks Clusters UI, add the IMMUTA_SPARK_DATABRICKS_TRUSTED_LIB_URIS property as a Spark environment variable and set it to your artifact's URI:

1. Once you've finished making your changes, restart the cluster.

2. Once the cluster is up, execute a command in a notebook. If the trusted library installation is successful, you should see driver log messages like this:

Permissions

• APPLICATION_ADMIN Immuta permission

• CAN MANAGE Databricks privilege on the cluster

Requirements

• A Databricks workspace with the Premium tier, which includes cluster policies (required to configure the Spark integration)

• A cluster that uses one of these supported Databricks Runtimes:

  • 9.1 LTS

  • 10.4 LTS

  • 11.3 LTS

  • 14.3 (private preview)

• Supported languages

  • Python

  • R (not supported for Databricks Runtime 14.3)

  • Scala (not supported for Databricks Runtime 14.3)

  • SQL

• A Databricks cluster that is one of these supported compute types:
• Custom access mode

• The Databricks Spark integration only works with Spark 3.

Prerequisites

• If Databricks Unity Catalog is not enabled in your Databricks workspace, you must disable Unity Catalog in your Immuta tenant before proceeding with your configuration of Databricks Spark:

  1. Navigate to the App Settings page and click Integration Settings.

  2. Uncheck the Enable Unity Catalog checkbox.

  3. Click Save.

Add the integration on the app settings page

1. Click the App Settings icon in Immuta.

2. Navigate to HDFS > System API Key and click Generate Key.

3. Click Save and then Confirm. If you do not save and confirm, the system API key will not be saved.

4. Scroll to the Integration Settings section.

5. Click + Add Native Integration and select Databricks Spark Integration from the dropdown menu.

6. Complete the Hostname field.

7. Enter a Unique ID for the integration. The unique ID is used to name cluster policies clearly, which is important when managing several Databricks Spark integrations. As cluster policies are workspace-scoped, but multiple integrations might be made in one workspace, this ID lets you distinguish between different sets of cluster policies.

8. Select the identity manager that should be used when mapping the current Spark user to their corresponding identity in Immuta from the Immuta IAM dropdown menu. This should be set to reflect the identity manager you use in Immuta (such as Entra ID or Okta).

10. Select the Storage Access Type from the dropdown menu.

11. Opt to add any Additional Hadoop Configuration Files.

12. Click Add Native Integration, and then click Save and Confirm. This will restart the application and save your Databricks Spark integration. (It is normal for this restart to take some time.)

The Databricks Spark integration will not do anything until your cluster policies are configured, so even though your integration is saved, continue to the next section to configure your cluster policies so the Spark plugin can manage authorization on the Databricks cluster.

Configure cluster policies

1. Click Configure Cluster Policies.

3. Select your Databricks Runtime.

4. Use one of the two installation types described below to apply the policies to your cluster:

   • Automatically push cluster policies: This option allows you to automatically push the cluster policies to the configured Databricks workspace. This will overwrite any cluster policy templates previously applied to this workspace.

     1. Select the Automatically Push Cluster Policies radio button.

     3. Click Apply Policies.

   • Manually push cluster policies: Enabling this option allows you to manually push the cluster policies and the init script to the configured Databricks workspace.

     1. Select the Manually Push Cluster Policies radio button.

     3. 1. Ensure that the init_scripts.0.workspace.destination in the policy matches the file path to the init script you configured above.

        2. The Immuta cluster policy references Databricks Secrets for several of the sensitive fields. These secrets must be manually created if the cluster policy is not automatically pushed. Use Databricks API or CLI to push the proper secrets.

5. Click Close, and then click Save and Confirm.

Map users and grant them access to the cluster

2. Give users the Can Attach To permission on the cluster.

This page outlines how to enable access to DBFS in Databricks for non-sensitive data. Databricks administrators should place the desired configuration in the Spark environment variables.

DBFS FUSE mount

This Databricks feature mounts DBFS to the local cluster filesystem at /dbfs. Although disabled when using process isolation, this feature can safely be enabled if raw, unfiltered data is not stored in DBFS and all users on the cluster are authorized to see each other’s files. When enabled, the entirety of DBFS essentially becomes a scratch path where users can read and write files in /dfbs/path/to/my/file as though they were local files.

For example,

In Python,

Note: This solution also works in R and Scala.

Enable DBFS FUSE mount

To enable the DBFS FUSE mount, set this configuration in the Spark environment variables: IMMUTA_SPARK_DATABRICKS_DBFS_MOUNT_ENABLED=true.

Scala DBUtils (and %fs magic) with scratch paths

Scratch paths will work when performing arbitrary remote filesystem operations with fs magic or Scala dbutils.fs functions. For example,

Configure Scala DBUtils (and %fs magic) with scratch paths

To support %fs magic and Scala DBUtils with scratch paths, configure

Configure DBUtils in Python

To use dbutils in Python, set this configuration: immuta.spark.databricks.py4j.strict.enabled=false.

Example workflow

This section illustrates the workflow for getting a file from a remote scratch path, editing it locally with Python, and writing it back to a remote scratch path.

1. Get the file from remote storage:
2. Make a copy if you want to explicitly edit localScratchFile, as it will be read-only and owned by root:
3. Write the new file back to remote storage:

When the Databricks Spark plugin is running on a Databricks cluster, all Databricks users running jobs or queries are either a privileged user or a non-privileged user:

• Non-privileged users: Non-privileged users are any users who are not privileged users, and all authorization for non-privileged users is determined by Immuta policies.

Mapping Databricks users to Immuta

User impersonation

Databricks user impersonation allows a Databricks user to impersonate an Immuta user. With this feature,

• the Immuta user who is being impersonated does not have to have a Databricks account, but they must have an Immuta account.

• the Databricks user who is impersonating an Immuta user does not have to be associated with Immuta. For example, this could be a service account.

When acting under impersonation, the Databricks user loses their privileged access, so they can only access the tables the Immuta user has access to and only perform DDL commands when that user is acting under an allowed circumstance (such as workspaces, scratch paths, or non-Immuta reads/writes).

The Databricks Spark integration is one of two integrations Immuta offers for Databricks.

In this integration, Immuta installs an Immuta-maintained Spark plugin on your Databricks cluster. When a user queries data that has been registered in Immuta as a data source, the plugin injects policy logic into the plan Spark builds so that the results returned to the user only include data that specific user should see.

The reference guides in this section are written for Databricks administrators who are responsible for setting up the integration, securing Databricks clusters, and setting up users:

Overview

The Azure Synapse Analytics is a policy push integration that allows Immuta to apply policies directly in Azure Synapse Analytics Dedicated SQL pools without the need for users to go through a proxy. Instead, users can work within their existing Synapse Studio and have per-user policies dynamically applied at query time.

Architecture

This integration works on a per-Dedicated-SQL-pool basis: all of Immuta's policy definitions and user entitlements data need to be in the same pool as the target data sources because Dedicated SQL pools do not support cross-database joins. Immuta creates schemas inside the configured Dedicated SQL pool that contain policy-enforced views that users query.

When the integration is configured, the Application Admin specifies the

• Immuta Database: This is the pre-existing database Immuta uses. Immuta will create views from the tables contained in this database, and all schemas and views created by Immuta will exist in this database, such as the schemas immuta_system, immuta_functions, and the immuta_procedures that contain the tables, views, UDFs, and stored procedures that support the integration.

• Immuta Schema: The schema that Immuta manages. All views generated by Immuta for tables registered as data sources will be created in this schema.

• User Profile Delimiters: Since Azure Synapse Analytics dedicated SQL pools do not support array or hash objects, certain user access information is stored as delimited strings; the Application Admin can modify those delimiters to ensure they do not conflict with possible characters in strings.

Data source naming convention

Synapse data sources are represented as views and are under one schema instead of a database, so their view names are a combination of their schema and table name, separated by an underscore.

For example, with a configuration that uses IMMUTA as the schema in the database dedicated_pool, the view name for the data source dedicated_pool.tpc.case would be dedicated_pool.IMMUTA.tpc_case.

You can see the view information on the data source details page under Connection Information.

Policy enforcement

This integration uses webhooks to keep views up-to-date with the corresponding Immuta data sources. When a data source or policy is created, updated, or disabled, a webhook is called that creates, modifies, or deletes the dynamic view in the Immuta schema. Note that only standard views are available because Azure Synapse Analytics Dedicated SQL pools do not support secure views.

Integration health status

The status of the integration is visible on the integrations tab of the Immuta application settings page. If errors occur in the integration, a banner will appear in the Immuta UI with guidance for remediating the error.

Data flow

2. Immuta creates Immuta schemas inside the configured Synapse Dedicated SQL pool.

4. Data source metadata, tags, user metadata, and policy definitions are stored in Immuta's Metadata Database.

5. The Immuta Web Service calls a stored procedure that modifies the user entitlements or policies and updates data source view definitions as necessary.

1. Lower the web service cache timeout in Immuta:

   1. Click the App Settings icon and scroll to the HDFS Cache Settings section.

   2. Lower the Cache TTL of HDFS user names (ms) to 0.

   3. Click Save.

2. Raise the cache timeout on your Databricks cluster: In the Spark environment variables section, set the IMMUTA_CURRENT_PROJECT_CACHE_TIMEOUT_SECONDS and IMMUTA_PROJECT_CACHE_TIMEOUT_SECONDS to high values (like 10000).

   Note: These caches will be invalidated on cluster when a user calls immuta.set_current_project, so they can effectively be cached permanently on cluster to avoid periodically reaching out to the web service.

In the Databricks Spark integration, Immuta installs an Immuta-maintained Spark plugin on your Databricks cluster. When a user queries data that has been registered in Immuta as a data source, the plugin injects policy logic into the plan Spark builds so that the results returned to the user only include data that specific user should see.

The sequence diagram below breaks down this process of events when an Immuta user queries data in Databricks.

System requirements

• A Databricks workspace with the Premium tier, which includes cluster policies (required to configure the Spark integration)

• A cluster that uses one of these supported Databricks Runtimes:

  • 9.1 LTS

  • 10.4 LTS

  • 11.3 LTS

  • 14.3 (private preview) - Requires Immuta version 2025.1.x or newer

• Supported languages

  • Python

  • R (not supported for Databricks Runtime 14.3)

  • Scala (not supported for Databricks Runtime 14.3)

  • SQL

• A Databricks cluster that is one of these supported compute types:
• Custom access mode

• The Databricks Spark integration only works with Spark 3.

What does Immuta do in my Databricks environment?

Once the init script runs, the Spark application running on the Databricks cluster will have the appropriate artifacts on its CLASSPATH to use Immuta for authorization and policy enforcement.

Immuta adds the following artifacts to your Databricks environment:

IMMUTA_SPARK_SHOW_IMMUTA_DATABASE=false

Once the Immuta-enabled cluster is running, the following user actions spur various processes. The list below provides an overview of each process:

• A policy is deleted: When a policy is deleted, the policy information is deleted from the Metadata Database. If users were granted access to the data source by that policy, their access is revoked.

• Databricks user queries data: When a user queries the data in Databricks, Immuta intercepts the call from Spark down to the Metastore. Then, the Immuta-maintained Spark plugin retrieves the policy information, the user metadata, and the data source metadata from the Metadata Database and injects this information as policy logic into the Spark logical plan. Once the physical plan is applied, Databricks returns policy-enforced data to the user.

The image below illustrates these processes and how they interact.

Supported policies

The Databricks Spark integration allows users to author subscription and data policies to enforce access controls. See the corresponding pages for details about specific types of policies supported:

Databricks Runtime 14.3

Immuta supports clusters on Databricks Runtime 14.3. The integration for this Databricks Runtime differs from the integration for other supported Runtimes in the following ways:

• Py4J security and process isolation automatically enabled: Immuta relies on Databricks process isolation and Py4J security to prevent user code from performing unauthorized actions. After selecting Runtime 14.3 during configuration, Immuta will automatically enable process isolation and Py4J security.

• dbutils is unsupported: Immuta relies on Databricks process isolation and Py4J security to prevent user code from performing unauthorized actions. This means that dbutils is not supported for Databricks Spark integrations using Runtime 14.3.

Cluster security and compliance

Authentication methods

The Databricks Spark integration supports the following authentication methods to configure the integration:

• Personal access token (PAT): This token gives Immuta temporary permission to push the cluster policies to the configured Databricks workspace and overwrite any cluster policy templates previously applied to the workspace when configuring the integration or to register securables as Immuta data sources.

Audit

Immuta captures the code or query that triggers the Spark plan in Databricks, making audit records more useful in assessing what users are doing. To audit what triggers the Spark plan, Immuta hooks into Databricks where notebook cells and JDBC queries execute and saves the cell or query text. Then, Immuta pulls this information into the audits of the resulting Spark jobs.

Protecting the Immuta configuration

Databricks secrets can be used in the environment variables configuration section for a cluster by referencing the secret path instead of the actual value of the environment variable. For example, if a user wanted to make the MY_SECRET_ENV_VAR=abcd_1234 value secret, they could instead create a Databricks secret and reference it as the value of that variable by following these steps:

1. Create the secret scope my_secrets and add a secret with the key my_secret_env_var containing the sensitive environment variable.

2. Reference the secret in the environment variables section as MY_SECRET_ENV_VAR={{secrets/my_secrets/my_secret_env_var}}.

At runtime, {{secrets/my_secrets/my_secret_env_var}} would be replaced with the actual value of the secret if the owner of the cluster has access to that secret.

Scala clusters

There are limitations to isolation among users in Scala jobs on a Databricks cluster, even when using Immuta’s Security Manager. When data is broadcast, cached (spilled to disk), or otherwise saved to SPARK_LOCAL_DIR, it's impossible to distinguish between which user’s data is composed in each file/block. If you are concerned about this vulnerability, Immuta suggests that you

• limit Scala clusters to Scala jobs only and

When data is read in Spark using an Immuta policy-enforced plan, the masking and redaction of rows is performed at the leaf level of the physical Spark plan, so a policy such as "Mask using hashing the column social_security_number for everyone" would be implemented as an expression on a project node right above the FileSourceScanExec/LeafExec node at the bottom of the plan. This process prevents raw data from being shuffled in a Spark application and, consequently, from ending up in SPARK_LOCAL_DIR.

This policy implementation coupled with an equalized project guarantees that data being dropped into SPARK_LOCAL_DIR will have policies enforced and that those policies will be homogeneous for all users on the cluster. Since each user will have access to the same data, if they attempt to manually access other users' cached data, they will only see what they have access to via equalized permissions on the cluster. If project equalization is not turned on, users could dig through that directory and find data from another user with heightened access, which would result in a data leak.

Troubleshooting the installation

This page outlines configuration details for Immuta-enabled Databricks clusters. Databricks administrators should place the desired configuration in the Spark environment variables.

IMMUTA_INIT_ADDITIONAL_CONF_URI

If you add additional Hadoop configuration during the integration setup, this variable sets the path to that file.

The additional Hadoop configuration is where sensitive configuration goes for remote filesystems (if you are using a secret key pair to access S3, for example).

IMMUTA_EPHEMERAL_HOST_OVERRIDE

Default value: true

Set this to false if ephemeral overrides should not be enabled for Spark. When true, this will automatically override ephemeral data source httpPaths with the httpPath of the Databricks cluster running the user's Spark application.

IMMUTA_EPHEMERAL_HOST_OVERRIDE_HTTPPATH

This configuration item can be used if automatic detection of the Databricks httpPath should be disabled in favor of a static path to use for ephemeral overrides.

IMMUTA_EPHEMERAL_TABLE_PATH_CHECK_ENABLED

Default value: true

When querying Immuta data sources in Spark, the metadata from the Metastore is compared to the metadata for the target source in Immuta to validate that the source being queried exists and is queryable on the current cluster. This check typically validates that the target (database, table) pair exists in the Metastore and that the table’s underlying location matches what is in Immuta. This configuration can be used to disable location checking if that location is dynamic or changes over time. Note: This may lead to undefined behavior if the same table names exist in multiple workspaces but do not correspond to the same underlying data.

IMMUTA_INIT_ALLOWED_CALLING_CLASSES_URI

IMMUTA_SPARK_ACL_ALLOWLIST

This is a comma-separated list of Databricks users who can access any table or view in the cluster metastore without restriction.

IMMUTA_SPARK_ACL_PRIVILEGED_TIMEOUT_SECONDS

Default value: 3600

The number of seconds to cache privileged user status for the Immuta ACL. A privileged Databricks user is an admin or is allowlisted in IMMUTA_SPARK_ACL_ALLOWLIST.

IMMUTA_SPARK_AUDIT_ALL_QUERIES

Default value: false

Enables auditing all queries run on a Databricks cluster, regardless of whether users touch Immuta-protected data or not.

IMMUTA_SPARK_DATABRICKS_ALLOW_NON_IMMUTA_READS

Default value: false

IMMUTA_SPARK_DATABRICKS_ALLOW_NON_IMMUTA_WRITES

Default value: false

IMMUTA_SPARK_DATABRICKS_ALLOWED_IMPERSONATION_USERS

This is a comma-separated list of Databricks users who are allowed to impersonate Immuta users:

"spark_env_vars.IMMUTA_SPARK_DATABRICKS_ALLOWED_IMPERSONATION_USERS": {
  "type": "fixed",
  "value": "edixon@example.com,dakota@example.com"
}

IMMUTA_SPARK_DATABRICKS_DBFS_MOUNT_ENABLED

Default value: false

Exposes the DBFS FUSE mount located at /dbfs. Granular permissions are not possible, so all users will have read/write access to all objects therein. Note: Raw, unfiltered source data should never be stored in DBFS.

IMMUTA_SPARK_DATABRICKS_DISABLED_UDFS

IMMUTA_SPARK_DATABRICKS_JAR_URI

Default value: file:///databricks/jars/immuta-spark-hive.jar

The location of immuta-spark-hive.jar on the filesystem for Databricks. This should not need to change unless a custom initialization script that places immuta-spark-hive in a non-standard location is necessary.

IMMUTA_SPARK_DATABRICKS_LOCAL_SCRATCH_DIR_ENABLED

Default value: true

Creates a world-readable or writable scratch directory on local disk to facilitate the use of dbutils and 3rd party libraries that may write to local disk. Its location is non-configurable and is stored in the environment variable IMMUTA_LOCAL_SCRATCH_DIR. Note: Sensitive data should not be stored at this location.

IMMUTA_SPARK_DATABRICKS_LOG_LEVEL

Default value: INFO

The SLF4J log level to apply to Immuta's Spark plugins.

IMMUTA_SPARK_DATABRICKS_LOG_STDOUT_ENABLED

Default value: false

If true, writes logging output to stdout/the console as well as the log4j-active.txt file (default in Databricks).

IMMUTA_SPARK_DATABRICKS_SCRATCH_DATABASE

Additionally, this configuration will only display the scratch databases that are configured and will not validate that the configured databases exist in the Metastore. Therefore, it is up to the Databricks administrator to properly set this value and keep it current.

IMMUTA_SPARK_DATABRICKS_SCRATCH_PATHS

Comma-separated list of remote paths that Databricks users are allowed to directly read/write. These paths amount to unprotected "scratch spaces." You can create a scratch database by configuring its specified location (or configure dbfs:/user/hive/warehouse/<db_name>.db for the default location).

To create a scratch path to a location or a database stored at that location, configure

IMMUTA_SPARK_DATABRICKS_SCRATCH_PATHS=s3://path/to/the/dir

To create a scratch path to a database created using the default location,

IMMUTA_SPARK_DATABRICKS_SCRATCH_PATHS=s3://path/to/the/dir,dbfs:/user/hive/warehouse/any_db_name.db</value>

IMMUTA_SPARK_DATABRICKS_SCRATCH_PATHS_CREATE_DB_ENABLED

Default value: false

Enables non-privileged users to create or drop scratch databases.

IMMUTA_SPARK_DATABRICKS_SINGLE_IMPERSONATION_USER

Default value: false

When true, this configuration prevents users from changing their impersonation user once it has been set for a given Spark session. This configuration should be set when the BI tool or other service allows users to submit arbitrary SQL or issue SET commands.

IMMUTA_SPARK_DATABRICKS_SUBMIT_TAG_JOB

Default value: true

Denotes whether the Spark job will be run that "tags" a Databricks cluster as being associated with Immuta.

IMMUTA_SPARK_DATABRICKS_TRUSTED_LIB_URIS

IMMUTA_SPARK_NON_IMMUTA_TABLE_CACHE_SECONDS

Default value: 3600

The number of seconds Immuta caches whether a table has been exposed as a data source in Immuta. This setting only applies when IMMUTA_SPARK_DATABRICKS_ALLOW_NON_IMMUTA_WRITES or IMMUTA_SPARK_DATABRICKS_ALLOW_NON_IMMUTA_READS is enabled.

IMMUTA_SPARK_REQUIRE_EQUALIZATION

Default value: false

Requires that users act through a single, equalized project. A cluster should be equalized if users need to run Scala jobs on it, and it should be limited to Scala jobs only via spark.databricks.repl.allowedLanguages.

IMMUTA_SPARK_RESOLVE_RAW_TABLES_ENABLED

Default value: true

Enables use of the underlying database and table name in queries against a table-backed Immuta data source. Administrators or allowlisted users can set IMMUTA_SPARK_RESOLVE_RAW_TABLES_ENABLED to false to bypass resolving raw databases or tables as Immuta data sources. This is useful if an admin wants to read raw data but is also an Immuta user. By default, data policies will be applied to a table even for an administrative user if that admin is also an Immuta user.

IMMUTA_SPARK_SESSION_RESOLVE_RAW_TABLES_ENABLED

Default value: true

IMMUTA_SPARK_SHOW_IMMUTA_DATABASE